Long import (start up) time

roboflow / inference

A fast, easy-to-use, production-ready inference server for computer vision supporting deployment of many popular model architectures and fine-tuned models.

https://inference.roboflow.com

Other

1.12k stars 84 forks source link

Long import (start up) time #479

Closed AJamesPhillips closed 2 weeks ago

AJamesPhillips commented 2 weeks ago

Search before asking

[X] I have searched the Inference issues and found no similar feature requests.

Question

I'm running some code on a Mac and deploying on an AWS lambda. I noticed that many of the dependencies take 3 to 5 seconds each to import, e.g. pandas ~4.5 and rasterio ~4. When I use from inference_sdk import InferenceHTTPClient, InferenceConfiguration that single import seems to take ~11 seconds (on a Mac). Do you have any advice for precompiling the library or speeding up the import please? It's not specific to inference_sdk but from my investigation I think it's got the slowest import time. The problem comes when AWS creates a new Lambda and it fails to respond within the timeout and the lambda instance gets removed, another starts up, same thing happens to it, etc.

Great library otherwise! Thanks in advance for anyone's advice.

Additional

No response

PawelPeczek-Roboflow commented 2 weeks ago

Hi there - this is most likely due to verification of library version against the newest release which happens at first import. Maybe you see something like that in logs:

[06/20/24 10:31:39] WARNING  Your inference package version 0.11.2 is out of date! Please  __init__.py:35
                             upgrade to version 0.12.1 of inference for the latest
                             features and bug fixes by running `pip install --upgrade
                             inference`.

To prevent version check set DISABLE_VERSION_CHECK=True

There may also be the case that we require plenty of dependent libraries and the library is big itself. We have docker image with inference server that can be hosted in AWS lambda, from our experience it works well, without timeouts as u described - so maybe that's an option to consider. If u could provide more details about deployment you created I am happy to help investigating the issue.

AJamesPhillips commented 2 weeks ago

We've removed the library and replaced it with a call to the API so we're ok now thank you. The inference-sdk version was 0.11.2 so yes it could have been the version check adding to the time. I'm not sure. I don't remember seeing that message otherwise I would have followed it so perhaps it was just getting swallowed by the logs somehow.

I'll close this now as it's no longer an active issue for us. Thank you for you help and response though. Much appreciated! :)

PawelPeczek-Roboflow commented 2 weeks ago

👍