search

milvus-io / pymilvus

Python SDK for Milvus.
Apache License 2.0
1.02k stars 325 forks source link

[QUESTION]: How to reduce pymilvus package size? #2020

Open mahaveer08 opened 7 months ago

mahaveer08 commented 7 months ago

Is there an existing issue for this?

What is your question?

I need to build my script so where I used pymilvus package and I want to install locally zip it then upload to lambda but the size of folders seems 400MB so lambda doesn’t accept more than 250MB file size so how to reduce the package size and what are the supported sub packages as it installed lot of supportive packages?

Anything else?

No response

mahaveer08 commented 7 months ago

@XuanYang-cn any inputs on this?

XuanYang-cn commented 7 months ago

@mahaveer08 From my test, pymilvus.tar.gz only uses 1.2MB, what else does your folders contain? It seems that 400MB contains not only pymilvus but many other things.

mahaveer08 commented 7 months ago

List of all supported packages like below: Pyarrow, urllib3, azure, grpc, numpy, pandas, pymilvus, setuptools etc. so these kind of packages get added when you run “pip install pymilvus -t .” While building the folder for lambda function. So only pymilvus.tar.gz is enough to support ?

mahaveer08 commented 7 months ago 
Below packages installed after running "pip install pymilvus -t ."

Successfully installed argon2-cffi-23.1.0 argon2-cffi-bindings-21.2.0 azure-core-1.30.1 azure-storage-blob-12.19.1 boto3-1.34.79 botocore-1.34.79 certifi-2024.2.2 cffi-1.16.0 charset-normalizer-3.3.2 cryptography-42.0.5 environs-9.5.0 grpcio-1.60.0 idna-3.6 isodate-0.6.1 jmespath-1.0.1 marshmallow-3.21.1 minio-7.2.5 numpy-1.26.4 packaging-24.0 pandas-2.2.1 protobuf-5.26.1 pyarrow-15.0.2 pycparser-2.22 pycryptodome-3.20.0 pymilvus-2.4.0 python-dateutil-2.9.0.post0 python-dotenv-1.0.1 pytz-2024.1 requests-2.31.0 s3transfer-0.10.1 scipy-1.13.0 setuptools-69.2.0 six-1.16.0 typing-extensions-4.11.0 tzdata-2024.1 ujson-5.9.0 urllib3-1.26.18

Let me know what are the necessary dependencies need to install from above @XuanYang-cn ?

XuanYang-cn commented 7 months ago

@mahaveer08 They are all essentials for pymilvus to run.

What do you mean when you said "build the folder for lambda function"? What's the difference to use pymilvus as lambda function to normal usage by pip install pymilvus?

mahaveer08 commented 7 months ago

I have a Python script that inserts data into a Milvus database using the pymilvus package. To utilize this script on a Lambda function, I need to compress it along with the pymilvus folder, ensuring that the folder size remains below 250MB.

mahaveer08 commented 7 months ago

Anyhow I deployed using container image rather than zipping the pymilvus package folder.