search

pytorch / serve

Serve, optimize and scale PyTorch models in production
https://pytorch.org/serve/
Apache License 2.0
4.04k stars 821 forks source link

ImportError: cannot import name 'packaging' from 'pkg_resources' #3176

Open Sieltek opened 3 weeks ago

Sieltek commented 3 weeks ago

🐛 Describe the bug

Problem seems to arise when loading a model using install_py_dep_per_model=true which will upgrade the setuptools version to latest 70.0 and thus breaking the handler loading.

Solution might be to replace the deprecated import in base handler from pkg_resources import packaging by import packaging

Error logs

2024-06-05T10:24:37,046 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - Backend worker process died. 2024-06-05T10:24:37,047 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - Traceback (most recent call last): 2024-06-05T10:24:37,047 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - File "/home/torch/.local/lib/python3.11/site-packages/ts/model_loader.py", line 108, in load 2024-06-05T10:24:37,047 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - module, function_name = self._load_handler_file(handler) 2024-06-05T10:24:37,047 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-06-05T10:24:37,047 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - File "/home/torch/.local/lib/python3.11/site-packages/ts/model_loader.py", line 153, in _load_handler_file 2024-06-05T10:24:37,047 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - module = importlib.import_module(module_name) 2024-06-05T10:24:37,047 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-06-05T10:24:37,048 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - File "/usr/lib64/python3.11/importlib/init.py", line 126, in import_module 2024-06-05T10:24:37,048 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - return _bootstrap._gcd_import(name[level:], package, level) 2024-06-05T10:24:37,048 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-06-05T10:24:37,048 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - File "", line 1206, in _gcd_import 2024-06-05T10:24:37,048 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - File "", line 1178, in _find_and_load 2024-06-05T10:24:37,048 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - File "", line 1149, in _find_and_load_unlocked 2024-06-05T10:24:37,048 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - File "", line 690, in _load_unlocked 2024-06-05T10:24:37,048 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - File "", line 940, in exec_module 2024-06-05T10:24:37,048 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - File "", line 241, in _call_with_frames_removed 2024-06-05T10:24:37,049 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - File "/tmp/models/9a75af565f7e478ebcf169da7ae7cbc0/handler.py", line 1, in 2024-06-05T10:24:37,049 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - from handler_yolo import YoloHandler 2024-06-05T10:24:37,049 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - File "/tmp/models/9a75af565f7e478ebcf169da7ae7cbc0/handler_yolo.py", line 6, in 2024-06-05T10:24:37,049 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - from ts.torch_handler.object_detector import ObjectDetector 2024-06-05T10:24:37,049 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - File "/home/torch/.local/lib/python3.11/site-packages/ts/torch_handler/object_detector.py", line 8, in 2024-06-05T10:24:37,049 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - from .vision_handler import VisionHandler 2024-06-05T10:24:37,049 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - File "/home/torch/.local/lib/python3.11/site-packages/ts/torch_handler/vision_handler.py", line 16, in 2024-06-05T10:24:37,049 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - from .base_handler import BaseHandler 2024-06-05T10:24:37,050 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - File "/home/torch/.local/lib/python3.11/site-packages/ts/torch_handler/base_handler.py", line 13, in 2024-06-05T10:24:37,050 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - from pkg_resources import packaging 2024-06-05T10:24:37,050 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - ImportError: cannot import name 'packaging' from 'pkg_resources' (/tmp/models/9a75af565f7e478ebcf169da7ae7cbc0/pkg_resources/init.py) 2024-06-05T10:24:37,050 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - 2024-06-05T10:24:37,050 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - During handling of the above exception, another exception occurred: 2024-06-05T10:24:37,050 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - 2024-06-05T10:24:37,050 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - Traceback (most recent call last): 2024-06-05T10:24:37,050 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - File "/home/torch/.local/lib/python3.11/site-packages/ts/model_service_worker.py", line 263, in 2024-06-05T10:24:37,050 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - worker.run_server() 2024-06-05T10:24:37,050 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - File "/home/torch/.local/lib/python3.11/site-packages/ts/model_service_worker.py", line 231, in run_server 2024-06-05T10:24:37,051 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - self.handle_connection(cl_socket) 2024-06-05T10:24:37,051 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - File "/home/torch/.local/lib/python3.11/site-packages/ts/model_service_worker.py", line 194, in handle_connection 2024-06-05T10:24:37,051 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - service, result, code = self.load_model(msg) 2024-06-05T10:24:37,051 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^ 2024-06-05T10:24:37,051 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - File "/home/torch/.local/lib/python3.11/site-packages/ts/model_service_worker.py", line 131, in load_model 2024-06-05T10:24:37,051 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - service = model_loader.load( 2024-06-05T10:24:37,051 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^ 2024-06-05T10:24:37,051 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - File "/home/torch/.local/lib/python3.11/site-packages/ts/model_loader.py", line 110, in load 2024-06-05T10:24:37,051 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - module = self._load_default_handler(handler) 2024-06-05T10:24:37,051 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-06-05T10:24:37,052 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - File "/home/torch/.local/lib/python3.11/site-packages/ts/model_loader.py", line 159, in _load_default_handler 2024-06-05T10:24:37,052 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - module = importlib.import_module(module_name, "ts.torch_handler") 2024-06-05T10:24:37,052 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-06-05T10:24:37,052 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - File "/usr/lib64/python3.11/importlib/init.py", line 126, in import_module 2024-06-05T10:24:37,052 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - return _bootstrap._gcd_import(name[level:], package, level) 2024-06-05T10:24:37,052 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-06-05T10:24:37,052 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - File "", line 1206, in _gcd_import 2024-06-05T10:24:37,052 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - File "", line 1178, in _find_and_load 2024-06-05T10:24:37,052 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - File "", line 1128, in _find_and_load_unlocked 2024-06-05T10:24:37,052 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - File "", line 241, in _call_with_frames_removed 2024-06-05T10:24:37,053 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - File "", line 1206, in _gcd_import 2024-06-05T10:24:37,053 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - File "", line 1178, in _find_and_load 2024-06-05T10:24:37,053 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - File "", line 1142, in _find_and_load_unlocked 2024-06-05T10:24:37,053 [INFO ] W-9000-crochet_yolov8s_512_240207_1.0-stdout MODEL_LOG - ModuleNotFoundError: No module named 'ts.torch_handler.handler' 2024-06-05T10:24:37,061 [INFO ] epollEventLoopGroup-5-1 org.pytorch.serve.wlm.WorkerThread - 9000 Worker disconnected. WORKER_STARTED 2024-06-05T10:24:37,062 [DEBUG] W-9000-crochet_yolov8s_512_240207_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED 2024-06-05T10:24:37,062 [DEBUG] W-9000-crochet_yolov8s_512_240207_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died., responseTimeout:120sec

Installation instructions

python3.11 -m pip install --trusted-host pypi.org --trusted-host pypi.python.org --trusted-host files.pythonhosted.org pip==24.0 setuptools==69.5.1

python3.11 -m pip install --trusted-host pypi.org --trusted-host pypi.python.org --trusted-host files.pythonhosted.org --trusted-host download.pytorch.org --no-cache-dir numpy==1.24.3 cython==0.29.34 wheel==0.40.0 pillow==9.3.0 psutil==5.9.5 requests==2.31.0 captum==0.6.0 packaging==23.1 pynvml==11.4.1 pyyaml==6.0

python3.11 -m pip install --trusted-host pypi.org --trusted-host pypi.python.org --trusted-host files.pythonhosted.org --trusted-host download.pytorch.org --no-cache-dir torch==2.3.0+cpu torchvision==0.18.0+cpu torchtext==0.18.0 torchaudio==2.3.0+cpu --extra-index-url https://download.pytorch.org/whl/cpu

python3.11 -m pip install --trusted-host pypi.org --trusted-host pypi.python.org --trusted-host files.pythonhosted.org --trusted-host download.pytorch.org --no-cache-dir torchserve==0.11.0 torch-model-archiver==0.11.0 torch-workflow-archiver==0.2.13

mkdir /model-server mkdir /model-server/model-store mkdir /model-server/wf-store

Model Packaging

torch-model-archiver --model-name my_model_name --handler handler.py --extra-files index_to_name.json --serialized-file best.pt -r requirements.txt -v 1.0 --force

crash:

requirements.txt: numpy

workaround:

requirements.txt: numpy setuptools==69.5.1

loader failing at

serve/ts/torch_handler/base_handler.py l.13 from pkg_resources import packaging

config.properties

echo " inference_address=http://0.0.0.0:8080 management_address=http://0.0.0.0:8081 metrics_address=http://0.0.0.0:8082 grpc_inference_port=7070 grpc_management_port=7071 cors_allowed_origin=* cors_allowed_methods=GET, POST, PUT, DELETE, OPTIONS default_workers_per_model=2 job_queue_size=1000 model_store=/model-server/model-store workflow_store=/model-server/wf-store load_models=all prefer_direct_buffer=true install_py_dep_per_model=true enable_metrics_api=true prefer_direct_buffer=true disable_system_metrics=false metrics_mode=prometheus cpu_launcher_enable=true cpu_launcher_args=--use_logical_core " > /model-server/config.properties

Versions

Environment headers

Torchserve branch:

torchserve==0.11.0 torch-model-archiver==0.11.0

Python version: 3.11 (64-bit runtime) Python executable: /usr/bin/python3

Versions of relevant python libraries: captum==0.6.0 numpy==1.24.3 psutil==5.9.5 requests==2.31.0 torch==2.3.0+cpu torch-model-archiver==0.11.0 torch-workflow-archiver==0.2.13 torchaudio==2.3.0+cpu torchserve==0.11.0 torchtext==0.18.0+cpu torchvision==0.18.0+cpu wheel==0.40.0 torch==2.3.0+cpu torchtext==0.18.0+cpu torchvision==0.18.0+cpu torchaudio==2.3.0+cpu

Java Version:

OS: N/A GCC version: (GCC) 8.5.0 20210514 (Red Hat 8.5.0-18) Clang version: N/A CMake version: N/A

Environment: librarypath (LD/DYLD_):

Repro instructions

torchserve --start --ts-config /model-server/config.properties

Possible Solution

Specify setuptool version in model.mar requirement.txt

workaround:

requirements.txt: numpy setuptools==69.5.1

Lix1993 commented 2 weeks ago

python -m pip install setuptools==69.5.1

RainJayTsai commented 2 weeks ago

Exporting export PIP_NO_DEPS=true in the environment is also a workaround

If running in a Docker environment, you can use the following command:

docker run -e PIP_NO_CACHE_DIR=true -e PIP_NO_DEPS=true ...

This sets PIP_NO_CACHE_DIR=true and PIP_NO_DEPS=true as environment variables within the Docker container.

BUT If you choose to run with PIP_NO_CACHE_DIR=true and PIP_NO_DEPS=true in environment, you need to manage the dependencies manually.

jtharrell2023 commented 4 days ago

Also, if you get this or similar when trying to host Whisper, notice the comments in the overview regarding setuptools-rust.

udaij12 commented 4 days ago

#3187 Solves this by changing the import to import packaging.version which eliminates the need to pin setuptools