msaroufim
commented
2 years ago I see two different issues
Unsuccessful NVIDIA driver installation
AttributeError: /lib64/libnvidia-ml.so.1: undefined symbol: nvmlDeviceGetComputeRunningProcesses_v2
So would suggest trying to run a simpler example with just some PyTorch model on GPU and seeing if it works
Importing a function that does not exist
I also see 2022-06-19T20:37:26,893 [INFO ] W-9001-my_tc_1.0-stdout MODEL_LOG - ModuleNotFoundError: No module named 'ts.torch_handler.Transformer_handler_generalized' which is expected at examples/Huggingface_Transformers/Transformer_handler_generalized.py but is not ideal and I agree that we should make our handlers easier to import.
TODO for us: Make all handlers in examples possible to import like so import ts.torch_handler.Transformer
rbavery
choshiho
🐛 Describe the bug
When I ran the official demo Serving Huggingface Transformers using TorchServe - Sequence Classification, I got the error logs as follows.
Error logs
Installation instructions
JDK11 pip install transformers==4.6.0 TorchServe
Install dependencies
cuda is optional
python ./ts_scripts/install_dependencies.py --cuda=cu111
Latest release
pip install torchserve torch-model-archiver torch-workflow-archiver
Model Packaing
downloaded pre-trained models python Download_Transformer_models.py
config.properties
inference_address=http://127.0.0.1:8083
Versions
Environment headers
Torchserve branch:
torchserve==0.6.0 torch-model-archiver==0.6.0
Python version: 3.9 (64-bit runtime) Python executable: /root/anaconda3/bin/python
Versions of relevant python libraries: captum==0.5.0 future==0.18.2 numpy==1.22.4 numpydoc==1.1.0 nvgpu==0.9.0 psutil==5.9.1 pylint==2.9.6 pytest==6.2.4 pytorch-crf==0.7.2 requests==2.28.0 requests-oauthlib==1.3.0 torch==1.9.0+cu111 torch-model-archiver==0.6.0 torch-workflow-archiver==0.2.4 torchaudio==0.9.0 torchserve==0.6.0 torchtext==0.10.0 torchvision==0.10.0+cu111 transformers==4.20.0.dev0 wheel==0.37.1 torch==1.9.0+cu111 torchtext==0.10.0 torchvision==0.10.0+cu111 torchaudio==0.9.0
Java Version:
OS: N/A GCC version: (GCC) 4.8.5 20150623 (Red Hat 4.8.5-44) Clang version: N/A CMake version: version 3.14.2
Repro instructions
torch-model-archiver --model-name BERTSeqClassification --version 1.0 --serialized-file Transformer_model/pytorch_model.bin --handler ./Transformer_handler_generalized.py --extra-files "Transformer_model/config.json,./setup_config.json,./Seq_classification_artifacts/index_to_name.json"
mkdir model_store
mv BERTSeqClassification.mar model_store/
torchserve --start --model-store model_store --models my_tc=BERTSeqClassification.mar --ncs --ts-config config.properties
Possible Solution
No response