Closed kreamyu closed 1 year ago
CUDA error: CUBLAS_STATUS_NOT_INITIALIZED when calling
cublasCreate(handle)` typically happens if you're running out of GPU memory. So your options are either to get a GPU with a bigger on device memory or try out a smaller model
@msaroufim thanks for your reply,and I tried the example again
The setup_config.json
is
{
"model_name":"gpt2",
"mode":"text_generation",
"do_lower_case":true,
"num_labels":"0",
"save_mode":"pretrained",
"max_length":"50",
"captum_explanation":false,
"FasterTransformer":false,
"embedding_name": "gpt2",
"model_parallel": true,
"batch_size": 1
}
and then
rm -rf Transformer_model
python Download_Transformer_models.py
torch-model-archiver --model-name Textgeneration --version 1.0 --serialized-file Transformer_model/pytorch_model.bin --handler ./Transformer_handler_generalized.py --extra-files "Transformer_model/config.json,Transformer_model/special_tokens_map.json,Transformer_model/tokenizer_config.json,Transformer_model/tokenizer.json,Transformer_model/vocab.json,Transformer_model/merges.txt,./setup_config.json"
mv Textgeneration.mar model_store/
torchserve --start --model-store model_store --models my_tc=Textgeneration.mar --ncs
and then
curl -X POST http://127.0.0.1:8080/predictions/my_tc -T input.txt
the input.txt is
Today the weather is really nice and I am planning on
and the output is
(base) [root@server-t493yc mydeploy]# curl -X POST http://127.0.0.1:8080/predictions/my_tc -T input.txt
Today the weather is really nice and I am planning on
(base) [root@server-t493yc mydeploy]#
Predicted content is empty
the log is
(mydeploy) [root@server-t493yc Huggingface_Transformers]# WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance.
2022-11-23T03:36:45,794 [INFO ] main org.pytorch.serve.servingsdk.impl.PluginsManager - Initializing plugins manager...
2022-11-23T03:36:45,951 [INFO ] main org.pytorch.serve.ModelServer -
Torchserve version: 0.6.1
TS Home: /root/miniconda3/envs/mydeploy/lib/python3.8/site-packages
Current directory: /root/mydeploy/serve/examples/Huggingface_Transformers
Temp directory: /tmp
Metrics config path: /root/miniconda3/envs/mydeploy/lib/python3.8/site-packages/ts/configs/metrics.yaml
Number of GPUs: 1
Number of CPUs: 8
Max heap size: 8004 M
Python executable: /root/miniconda3/envs/mydeploy/bin/python
Config file: N/A
Inference address: http://127.0.0.1:8080
Management address: http://127.0.0.1:8081
Metrics address: http://127.0.0.1:8082
Model Store: /root/mydeploy/serve/examples/Huggingface_Transformers/model_store
Initial Models: my_tc=Textgeneration.mar
Log dir: /root/mydeploy/serve/examples/Huggingface_Transformers/logs
Metrics dir: /root/mydeploy/serve/examples/Huggingface_Transformers/logs
Netty threads: 0
Netty client threads: 0
Default workers per model: 1
Blacklist Regex: N/A
Maximum Response Size: 6553500
Maximum Request Size: 6553500
Limit Maximum Image Pixels: true
Prefer direct buffer: false
Allowed Urls: [file://.*|http(s)?://.*]
Custom python dependency for model allowed: false
Metrics report format: prometheus
Enable metrics API: true
Workflow Store: /root/mydeploy/serve/examples/Huggingface_Transformers/model_store
Model config: N/A
2022-11-23T03:36:45,963 [INFO ] main org.pytorch.serve.servingsdk.impl.PluginsManager - Loading snapshot serializer plugin...
2022-11-23T03:36:46,002 [INFO ] main org.pytorch.serve.ModelServer - Loading initial models: Textgeneration.mar
2022-11-23T03:36:54,303 [DEBUG] main org.pytorch.serve.wlm.ModelVersionedRefs - Adding new version 1.0 for model my_tc
2022-11-23T03:36:54,303 [DEBUG] main org.pytorch.serve.wlm.ModelVersionedRefs - Setting default version to 1.0 for model my_tc
2022-11-23T03:36:54,303 [INFO ] main org.pytorch.serve.wlm.ModelManager - Model my_tc loaded.
2022-11-23T03:36:54,303 [DEBUG] main org.pytorch.serve.wlm.ModelManager - updateModel: my_tc, count: 1
2022-11-23T03:36:54,322 [INFO ] main org.pytorch.serve.ModelServer - Initialize Inference server with: EpollServerSocketChannel.
2022-11-23T03:36:54,322 [DEBUG] W-9000-my_tc_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/root/miniconda3/envs/mydeploy/bin/python, /root/miniconda3/envs/mydeploy/lib/python3.8/site-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9000, --metrics-config, /root/miniconda3/envs/mydeploy/lib/python3.8/site-packages/ts/configs/metrics.yaml]
2022-11-23T03:36:54,466 [INFO ] main org.pytorch.serve.ModelServer - Inference API bind to: http://127.0.0.1:8080
2022-11-23T03:36:54,467 [INFO ] main org.pytorch.serve.ModelServer - Initialize Management server with: EpollServerSocketChannel.
2022-11-23T03:36:54,470 [INFO ] main org.pytorch.serve.ModelServer - Management API bind to: http://127.0.0.1:8081
2022-11-23T03:36:54,471 [INFO ] main org.pytorch.serve.ModelServer - Initialize Metrics server with: EpollServerSocketChannel.
2022-11-23T03:36:54,473 [INFO ] main org.pytorch.serve.ModelServer - Metrics API bind to: http://127.0.0.1:8082
Model server started.
2022-11-23T03:36:54,900 [WARN ] pool-3-thread-1 org.pytorch.serve.metrics.MetricCollector - worker pid is not available yet.
2022-11-23T03:36:55,476 [INFO ] W-9000-my_tc_1.0-stdout MODEL_LOG - Listening on port: /tmp/.ts.sock.9000
2022-11-23T03:36:55,479 [INFO ] W-9000-my_tc_1.0-stdout MODEL_LOG - Successfully loaded /root/miniconda3/envs/mydeploy/lib/python3.8/site-packages/ts/configs/metrics.yaml.
2022-11-23T03:36:55,480 [INFO ] W-9000-my_tc_1.0-stdout MODEL_LOG - [PID]10766
2022-11-23T03:36:55,481 [INFO ] W-9000-my_tc_1.0-stdout MODEL_LOG - Torch worker started.
2022-11-23T03:36:55,481 [INFO ] W-9000-my_tc_1.0-stdout MODEL_LOG - Python runtime: 3.8.15
2022-11-23T03:36:55,482 [DEBUG] W-9000-my_tc_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-my_tc_1.0 State change null -> WORKER_STARTED
2022-11-23T03:36:55,492 [INFO ] pool-3-thread-1 TS_METRICS - CPUUtilization.Percent:0.0|#Level:Host|#hostname:server-t493yc,timestamp:1669174615
2022-11-23T03:36:55,493 [INFO ] pool-3-thread-1 TS_METRICS - DiskAvailable.Gigabytes:108.4411392211914|#Level:Host|#hostname:server-t493yc,timestamp:1669174615
2022-11-23T03:36:55,494 [INFO ] pool-3-thread-1 TS_METRICS - DiskUsage.Gigabytes:91.54809951782227|#Level:Host|#hostname:server-t493yc,timestamp:1669174615
2022-11-23T03:36:55,494 [INFO ] pool-3-thread-1 TS_METRICS - DiskUtilization.Percent:45.8|#Level:Host|#hostname:server-t493yc,timestamp:1669174615
2022-11-23T03:36:55,494 [INFO ] pool-3-thread-1 TS_METRICS - GPUMemoryUtilization.Percent:0.0|#Level:Host,device_id:0|#hostname:server-t493yc,timestamp:1669174615
2022-11-23T03:36:55,495 [INFO ] pool-3-thread-1 TS_METRICS - GPUMemoryUsed.Megabytes:0|#Level:Host,device_id:0|#hostname:server-t493yc,timestamp:1669174615
2022-11-23T03:36:55,495 [INFO ] pool-3-thread-1 TS_METRICS - GPUUtilization.Percent:0|#Level:Host,device_id:0|#hostname:server-t493yc,timestamp:16691746152022-11-23T03:36:55,495 [INFO ] pool-3-thread-1 TS_METRICS - MemoryAvailable.Megabytes:26329.0703125|#Level:Host|#hostname:server-t493yc,timestamp:1669174615
2022-11-23T03:36:55,496 [INFO ] pool-3-thread-1 TS_METRICS - MemoryUsed.Megabytes:5228.625|#Level:Host|#hostname:server-t493yc,timestamp:1669174615
2022-11-23T03:36:55,496 [INFO ] pool-3-thread-1 TS_METRICS - MemoryUtilization.Percent:17.8|#Level:Host|#hostname:server-t493yc,timestamp:1669174615
2022-11-23T03:36:55,497 [INFO ] W-9000-my_tc_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9000
2022-11-23T03:36:55,516 [INFO ] W-9000-my_tc_1.0-stdout MODEL_LOG - Connection accepted: /tmp/.ts.sock.9000.
2022-11-23T03:36:55,523 [INFO ] W-9000-my_tc_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req. to backend at: 1669174615523
2022-11-23T03:36:55,594 [INFO ] W-9000-my_tc_1.0-stdout MODEL_LOG - model_name: my_tc, batchSize: 1
2022-11-23T03:36:56,715 [INFO ] W-9000-my_tc_1.0-stdout MODEL_LOG - Transformers version 4.6.0
2022-11-23T03:37:09,067 [INFO ] W-9000-my_tc_1.0-stdout MODEL_LOG - Transformer model from path /tmp/models/e0ca41298bd342b69291cecfa59ea6a4 loaded successfully
2022-11-23T03:37:09,070 [INFO ] W-9000-my_tc_1.0 org.pytorch.serve.wlm.WorkerThread - Backend response time: 13476
2022-11-23T03:37:09,071 [DEBUG] W-9000-my_tc_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-my_tc_1.0 State change WORKER_STARTED -> WORKER_MODEL_LOADED
2022-11-23T03:37:09,072 [INFO ] W-9000-my_tc_1.0 TS_METRICS - W-9000-my_tc_1.0.ms:14761|#Level:Host|#hostname:server-t493yc,timestamp:1669174629
2022-11-23T03:37:09,073 [INFO ] W-9000-my_tc_1.0 TS_METRICS - WorkerThreadTime.ms:74|#Level:Host|#hostname:server-t493yc,timestamp:1669174629
2022-11-23T03:37:09,074 [INFO ] W-9000-my_tc_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req. to backend at: 1669174629074
2022-11-23T03:37:09,155 [INFO ] W-9000-my_tc_1.0-stdout MODEL_LOG - Backend received inference at: 1669174629
2022-11-23T03:37:09,157 [INFO ] W-9000-my_tc_1.0-stdout MODEL_LOG - Received text: 'Today the weather is really nice and I am planning on
2022-11-23T03:37:09,156 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - Truncation was not explicitly activated but `max_length` is provided a specific value, please use `truncation=True` to explicitly truncate examples to max length. Defaulting to 'longest_first' truncation strategy. If you encode pairs of sequences (GLUE-style) with the tokenizer you can select this strategy more precisely by providing a specific strategy to `truncation`.
2022-11-23T03:37:09,157 [INFO ] W-9000-my_tc_1.0-stdout MODEL_LOG - '
2022-11-23T03:37:09,161 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - /root/miniconda3/envs/mydeploy/lib/python3.8/site-packages/transformers/tokenization_utils_base.py:2104: FutureWarning: The `pad_to_max_length` argument is deprecated and will be removed in a future version, use `padding=True` or `padding='longest'` to pad to the longest sequence in the batch, or use `padding='max_length'` to pad to a max length. In this case, you can give a specific length with `max_length` (e.g. `max_length=45`) or leave max_length to None to pad to the maximal input size of the model (e.g. 512 for Bert).
2022-11-23T03:37:09,162 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - warnings.warn(
2022-11-23T03:37:09,162 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.
2022-11-23T03:37:09,162 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - Input length of input_ids is 50, but ``max_length`` is set to 50.This can lead to unexpected behavior. You should consider increasing ``config.max_length`` or ``max_length``.
2022-11-23T03:37:09,865 [INFO ] W-9000-my_tc_1.0-stdout MODEL_LOG - Generated text: '['Today the weather is really nice and I am planning on\nIn']'
2022-11-23T03:37:09,865 [INFO ] W-9000-my_tc_1.0 org.pytorch.serve.wlm.WorkerThread - Backend response time: 789
2022-11-23T03:37:09,866 [INFO ] W-9000-my_tc_1.0-stdout MODEL_LOG - Generated text ['Today the weather is really nice and I am planning on\nIn']
2022-11-23T03:37:09,867 [INFO ] W-9000-my_tc_1.0 ACCESS_LOG - /127.0.0.1:57290 "POST /predictions/my_tc HTTP/1.1" 200 3031
2022-11-23T03:37:09,866 [INFO ] W-9000-my_tc_1.0-stdout MODEL_METRICS - HandlerTime.Milliseconds:708.87|#ModelName:my_tc,Level:Model|#hostname:server-t493yc,requestID:036f3482-7533-478b-8dd7-ed722a255178,timestamp:1669174629
2022-11-23T03:37:09,867 [INFO ] W-9000-my_tc_1.0 TS_METRICS - Requests2XX.Count:1|#Level:Host|#hostname:server-t493yc,timestamp:1669174626
2022-11-23T03:37:09,868 [DEBUG] W-9000-my_tc_1.0 org.pytorch.serve.job.Job - Waiting time ns: 2225885894, Backend time ns: 793466486
2022-11-23T03:37:09,868 [INFO ] W-9000-my_tc_1.0 TS_METRICS - QueueTime.ms:2225|#Level:Host|#hostname:server-t493yc,timestamp:1669174629
2022-11-23T03:37:09,868 [INFO ] W-9000-my_tc_1.0-stdout MODEL_METRICS - PredictionTime.Milliseconds:709.02|#ModelName:my_tc,Level:Model|#hostname:server-t493yc,requestID:036f3482-7533-478b-8dd7-ed722a255178,timestamp:1669174629
2022-11-23T03:37:09,868 [INFO ] W-9000-my_tc_1.0 TS_METRICS - WorkerThreadTime.ms:5|#Level:Host|#hostname:server-t493yc,timestamp:1669174629
Is this a error please?
Looks like it's working
Looks like it's working
Thanks for your helpļ¼
š Describe the bug
I want to deploy this model on cent os 7 server using torchserve. and I follow the official example of deploying gpt2 to deploy.
firstly, I change
setup_config.json
toThen
Then run an inference
and input.txt is
finally
And the error is
and I tried to set batch_size to 1, but no use.
Error logs
Installation instructions
no no
Model Packaing
follow the official example of deploying gpt2
config.properties
No response
Versions
Environment headers
Torchserve branch:
torchserve==0.6.1b20221114 torch-model-archiver==0.6.1b20221114
Python version: 3.8 (64-bit runtime) Python executable: /root/miniconda3/envs/mydeploy/bin/python
Versions of relevant python libraries: captum==0.5.0 future==0.18.2 numpy==1.23.5 nvgpu==0.9.0 psutil==5.9.4 requests==2.28.1 torch==1.12.0+cu116 torch-model-archiver==0.6.1b20221114 torch-workflow-archiver==0.2.5b20221114 torchaudio==0.12.0+cu116 torchserve==0.6.1b20221114 torchtext==0.13.0 torchvision==0.13.0+cu116 transformers==4.6.0 wheel==0.38.4 torch==1.12.0+cu116 torchtext==0.13.0 torchvision==0.13.0+cu116 torchaudio==0.12.0+cu116
Java Version:
OS: N/A GCC version: (GCC) 4.8.5 20150623 (Red Hat 4.8.5-44) Clang version: N/A CMake version: N/A
Is CUDA available: Yes CUDA runtime version: 11.6.124 GPU models and configuration: GPU 0: Tesla V100-PCIE-32GB Nvidia driver version: 510.85.02 cuDNN version: None
Repro instructions
as in the description
Possible Solution
No response