search

pytorch / serve

Serve, optimize and scale PyTorch models in production
https://pytorch.org/serve/
Apache License 2.0
4.13k stars 832 forks source link

Deploy GPT2-Chineseļ¼šRuntimeError: CUDA error: CUBLAS_STATUS_NOT_INITIALIZED when calling `cublasCreate(handle)` #1993

Closed kreamyu closed 1 year ago

kreamyu commented 1 year ago

šŸ› Describe the bug

I want to deploy this model on cent os 7 server using torchserve. and I follow the official example of deploying gpt2 to deploy.

firstly, I change setup_config.json to

{
 "model_name":"uer/gpt2-chinese-cluecorpussmall",
 "mode":"text_generation",
 "do_lower_case":true,
 "num_labels":"0",
 "save_mode":"pretrained",
 "max_length":"50",
 "captum_explanation":false,
 "FasterTransformer":false,
 "embedding_name": "gpt2",
 "model_parallel": true,
 "batch_size": 1
}

Then

rm -rf Transformer_model

torch-model-archiver --model-name Textgeneration --version 1.0 --serialized-file Transformer_model/pytorch_model.bin --handler ./Transformer_handler_generalized.py --extra-files "Transformer_model/config.json,Transformer_model/special_tokens_map.json,Transformer_model/tokenizer.json,Transformer_model/tokenizer.json,Transformer_model/vocab.txt,./setup_config.json"

mv Textgeneration.mar model_store/

torchserve --start --model-store model_store --models my_tc=Textgeneration.mar --ncs

Then run an inference

curl -X POST http://127.0.0.1:8080/predictions/my_tc -T input.txt

and input.txt is 

你儽

finally

(base) [root@server-t493yc mydeploy]# curl -X POST http://127.0.0.1:8080/predictions/my_tc -T input.txt
{
  "code": 503,
  "type": "InternalServerException",
  "message": "Prediction failed"
}

And the error is

2022-11-22T15:40:53,970 [INFO ] W-9000-my_tc_1.0-stdout MODEL_LOG -     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2022-11-22T15:40:53,970 [INFO ] W-9000-my_tc_1.0-stdout MODEL_LOG - RuntimeError: CUDA error: CUBLAS_STATUS_NOT_INITIALIZED when calling `cublasCreate(handle)`

and I tried to set batch_size to 1, but no use.

Error logs

(mydeploy) [root@server-t493yc Huggingface_Transformers]# torchserve --start --model-store model_store --models my_tc=Textgeneration.mar --ncs
(mydeploy) [root@server-t493yc Huggingface_Transformers]# WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance.
2022-11-22T15:40:27,921 [INFO ] main org.pytorch.serve.servingsdk.impl.PluginsManager - Initializing plugins manager...
2022-11-22T15:40:28,050 [INFO ] main org.pytorch.serve.ModelServer - 
Torchserve version: 0.6.1
TS Home: /root/miniconda3/envs/mydeploy/lib/python3.8/site-packages
Current directory: /root/mydeploy/serve/examples/Huggingface_Transformers
Temp directory: /tmp
Metrics config path: /root/miniconda3/envs/mydeploy/lib/python3.8/site-packages/ts/configs/metrics.yaml
Number of GPUs: 1
Number of CPUs: 8
Max heap size: 8004 M
Python executable: /root/miniconda3/envs/mydeploy/bin/python
Config file: N/A
Inference address: http://127.0.0.1:8080
Management address: http://127.0.0.1:8081
Metrics address: http://127.0.0.1:8082
Model Store: /root/mydeploy/serve/examples/Huggingface_Transformers/model_store
Initial Models: my_tc=Textgeneration.mar
Log dir: /root/mydeploy/serve/examples/Huggingface_Transformers/logs
Metrics dir: /root/mydeploy/serve/examples/Huggingface_Transformers/logs
Netty threads: 0
Netty client threads: 0
Default workers per model: 1
Blacklist Regex: N/A
Maximum Response Size: 6553500
Maximum Request Size: 6553500
Limit Maximum Image Pixels: true
Prefer direct buffer: false
Allowed Urls: [file://.*|http(s)?://.*]
Custom python dependency for model allowed: false
Metrics report format: prometheus
Enable metrics API: true
Workflow Store: /root/mydeploy/serve/examples/Huggingface_Transformers/model_store
Model config: N/A
2022-11-22T15:40:28,059 [INFO ] main org.pytorch.serve.servingsdk.impl.PluginsManager -  Loading snapshot serializer plugin...
2022-11-22T15:40:28,086 [INFO ] main org.pytorch.serve.ModelServer - Loading initial models: Textgeneration.mar
2022-11-22T15:40:34,971 [DEBUG] main org.pytorch.serve.wlm.ModelVersionedRefs - Adding new version 1.0 for model my_tc
2022-11-22T15:40:34,971 [DEBUG] main org.pytorch.serve.wlm.ModelVersionedRefs - Setting default version to 1.0 for model my_tc
2022-11-22T15:40:34,972 [INFO ] main org.pytorch.serve.wlm.ModelManager - Model my_tc loaded.
2022-11-22T15:40:34,972 [DEBUG] main org.pytorch.serve.wlm.ModelManager - updateModel: my_tc, count: 1
2022-11-22T15:40:34,987 [DEBUG] W-9000-my_tc_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/root/miniconda3/envs/mydeploy/bin/python, /root/miniconda3/envs/mydeploy/lib/python3.8/site-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9000, --metrics-config, /root/miniconda3/envs/mydeploy/lib/python3.8/site-packages/ts/configs/metrics.yaml]
2022-11-22T15:40:34,992 [INFO ] main org.pytorch.serve.ModelServer - Initialize Inference server with: EpollServerSocketChannel.
2022-11-22T15:40:35,160 [INFO ] main org.pytorch.serve.ModelServer - Inference API bind to: http://127.0.0.1:8080
2022-11-22T15:40:35,160 [INFO ] main org.pytorch.serve.ModelServer - Initialize Management server with: EpollServerSocketChannel.
2022-11-22T15:40:35,163 [INFO ] main org.pytorch.serve.ModelServer - Management API bind to: http://127.0.0.1:8081
2022-11-22T15:40:35,164 [INFO ] main org.pytorch.serve.ModelServer - Initialize Metrics server with: EpollServerSocketChannel.
2022-11-22T15:40:35,167 [INFO ] main org.pytorch.serve.ModelServer - Metrics API bind to: http://127.0.0.1:8082
Model server started.
2022-11-22T15:40:35,460 [WARN ] pool-3-thread-1 org.pytorch.serve.metrics.MetricCollector - worker pid is not available yet.
2022-11-22T15:40:35,951 [INFO ] pool-3-thread-1 TS_METRICS - CPUUtilization.Percent:100.0|#Level:Host|#hostname:server-t493yc,timestamp:1669131635
2022-11-22T15:40:35,951 [INFO ] pool-3-thread-1 TS_METRICS - DiskAvailable.Gigabytes:109.04792022705078|#Level:Host|#hostname:server-t493yc,timestamp:1669131635
2022-11-22T15:40:35,952 [INFO ] pool-3-thread-1 TS_METRICS - DiskUsage.Gigabytes:90.94131851196289|#Level:Host|#hostname:server-t493yc,timestamp:1669131635
2022-11-22T15:40:35,952 [INFO ] pool-3-thread-1 TS_METRICS - DiskUtilization.Percent:45.5|#Level:Host|#hostname:server-t493yc,timestamp:1669131635
2022-11-22T15:40:35,952 [INFO ] pool-3-thread-1 TS_METRICS - GPUMemoryUtilization.Percent:0.0|#Level:Host,device_id:0|#hostname:server-t493yc,timestamp:1669131635
2022-11-22T15:40:35,952 [INFO ] pool-3-thread-1 TS_METRICS - GPUMemoryUsed.Megabytes:0|#Level:Host,device_id:0|#hostname:server-t493yc,timestamp:1669131635
2022-11-22T15:40:35,952 [INFO ] pool-3-thread-1 TS_METRICS - GPUUtilization.Percent:0|#Level:Host,device_id:0|#hostname:server-t493yc,timestamp:16691316352022-11-22T15:40:35,952 [INFO ] pool-3-thread-1 TS_METRICS - MemoryAvailable.Megabytes:26469.1796875|#Level:Host|#hostname:server-t493yc,timestamp:1669131635
2022-11-22T15:40:35,953 [INFO ] pool-3-thread-1 TS_METRICS - MemoryUsed.Megabytes:5088.515625|#Level:Host|#hostname:server-t493yc,timestamp:1669131635
2022-11-22T15:40:35,953 [INFO ] pool-3-thread-1 TS_METRICS - MemoryUtilization.Percent:17.3|#Level:Host|#hostname:server-t493yc,timestamp:1669131635
2022-11-22T15:40:36,234 [INFO ] W-9000-my_tc_1.0-stdout MODEL_LOG - Listening on port: /tmp/.ts.sock.9000
2022-11-22T15:40:36,244 [INFO ] W-9000-my_tc_1.0-stdout MODEL_LOG - Successfully loaded /root/miniconda3/envs/mydeploy/lib/python3.8/site-packages/ts/configs/metrics.yaml.
2022-11-22T15:40:36,245 [INFO ] W-9000-my_tc_1.0-stdout MODEL_LOG - [PID]1498
2022-11-22T15:40:36,246 [INFO ] W-9000-my_tc_1.0-stdout MODEL_LOG - Torch worker started.
2022-11-22T15:40:36,247 [INFO ] W-9000-my_tc_1.0-stdout MODEL_LOG - Python runtime: 3.8.15
2022-11-22T15:40:36,247 [DEBUG] W-9000-my_tc_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-my_tc_1.0 State change null -> WORKER_STARTED
2022-11-22T15:40:36,260 [INFO ] W-9000-my_tc_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9000
2022-11-22T15:40:36,282 [INFO ] W-9000-my_tc_1.0-stdout MODEL_LOG - Connection accepted: /tmp/.ts.sock.9000.
2022-11-22T15:40:36,290 [INFO ] W-9000-my_tc_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req. to backend at: 1669131636290
2022-11-22T15:40:36,367 [INFO ] W-9000-my_tc_1.0-stdout MODEL_LOG - model_name: my_tc, batchSize: 1
2022-11-22T15:40:37,536 [INFO ] W-9000-my_tc_1.0-stdout MODEL_LOG - Transformers version 4.6.0
2022-11-22T15:40:53,749 [INFO ] W-9000-my_tc_1.0-stdout MODEL_LOG - Transformer model from path /tmp/models/087d3a3f58404ab7b992aea4a4027088 loaded successfully
2022-11-22T15:40:53,752 [INFO ] W-9000-my_tc_1.0 org.pytorch.serve.wlm.WorkerThread - Backend response time: 17386
2022-11-22T15:40:53,754 [DEBUG] W-9000-my_tc_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-my_tc_1.0 State change WORKER_STARTED -> WORKER_MODEL_LOADED
2022-11-22T15:40:53,754 [INFO ] W-9000-my_tc_1.0 TS_METRICS - W-9000-my_tc_1.0.ms:18775|#Level:Host|#hostname:server-t493yc,timestamp:1669131653
2022-11-22T15:40:53,755 [INFO ] W-9000-my_tc_1.0 TS_METRICS - WorkerThreadTime.ms:79|#Level:Host|#hostname:server-t493yc,timestamp:1669131653
2022-11-22T15:40:53,756 [INFO ] W-9000-my_tc_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req. to backend at: 1669131653756
2022-11-22T15:40:53,816 [INFO ] W-9000-my_tc_1.0-stdout MODEL_LOG - Backend received inference at: 1669131653
2022-11-22T15:40:53,818 [INFO ] W-9000-my_tc_1.0-stdout MODEL_LOG - Received text: '你儽
2022-11-22T15:40:53,817 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - Truncation was not explicitly activated but `max_length` is provided a specific value, please use `truncation=True` to explicitly truncate examples to max length. Defaulting to 'longest_first' truncation strategy. If you encode pairs of sequences (GLUE-style) with the tokenizer you can select this strategy more precisely by providing a specific strategy to `truncation`.
2022-11-22T15:40:53,819 [INFO ] W-9000-my_tc_1.0-stdout MODEL_LOG - '
2022-11-22T15:40:53,823 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - /root/miniconda3/envs/mydeploy/lib/python3.8/site-packages/transformers/tokenization_utils_base.py:2104: FutureWarning: The `pad_to_max_length` argument is deprecated and will be removed in a future version, use `padding=True` or `padding='longest'` to pad to the longest sequence in the batch, or use `padding='max_length'` to pad to a max length. In this case, you can give a specific length with `max_length` (e.g. `max_length=45`) or leave max_length to None to pad to the maximal input size of the model (e.g. 512 for Bert).
2022-11-22T15:40:53,824 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG -   warnings.warn(
2022-11-22T15:40:53,824 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.
2022-11-22T15:40:53,825 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - Input length of input_ids is 50, but ``max_length`` is set to 50.This can lead to unexpected behavior. You should consider increasing ``config.max_length`` or ``max_length``.
2022-11-22T15:40:53,913 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [85,0,0], thread: [0,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,916 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [85,0,0], thread: [1,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,916 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [85,0,0], thread: [2,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,917 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [85,0,0], thread: [3,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,917 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [85,0,0], thread: [4,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,918 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [85,0,0], thread: [5,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,919 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [85,0,0], thread: [6,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,919 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [85,0,0], thread: [7,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,919 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [85,0,0], thread: [8,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,920 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [85,0,0], thread: [9,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,920 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [85,0,0], thread: [10,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,921 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [85,0,0], thread: [11,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,921 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [85,0,0], thread: [12,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,922 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [85,0,0], thread: [13,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,923 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [85,0,0], thread: [14,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,924 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [85,0,0], thread: [15,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,925 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [85,0,0], thread: [16,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,925 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [85,0,0], thread: [17,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,927 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [85,0,0], thread: [18,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,927 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [85,0,0], thread: [19,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,928 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [85,0,0], thread: [20,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,928 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [85,0,0], thread: [21,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,928 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [85,0,0], thread: [22,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,929 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [85,0,0], thread: [23,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,929 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [85,0,0], thread: [24,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,929 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [85,0,0], thread: [25,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,930 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [85,0,0], thread: [26,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,930 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [85,0,0], thread: [27,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,930 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [85,0,0], thread: [28,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,931 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [85,0,0], thread: [29,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,931 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [85,0,0], thread: [30,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,931 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [85,0,0], thread: [31,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,931 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [85,0,0], thread: [32,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,932 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [85,0,0], thread: [33,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,932 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [85,0,0], thread: [34,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,932 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [85,0,0], thread: [35,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,933 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [85,0,0], thread: [36,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,933 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [85,0,0], thread: [37,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,934 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [85,0,0], thread: [38,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,934 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [85,0,0], thread: [39,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,934 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [85,0,0], thread: [40,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,934 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [85,0,0], thread: [41,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,935 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [85,0,0], thread: [42,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,935 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [85,0,0], thread: [43,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,935 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [85,0,0], thread: [44,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,936 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [85,0,0], thread: [45,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,936 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [85,0,0], thread: [46,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,936 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [85,0,0], thread: [47,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,937 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [85,0,0], thread: [48,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,937 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [85,0,0], thread: [49,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,937 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [85,0,0], thread: [50,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,937 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [85,0,0], thread: [51,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,938 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [85,0,0], thread: [52,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,938 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [85,0,0], thread: [53,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,938 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [85,0,0], thread: [54,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,938 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [85,0,0], thread: [55,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,939 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [85,0,0], thread: [56,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,939 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [85,0,0], thread: [57,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,939 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [85,0,0], thread: [58,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,940 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [85,0,0], thread: [59,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,940 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [85,0,0], thread: [60,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,940 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [85,0,0], thread: [61,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,941 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [85,0,0], thread: [62,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,941 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [85,0,0], thread: [63,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,941 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [165,0,0], thread: [32,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,942 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [165,0,0], thread: [33,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,942 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [165,0,0], thread: [34,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,942 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [165,0,0], thread: [35,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,942 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [165,0,0], thread: [36,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,942 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [165,0,0], thread: [37,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,943 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [165,0,0], thread: [38,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,943 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [165,0,0], thread: [39,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,944 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [165,0,0], thread: [40,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,944 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [165,0,0], thread: [41,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,944 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [165,0,0], thread: [42,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,944 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [165,0,0], thread: [43,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,945 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [165,0,0], thread: [44,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,945 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [165,0,0], thread: [45,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,946 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [165,0,0], thread: [46,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,946 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [165,0,0], thread: [47,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,946 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [165,0,0], thread: [48,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,946 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [165,0,0], thread: [49,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,947 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [165,0,0], thread: [50,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,947 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [165,0,0], thread: [51,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,947 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [165,0,0], thread: [52,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,947 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [165,0,0], thread: [53,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,948 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [165,0,0], thread: [54,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,948 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [165,0,0], thread: [55,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,948 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [165,0,0], thread: [56,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,948 [INFO ] W-9000-my_tc_1.0-stdout MODEL_LOG - Invoking custom service failed.
2022-11-22T15:40:53,948 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [165,0,0], thread: [57,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,949 [INFO ] W-9000-my_tc_1.0-stdout MODEL_LOG - Traceback (most recent call last):
2022-11-22T15:40:53,949 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [165,0,0], thread: [58,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,949 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [165,0,0], thread: [59,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,949 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [165,0,0], thread: [60,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,949 [INFO ] W-9000-my_tc_1.0-stdout MODEL_LOG -   File "/root/miniconda3/envs/mydeploy/lib/python3.8/site-packages/ts/service.py", line 120, in predict
2022-11-22T15:40:53,949 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [165,0,0], thread: [61,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,950 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [165,0,0], thread: [62,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,950 [INFO ] W-9000-my_tc_1.0-stdout MODEL_LOG -     ret = self._entry_point(input_batch, self.context)
2022-11-22T15:40:53,950 [INFO ] W-9000-my_tc_1.0 org.pytorch.serve.wlm.WorkerThread - Backend response time: 191
2022-11-22T15:40:53,950 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [165,0,0], thread: [63,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,951 [INFO ] W-9000-my_tc_1.0-stdout MODEL_LOG -   File "/root/miniconda3/envs/mydeploy/lib/python3.8/site-packages/ts/torch_handler/base_handler.py", line 282, in handle
2022-11-22T15:40:53,951 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [165,0,0], thread: [0,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,951 [INFO ] W-9000-my_tc_1.0-stdout MODEL_LOG -     output = self.inference(data_preprocess)
2022-11-22T15:40:53,951 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [165,0,0], thread: [1,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,952 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [165,0,0], thread: [2,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,952 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [165,0,0], thread: [3,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,953 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [165,0,0], thread: [4,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,952 [INFO ] W-9000-my_tc_1.0-stdout MODEL_LOG -   File "/tmp/models/087d3a3f58404ab7b992aea4a4027088/Transformer_handler_generalized.py", line 309, in inference
2022-11-22T15:40:53,953 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [165,0,0], thread: [5,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,953 [INFO ] W-9000-my_tc_1.0-stdout MODEL_LOG -     outputs = self.model.generate(
2022-11-22T15:40:53,954 [INFO ] W-9000-my_tc_1.0-stdout MODEL_LOG -   File "/root/miniconda3/envs/mydeploy/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-22T15:40:53,954 [INFO ] W-9000-my_tc_1.0-stdout MODEL_LOG -     return func(*args, **kwargs)
2022-11-22T15:40:53,955 [INFO ] W-9000-my_tc_1.0-stdout MODEL_LOG -   File "/root/miniconda3/envs/mydeploy/lib/python3.8/site-packages/transformers/generation_utils.py", line 997, in generate
2022-11-22T15:40:53,955 [INFO ] W-9000-my_tc_1.0-stdout MODEL_LOG -     return self.sample(
2022-11-22T15:40:53,955 [INFO ] W-9000-my_tc_1.0-stdout MODEL_LOG -   File "/root/miniconda3/envs/mydeploy/lib/python3.8/site-packages/transformers/generation_utils.py", line 1512, in sample
2022-11-22T15:40:53,955 [INFO ] W-9000-my_tc_1.0-stdout MODEL_LOG -     outputs = self(
2022-11-22T15:40:53,956 [INFO ] W-9000-my_tc_1.0-stdout MODEL_LOG -   File "/root/miniconda3/envs/mydeploy/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
2022-11-22T15:40:53,956 [INFO ] W-9000-my_tc_1.0-stdout MODEL_LOG -     return forward_call(*input, **kwargs)
2022-11-22T15:40:53,956 [INFO ] W-9000-my_tc_1.0-stdout MODEL_LOG -   File "/root/miniconda3/envs/mydeploy/lib/python3.8/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 941, in forward
2022-11-22T15:40:53,957 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [165,0,0], thread: [6,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,957 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [165,0,0], thread: [7,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,958 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [165,0,0], thread: [8,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,958 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [165,0,0], thread: [9,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,958 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [165,0,0], thread: [10,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,958 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [165,0,0], thread: [11,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,959 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [165,0,0], thread: [12,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,960 [INFO ] W-9000-my_tc_1.0 ACCESS_LOG - /127.0.0.1:37814 "POST /predictions/my_tc HTTP/1.1" 503 3290
2022-11-22T15:40:53,960 [INFO ] W-9000-my_tc_1.0-stdout MODEL_LOG -     transformer_outputs = self.transformer(
2022-11-22T15:40:53,960 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [165,0,0], thread: [13,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,960 [INFO ] W-9000-my_tc_1.0 TS_METRICS - Requests5XX.Count:1|#Level:Host|#hostname:server-t493yc,timestamp:1669131650
2022-11-22T15:40:53,961 [INFO ] W-9000-my_tc_1.0-stdout MODEL_LOG -   File "/root/miniconda3/envs/mydeploy/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
2022-11-22T15:40:53,961 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [165,0,0], thread: [14,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,961 [DEBUG] W-9000-my_tc_1.0 org.pytorch.serve.job.Job - Waiting time ns: 3067245218, Inference time ns: 3272552664
2022-11-22T15:40:53,961 [INFO ] W-9000-my_tc_1.0-stdout MODEL_LOG -     return forward_call(*input, **kwargs)
2022-11-22T15:40:53,961 [INFO ] W-9000-my_tc_1.0 TS_METRICS - WorkerThreadTime.ms:14|#Level:Host|#hostname:server-t493yc,timestamp:1669131653
2022-11-22T15:40:53,961 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [165,0,0], thread: [15,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,962 [INFO ] W-9000-my_tc_1.0-stdout MODEL_LOG -   File "/root/miniconda3/envs/mydeploy/lib/python3.8/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 789, in forward
2022-11-22T15:40:53,962 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [165,0,0], thread: [16,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,962 [INFO ] W-9000-my_tc_1.0-stdout MODEL_LOG -     outputs = block(
2022-11-22T15:40:53,963 [INFO ] W-9000-my_tc_1.0-stdout MODEL_LOG -   File "/root/miniconda3/envs/mydeploy/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
2022-11-22T15:40:53,963 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [165,0,0], thread: [17,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,964 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [165,0,0], thread: [18,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,964 [INFO ] W-9000-my_tc_1.0-stdout MODEL_LOG -     return forward_call(*input, **kwargs)
2022-11-22T15:40:53,964 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [165,0,0], thread: [19,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,964 [INFO ] W-9000-my_tc_1.0-stdout MODEL_LOG -   File "/root/miniconda3/envs/mydeploy/lib/python3.8/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 317, in forward
2022-11-22T15:40:53,965 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [165,0,0], thread: [20,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,965 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [165,0,0], thread: [21,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,966 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [165,0,0], thread: [22,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,965 [INFO ] W-9000-my_tc_1.0-stdout MODEL_LOG -     attn_outputs = self.attn(
2022-11-22T15:40:53,967 [INFO ] W-9000-my_tc_1.0-stdout MODEL_LOG -   File "/root/miniconda3/envs/mydeploy/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
2022-11-22T15:40:53,969 [INFO ] W-9000-my_tc_1.0-stdout MODEL_LOG -     return forward_call(*input, **kwargs)
2022-11-22T15:40:53,967 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [165,0,0], thread: [23,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,969 [INFO ] W-9000-my_tc_1.0-stdout MODEL_LOG -   File "/root/miniconda3/envs/mydeploy/lib/python3.8/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 242, in forward
2022-11-22T15:40:53,970 [INFO ] W-9000-my_tc_1.0-stdout MODEL_LOG -     query, key, value = self.c_attn(hidden_states).split(self.split_size, dim=2)
2022-11-22T15:40:53,970 [INFO ] W-9000-my_tc_1.0-stdout MODEL_LOG -   File "/root/miniconda3/envs/mydeploy/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
2022-11-22T15:40:53,970 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [165,0,0], thread: [24,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,970 [INFO ] W-9000-my_tc_1.0-stdout MODEL_LOG -     return forward_call(*input, **kwargs)
2022-11-22T15:40:53,970 [INFO ] W-9000-my_tc_1.0-stdout MODEL_LOG -   File "/root/miniconda3/envs/mydeploy/lib/python3.8/site-packages/transformers/modeling_utils.py", line 1400, in forward
2022-11-22T15:40:53,970 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [165,0,0], thread: [25,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,970 [INFO ] W-9000-my_tc_1.0-stdout MODEL_LOG -     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2022-11-22T15:40:53,970 [INFO ] W-9000-my_tc_1.0-stdout MODEL_LOG - RuntimeError: CUDA error: CUBLAS_STATUS_NOT_INITIALIZED when calling `cublasCreate(handle)`
2022-11-22T15:40:53,971 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [165,0,0], thread: [26,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,971 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [165,0,0], thread: [27,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,971 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [165,0,0], thread: [28,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,972 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [165,0,0], thread: [29,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-11-22T15:40:53,973 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - ../aten/src/ATen/native/cuda/Indexing.cu:975: indexSelectLargeIndex: block: [165,0,0], thread: [30,0,0] Assertion `srcIndex < srcSelectDimSize` failed.

Installation instructions

no no

Model Packaing

follow the official example of deploying gpt2

config.properties

No response

Versions

Environment headers

Torchserve branch:

torchserve==0.6.1b20221114 torch-model-archiver==0.6.1b20221114

Python version: 3.8 (64-bit runtime) Python executable: /root/miniconda3/envs/mydeploy/bin/python

Versions of relevant python libraries: captum==0.5.0 future==0.18.2 numpy==1.23.5 nvgpu==0.9.0 psutil==5.9.4 requests==2.28.1 torch==1.12.0+cu116 torch-model-archiver==0.6.1b20221114 torch-workflow-archiver==0.2.5b20221114 torchaudio==0.12.0+cu116 torchserve==0.6.1b20221114 torchtext==0.13.0 torchvision==0.13.0+cu116 transformers==4.6.0 wheel==0.38.4 torch==1.12.0+cu116 torchtext==0.13.0 torchvision==0.13.0+cu116 torchaudio==0.12.0+cu116

Java Version:

OS: N/A GCC version: (GCC) 4.8.5 20150623 (Red Hat 4.8.5-44) Clang version: N/A CMake version: N/A

Is CUDA available: Yes CUDA runtime version: 11.6.124 GPU models and configuration: GPU 0: Tesla V100-PCIE-32GB Nvidia driver version: 510.85.02 cuDNN version: None

Repro instructions

as in the description

Possible Solution

No response

msaroufim commented 1 year ago

CUDA error: CUBLAS_STATUS_NOT_INITIALIZED when callingcublasCreate(handle)` typically happens if you're running out of GPU memory. So your options are either to get a GPU with a bigger on device memory or try out a smaller model

kreamyu commented 1 year ago

@msaroufim thanks for your reply,and I tried the example again The setup_config.json is 

{
 "model_name":"gpt2",
 "mode":"text_generation",
 "do_lower_case":true,
 "num_labels":"0",
 "save_mode":"pretrained",
 "max_length":"50",
 "captum_explanation":false,
 "FasterTransformer":false,
 "embedding_name": "gpt2",
 "model_parallel": true,
 "batch_size": 1
}

and then

rm -rf Transformer_model

python Download_Transformer_models.py

torch-model-archiver --model-name Textgeneration --version 1.0 --serialized-file Transformer_model/pytorch_model.bin --handler ./Transformer_handler_generalized.py --extra-files "Transformer_model/config.json,Transformer_model/special_tokens_map.json,Transformer_model/tokenizer_config.json,Transformer_model/tokenizer.json,Transformer_model/vocab.json,Transformer_model/merges.txt,./setup_config.json"

mv Textgeneration.mar model_store/

torchserve --start --model-store model_store --models my_tc=Textgeneration.mar --ncs

and then

curl -X POST http://127.0.0.1:8080/predictions/my_tc -T input.txt

the input.txt is

Today the weather is really nice and I am planning on

and the output is

(base) [root@server-t493yc mydeploy]# curl -X POST http://127.0.0.1:8080/predictions/my_tc -T input.txt
Today the weather is really nice and I am planning on
(base) [root@server-t493yc mydeploy]#

Predicted content is empty

the log is

(mydeploy) [root@server-t493yc Huggingface_Transformers]# WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance.
2022-11-23T03:36:45,794 [INFO ] main org.pytorch.serve.servingsdk.impl.PluginsManager - Initializing plugins manager...
2022-11-23T03:36:45,951 [INFO ] main org.pytorch.serve.ModelServer - 
Torchserve version: 0.6.1
TS Home: /root/miniconda3/envs/mydeploy/lib/python3.8/site-packages
Current directory: /root/mydeploy/serve/examples/Huggingface_Transformers
Temp directory: /tmp
Metrics config path: /root/miniconda3/envs/mydeploy/lib/python3.8/site-packages/ts/configs/metrics.yaml
Number of GPUs: 1
Number of CPUs: 8
Max heap size: 8004 M
Python executable: /root/miniconda3/envs/mydeploy/bin/python
Config file: N/A
Inference address: http://127.0.0.1:8080
Management address: http://127.0.0.1:8081
Metrics address: http://127.0.0.1:8082
Model Store: /root/mydeploy/serve/examples/Huggingface_Transformers/model_store
Initial Models: my_tc=Textgeneration.mar
Log dir: /root/mydeploy/serve/examples/Huggingface_Transformers/logs
Metrics dir: /root/mydeploy/serve/examples/Huggingface_Transformers/logs
Netty threads: 0
Netty client threads: 0
Default workers per model: 1
Blacklist Regex: N/A
Maximum Response Size: 6553500
Maximum Request Size: 6553500
Limit Maximum Image Pixels: true
Prefer direct buffer: false
Allowed Urls: [file://.*|http(s)?://.*]
Custom python dependency for model allowed: false
Metrics report format: prometheus
Enable metrics API: true
Workflow Store: /root/mydeploy/serve/examples/Huggingface_Transformers/model_store
Model config: N/A
2022-11-23T03:36:45,963 [INFO ] main org.pytorch.serve.servingsdk.impl.PluginsManager -  Loading snapshot serializer plugin...
2022-11-23T03:36:46,002 [INFO ] main org.pytorch.serve.ModelServer - Loading initial models: Textgeneration.mar
2022-11-23T03:36:54,303 [DEBUG] main org.pytorch.serve.wlm.ModelVersionedRefs - Adding new version 1.0 for model my_tc
2022-11-23T03:36:54,303 [DEBUG] main org.pytorch.serve.wlm.ModelVersionedRefs - Setting default version to 1.0 for model my_tc
2022-11-23T03:36:54,303 [INFO ] main org.pytorch.serve.wlm.ModelManager - Model my_tc loaded.
2022-11-23T03:36:54,303 [DEBUG] main org.pytorch.serve.wlm.ModelManager - updateModel: my_tc, count: 1
2022-11-23T03:36:54,322 [INFO ] main org.pytorch.serve.ModelServer - Initialize Inference server with: EpollServerSocketChannel.
2022-11-23T03:36:54,322 [DEBUG] W-9000-my_tc_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/root/miniconda3/envs/mydeploy/bin/python, /root/miniconda3/envs/mydeploy/lib/python3.8/site-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9000, --metrics-config, /root/miniconda3/envs/mydeploy/lib/python3.8/site-packages/ts/configs/metrics.yaml]
2022-11-23T03:36:54,466 [INFO ] main org.pytorch.serve.ModelServer - Inference API bind to: http://127.0.0.1:8080
2022-11-23T03:36:54,467 [INFO ] main org.pytorch.serve.ModelServer - Initialize Management server with: EpollServerSocketChannel.
2022-11-23T03:36:54,470 [INFO ] main org.pytorch.serve.ModelServer - Management API bind to: http://127.0.0.1:8081
2022-11-23T03:36:54,471 [INFO ] main org.pytorch.serve.ModelServer - Initialize Metrics server with: EpollServerSocketChannel.
2022-11-23T03:36:54,473 [INFO ] main org.pytorch.serve.ModelServer - Metrics API bind to: http://127.0.0.1:8082
Model server started.
2022-11-23T03:36:54,900 [WARN ] pool-3-thread-1 org.pytorch.serve.metrics.MetricCollector - worker pid is not available yet.
2022-11-23T03:36:55,476 [INFO ] W-9000-my_tc_1.0-stdout MODEL_LOG - Listening on port: /tmp/.ts.sock.9000
2022-11-23T03:36:55,479 [INFO ] W-9000-my_tc_1.0-stdout MODEL_LOG - Successfully loaded /root/miniconda3/envs/mydeploy/lib/python3.8/site-packages/ts/configs/metrics.yaml.
2022-11-23T03:36:55,480 [INFO ] W-9000-my_tc_1.0-stdout MODEL_LOG - [PID]10766
2022-11-23T03:36:55,481 [INFO ] W-9000-my_tc_1.0-stdout MODEL_LOG - Torch worker started.
2022-11-23T03:36:55,481 [INFO ] W-9000-my_tc_1.0-stdout MODEL_LOG - Python runtime: 3.8.15
2022-11-23T03:36:55,482 [DEBUG] W-9000-my_tc_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-my_tc_1.0 State change null -> WORKER_STARTED
2022-11-23T03:36:55,492 [INFO ] pool-3-thread-1 TS_METRICS - CPUUtilization.Percent:0.0|#Level:Host|#hostname:server-t493yc,timestamp:1669174615
2022-11-23T03:36:55,493 [INFO ] pool-3-thread-1 TS_METRICS - DiskAvailable.Gigabytes:108.4411392211914|#Level:Host|#hostname:server-t493yc,timestamp:1669174615
2022-11-23T03:36:55,494 [INFO ] pool-3-thread-1 TS_METRICS - DiskUsage.Gigabytes:91.54809951782227|#Level:Host|#hostname:server-t493yc,timestamp:1669174615
2022-11-23T03:36:55,494 [INFO ] pool-3-thread-1 TS_METRICS - DiskUtilization.Percent:45.8|#Level:Host|#hostname:server-t493yc,timestamp:1669174615
2022-11-23T03:36:55,494 [INFO ] pool-3-thread-1 TS_METRICS - GPUMemoryUtilization.Percent:0.0|#Level:Host,device_id:0|#hostname:server-t493yc,timestamp:1669174615
2022-11-23T03:36:55,495 [INFO ] pool-3-thread-1 TS_METRICS - GPUMemoryUsed.Megabytes:0|#Level:Host,device_id:0|#hostname:server-t493yc,timestamp:1669174615
2022-11-23T03:36:55,495 [INFO ] pool-3-thread-1 TS_METRICS - GPUUtilization.Percent:0|#Level:Host,device_id:0|#hostname:server-t493yc,timestamp:16691746152022-11-23T03:36:55,495 [INFO ] pool-3-thread-1 TS_METRICS - MemoryAvailable.Megabytes:26329.0703125|#Level:Host|#hostname:server-t493yc,timestamp:1669174615
2022-11-23T03:36:55,496 [INFO ] pool-3-thread-1 TS_METRICS - MemoryUsed.Megabytes:5228.625|#Level:Host|#hostname:server-t493yc,timestamp:1669174615
2022-11-23T03:36:55,496 [INFO ] pool-3-thread-1 TS_METRICS - MemoryUtilization.Percent:17.8|#Level:Host|#hostname:server-t493yc,timestamp:1669174615
2022-11-23T03:36:55,497 [INFO ] W-9000-my_tc_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9000
2022-11-23T03:36:55,516 [INFO ] W-9000-my_tc_1.0-stdout MODEL_LOG - Connection accepted: /tmp/.ts.sock.9000.
2022-11-23T03:36:55,523 [INFO ] W-9000-my_tc_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req. to backend at: 1669174615523
2022-11-23T03:36:55,594 [INFO ] W-9000-my_tc_1.0-stdout MODEL_LOG - model_name: my_tc, batchSize: 1
2022-11-23T03:36:56,715 [INFO ] W-9000-my_tc_1.0-stdout MODEL_LOG - Transformers version 4.6.0
2022-11-23T03:37:09,067 [INFO ] W-9000-my_tc_1.0-stdout MODEL_LOG - Transformer model from path /tmp/models/e0ca41298bd342b69291cecfa59ea6a4 loaded successfully
2022-11-23T03:37:09,070 [INFO ] W-9000-my_tc_1.0 org.pytorch.serve.wlm.WorkerThread - Backend response time: 13476
2022-11-23T03:37:09,071 [DEBUG] W-9000-my_tc_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-my_tc_1.0 State change WORKER_STARTED -> WORKER_MODEL_LOADED
2022-11-23T03:37:09,072 [INFO ] W-9000-my_tc_1.0 TS_METRICS - W-9000-my_tc_1.0.ms:14761|#Level:Host|#hostname:server-t493yc,timestamp:1669174629
2022-11-23T03:37:09,073 [INFO ] W-9000-my_tc_1.0 TS_METRICS - WorkerThreadTime.ms:74|#Level:Host|#hostname:server-t493yc,timestamp:1669174629
2022-11-23T03:37:09,074 [INFO ] W-9000-my_tc_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req. to backend at: 1669174629074
2022-11-23T03:37:09,155 [INFO ] W-9000-my_tc_1.0-stdout MODEL_LOG - Backend received inference at: 1669174629
2022-11-23T03:37:09,157 [INFO ] W-9000-my_tc_1.0-stdout MODEL_LOG - Received text: 'Today the weather is really nice and I am planning on
2022-11-23T03:37:09,156 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - Truncation was not explicitly activated but `max_length` is provided a specific value, please use `truncation=True` to explicitly truncate examples to max length. Defaulting to 'longest_first' truncation strategy. If you encode pairs of sequences (GLUE-style) with the tokenizer you can select this strategy more precisely by providing a specific strategy to `truncation`.
2022-11-23T03:37:09,157 [INFO ] W-9000-my_tc_1.0-stdout MODEL_LOG - '
2022-11-23T03:37:09,161 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - /root/miniconda3/envs/mydeploy/lib/python3.8/site-packages/transformers/tokenization_utils_base.py:2104: FutureWarning: The `pad_to_max_length` argument is deprecated and will be removed in a future version, use `padding=True` or `padding='longest'` to pad to the longest sequence in the batch, or use `padding='max_length'` to pad to a max length. In this case, you can give a specific length with `max_length` (e.g. `max_length=45`) or leave max_length to None to pad to the maximal input size of the model (e.g. 512 for Bert).
2022-11-23T03:37:09,162 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG -   warnings.warn(
2022-11-23T03:37:09,162 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.
2022-11-23T03:37:09,162 [WARN ] W-9000-my_tc_1.0-stderr MODEL_LOG - Input length of input_ids is 50, but ``max_length`` is set to 50.This can lead to unexpected behavior. You should consider increasing ``config.max_length`` or ``max_length``.
2022-11-23T03:37:09,865 [INFO ] W-9000-my_tc_1.0-stdout MODEL_LOG - Generated text: '['Today the weather is really nice and I am planning on\nIn']'
2022-11-23T03:37:09,865 [INFO ] W-9000-my_tc_1.0 org.pytorch.serve.wlm.WorkerThread - Backend response time: 789
2022-11-23T03:37:09,866 [INFO ] W-9000-my_tc_1.0-stdout MODEL_LOG - Generated text ['Today the weather is really nice and I am planning on\nIn']
2022-11-23T03:37:09,867 [INFO ] W-9000-my_tc_1.0 ACCESS_LOG - /127.0.0.1:57290 "POST /predictions/my_tc HTTP/1.1" 200 3031
2022-11-23T03:37:09,866 [INFO ] W-9000-my_tc_1.0-stdout MODEL_METRICS - HandlerTime.Milliseconds:708.87|#ModelName:my_tc,Level:Model|#hostname:server-t493yc,requestID:036f3482-7533-478b-8dd7-ed722a255178,timestamp:1669174629
2022-11-23T03:37:09,867 [INFO ] W-9000-my_tc_1.0 TS_METRICS - Requests2XX.Count:1|#Level:Host|#hostname:server-t493yc,timestamp:1669174626
2022-11-23T03:37:09,868 [DEBUG] W-9000-my_tc_1.0 org.pytorch.serve.job.Job - Waiting time ns: 2225885894, Backend time ns: 793466486
2022-11-23T03:37:09,868 [INFO ] W-9000-my_tc_1.0 TS_METRICS - QueueTime.ms:2225|#Level:Host|#hostname:server-t493yc,timestamp:1669174629
2022-11-23T03:37:09,868 [INFO ] W-9000-my_tc_1.0-stdout MODEL_METRICS - PredictionTime.Milliseconds:709.02|#ModelName:my_tc,Level:Model|#hostname:server-t493yc,requestID:036f3482-7533-478b-8dd7-ed722a255178,timestamp:1669174629
2022-11-23T03:37:09,868 [INFO ] W-9000-my_tc_1.0 TS_METRICS - WorkerThreadTime.ms:5|#Level:Host|#hostname:server-t493yc,timestamp:1669174629

Is this a error please?

msaroufim commented 1 year ago

Looks like it's working

kreamyu commented 1 year ago

Looks like it's working

Thanks for your helpļ¼