search

ELS-RD / transformer-deploy

Efficient, scalable and enterprise-grade CPU/GPU inference server for šŸ¤— Hugging Face transformer models šŸš€
https://els-rd.github.io/transformer-deploy/
Apache License 2.0
1.64k stars 150 forks source link

Bug fix for embedding task #152

Closed fursovia closed 1 year ago

fursovia commented 1 year ago

I followed the instructions in readme: 

docker run -it --rm --gpus all \
  -v $PWD:/project ghcr.io/els-rd/transformer-deploy:0.5.1 \
  bash -c "cd /project && \
    convert_model -m \"sentence-transformers/msmarco-distilbert-cos-v5\" \
    --backend onnx \
    --task embedding \
    --seq-len 16 128 128"

docker run -it --rm --gpus all -p8000:8000 -p8001:8001 -p8002:8002 --shm-size 256m \
  -v $PWD/triton_models:/models nvcr.io/nvidia/tritonserver:22.07-py3 \
  bash -c "pip install transformers && tritonserver --model-repository=/models"

This fails with

I1101 17:05:46.144764 1 pinned_memory_manager.cc:240] Pinned memory pool is created at '0x7f9006000000' with size 268435456
I1101 17:05:46.145324 1 cuda_memory_manager.cc:105] CUDA memory pool is created on device 0 with size 67108864
I1101 17:05:46.152409 1 model_repository_manager.cc:1206] loading: transformer_onnx_tokenize:1
I1101 17:05:46.152455 1 model_repository_manager.cc:1206] loading: transformer_onnx_model:1
I1101 17:05:46.156017 1 onnxruntime.cc:2458] TRITONBACKEND_Initialize: onnxruntime
I1101 17:05:46.156072 1 onnxruntime.cc:2468] Triton TRITONBACKEND API version: 1.10
I1101 17:05:46.156099 1 onnxruntime.cc:2474] 'onnxruntime' TRITONBACKEND API version: 1.10
I1101 17:05:46.156122 1 onnxruntime.cc:2504] backend configuration:
{"cmdline":{"auto-complete-config":"true","min-compute-capability":"6.000000","backend-directory":"/opt/tritonserver/backends","default-max-batch-size":"4"}}
I1101 17:05:46.172462 1 onnxruntime.cc:2560] TRITONBACKEND_ModelInitialize: transformer_onnx_model (version 1)
I1101 17:05:46.173322 1 onnxruntime.cc:666] skipping model configuration auto-complete for 'transformer_onnx_model': inputs and outputs already specified
I1101 17:05:46.177306 1 onnxruntime.cc:2603] TRITONBACKEND_ModelInstanceInitialize: transformer_onnx_model_0 (GPU device 0)
I1101 17:05:47.870828 1 onnxruntime.cc:2637] TRITONBACKEND_ModelInstanceFinalize: delete instance state
I1101 17:05:47.870886 1 onnxruntime.cc:2583] TRITONBACKEND_ModelFinalize: delete model state
E1101 17:05:47.870905 1 model_repository_manager.cc:1355] failed to load 'transformer_onnx_model' version 1: Invalid argument: model 'transformer_onnx_model', tensor 'output': the model expects 2 dimensions (shape [-1,-1]) but the model configuration specifies 2 dimensions (shape [-1,768])
None of PyTorch, TensorFlow >= 2.0, or Flax have been found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.
I1101 17:05:49.552957 1 python_be.cc:1767] TRITONBACKEND_ModelInstanceInitialize: transformer_onnx_tokenize_0 (GPU device 0)
None of PyTorch, TensorFlow >= 2.0, or Flax have been found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.
I1101 17:05:50.105093 1 model_repository_manager.cc:1352] successfully loaded 'transformer_onnx_tokenize' version 1
E1101 17:05:50.105182 1 model_repository_manager.cc:1559] Invalid argument: ensemble 'transformer_onnx_inference' depends on 'transformer_onnx_model' which has no loaded version
I1101 17:05:50.105232 1 server.cc:559]
+------------------+------+
| Repository Agent | Path |
+------------------+------+
+------------------+------+

I1101 17:05:50.105281 1 server.cc:586]
+-------------+-----------------------------------------------------------------+------------------------------------------------------------------------+
| Backend     | Path                                                            | Config                                                                 |
+-------------+-----------------------------------------------------------------+------------------------------------------------------------------------+
| onnxruntime | /opt/tritonserver/backends/onnxruntime/libtriton_onnxruntime.so | {"cmdline":{"auto-complete-config":"true","min-compute-capability":"6. |
|             |                                                                 | 000000","backend-directory":"/opt/tritonserver/backends","default-max- |
|             |                                                                 | batch-size":"4"}}                                                      |
| python      | /opt/tritonserver/backends/python/libtriton_python.so           | {"cmdline":{"auto-complete-config":"true","min-compute-capability":"6. |
|             |                                                                 | 000000","backend-directory":"/opt/tritonserver/backends","default-max- |
|             |                                                                 | batch-size":"4"}}                                                      |
+-------------+-----------------------------------------------------------------+------------------------------------------------------------------------+

I1101 17:05:50.105335 1 server.cc:629]
+---------------------------+---------+------------------------------------------------------------------------------------------------------------------+
| Model                     | Version | Status                                                                                                           |
+---------------------------+---------+------------------------------------------------------------------------------------------------------------------+
| transformer_onnx_model    | 1       | UNAVAILABLE: Invalid argument: model 'transformer_onnx_model', tensor 'output': the model expects 2 dimensions ( |
|                           |         | shape [-1,-1]) but the model configuration specifies 2 dimensions (shape [-1,768])                               |
| transformer_onnx_tokenize | 1       | READY                                                                                                            |
+---------------------------+---------+------------------------------------------------------------------------------------------------------------------+

I1101 17:05:50.153105 1 metrics.cc:650] Collecting metrics for GPU 0: NVIDIA GeForce RTX 3090
I1101 17:05:50.153454 1 tritonserver.cc:2176]
+----------------------------------+----------------------------------------------------------------------------------------------------------------------+
| Option                           | Value                                                                                                                |
+----------------------------------+----------------------------------------------------------------------------------------------------------------------+
| server_id                        | triton                                                                                                               |
| server_version                   | 2.24.0                                                                                                               |
| server_extensions                | classification sequence model_repository model_repository(unload_dependents) schedule_policy model_configuration sys |
|                                  | tem_shared_memory cuda_shared_memory binary_tensor_data statistics trace                                             |
| model_repository_path[0]         | /models                                                                                                              |
| model_control_mode               | MODE_NONE                                                                                                            |
| strict_model_config              | 0                                                                                                                    |
| rate_limit                       | OFF                                                                                                                  |
| pinned_memory_pool_byte_size     | 268435456                                                                                                            |
| cuda_memory_pool_byte_size{0}    | 67108864                                                                                                             |
| response_cache_byte_size         | 0                                                                                                                    |
| min_supported_compute_capability | 6.0                                                                                                                  |
| strict_readiness                 | 1                                                                                                                    |
| exit_timeout                     | 30                                                                                                                   |
+----------------------------------+----------------------------------------------------------------------------------------------------------------------+

I1101 17:05:50.153486 1 server.cc:260] Waiting for in-flight requests to complete.
I1101 17:05:50.153492 1 server.cc:276] Timeout 30: Found 0 model versions that have in-flight inferences
I1101 17:05:50.153498 1 model_repository_manager.cc:1230] unloading: transformer_onnx_tokenize:1
I1101 17:05:50.153544 1 server.cc:291] All models are stopped, unloading models
I1101 17:05:50.153550 1 server.cc:298] Timeout 30: Found 1 live models and 0 in-flight non-inference requests
I1101 17:05:51.153647 1 server.cc:298] Timeout 29: Found 1 live models and 0 in-flight non-inference requests
I1101 17:05:51.241912 1 model_repository_manager.cc:1335] successfully unloaded 'transformer_onnx_tokenize' version 1
I1101 17:05:52.153849 1 server.cc:298] Timeout 28: Found 0 live models and 0 in-flight non-inference requests
error: creating server: Internal - failed to load all models

Setting output size to -1 solves the problem

fursovia commented 1 year ago

@ayoub-louati What do you think? :)