search

isarsoft / yolov4-triton-tensorrt

This repository deploys YOLOv4 as an optimized TensorRT engine to Triton Inference Server
http://www.isarsoft.com
Other
277 stars 63 forks source link

grpc error #18

Closed chiyukunpeng closed 3 years ago

chiyukunpeng commented 3 years ago

error

(py38torch16) cp@ubuntu:~/project/yolov5/client$ python client.py dummy

Traceback (most recent call last): File "client.py", line 120, in <module> if not triton_client.is_server_live(): File "/home/cp/anaconda3/envs/py38torch16/lib/python3.8/site-packages/tritonclient/grpc/__init__.py", line 217, in is_server_live raise_error_grpc(rpc_error) File "/home/cp/anaconda3/envs/py38torch16/lib/python3.8/site-packages/tritonclient/grpc/__init__.py", line 61, in raise_error_grpc raise get_error_grpc(rpc_error) from None tritonclient.utils.InferenceServerException: [StatusCode.UNIMPLEMENTED]

conda env

absl-py 0.10.0 pypi_0 pypi autopep8 1.5.4 pypi_0 pypi blas 1.0 mkl anaconda ca-certificates 2020.7.22 0 anaconda cachetools 4.1.1 pypi_0 pypi certifi 2020.6.20 py38_0 anaconda chardet 3.0.4 pypi_0 pypi click 7.1.2 pypi_0 pypi cudatoolkit 10.1.243 h6bb024c_0 anaconda cycler 0.10.0 pypi_0 pypi cython 0.29.21 pypi_0 pypi freetype 2.10.2 h5ab3b9f_0 anaconda future 0.18.2 pypi_0 pypi gevent 20.9.0 pypi_0 pypi geventhttpclient 1.4.4 pypi_0 pypi google-auth 1.21.3 pypi_0 pypi google-auth-oauthlib 0.4.1 pypi_0 pypi greenlet 0.4.17 pypi_0 pypi grpcio 1.32.0 pypi_0 pypi idna 2.10 pypi_0 pypi intel-openmp 2020.2 254 anaconda jpeg 9b habf39ab_1 anaconda kiwisolver 1.2.0 pypi_0 pypi lcms2 2.11 h396b838_0 anaconda ld_impl_linux-64 2.33.1 h53a641e_7 anaconda libedit 3.1.20191231 h14c3975_1 anaconda libffi 3.3 he6710b0_2 anaconda libgcc-ng 9.1.0 hdf63c60_0 anaconda libpng 1.6.37 hbc83047_0 anaconda libstdcxx-ng 9.1.0 hdf63c60_0 anaconda libtiff 4.1.0 h2733197_1 anaconda lz4-c 1.9.2 he6710b0_1 anaconda markdown 3.2.2 pypi_0 pypi matplotlib 3.3.2 pypi_0 pypi mkl 2020.2 256 anaconda mkl-service 2.3.0 py38he904b0f_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main mkl_fft 1.2.0 py38h23d657b_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main mkl_random 1.1.1 py38h0573a6f_0 anaconda ncurses 6.2 he6710b0_1 anaconda ninja 1.10.1 py38hfd86e86_0 anaconda numpy 1.19.1 py38hbc911f0_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main numpy-base 1.19.1 py38hfa32c7d_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main nvidia-pyindex 1.0.5 pypi_0 pypi oauthlib 3.1.0 pypi_0 pypi olefile 0.46 py_0 anaconda opencv-python 4.4.0.44 pypi_0 pypi openssl 1.1.1h h7b6447c_0 anaconda packaging 20.4 pypi_0 pypi pillow 7.2.0 py38hb39fc2d_0 anaconda pip 20.2.2 py38_0 anaconda protobuf 3.13.0 pypi_0 pypi pyasn1 0.4.8 pypi_0 pypi pyasn1-modules 0.2.8 pypi_0 pypi pycodestyle 2.6.0 pypi_0 pypi pyparsing 2.4.7 pypi_0 pypi pyqt5 5.15.1 pypi_0 pypi pyqt5-sip 12.8.1 pypi_0 pypi pyqt5-tools 5.15.1.1.7.4 pypi_0 pypi python 3.8.5 h7579374_1 anaconda python-dateutil 2.8.1 pypi_0 pypi python-dotenv 0.15.0 pypi_0 pypi python-rapidjson 0.9.4 pypi_0 pypi pytorch 1.6.0 py3.8_cuda10.1.243_cudnn7.6.3_0 pytorch pyyaml 5.3.1 pypi_0 pypi readline 8.0 h7b6447c_0 anaconda requests 2.24.0 pypi_0 pypi requests-oauthlib 1.3.0 pypi_0 pypi rsa 4.6 pypi_0 pypi scipy 1.5.2 pypi_0 pypi setuptools 49.6.0 py38_0 anaconda sip 5.4.0 pypi_0 pypi six 1.15.0 py_0 anaconda sqlite 3.33.0 h62c20be_0 anaconda tensorboard 2.3.0 pypi_0 pypi tensorboard-plugin-wit 1.7.0 pypi_0 pypi tk 8.6.10 hbc83047_0 anaconda toml 0.10.2 pypi_0 pypi torchvision 0.7.0 py38_cu101 pytorch tqdm 4.49.0 pypi_0 pypi tritonclient 2.5.0 pypi_0 pypi urllib3 1.25.10 pypi_0 pypi werkzeug 1.0.1 pypi_0 pypi wheel 0.35.1 py_0 anaconda xz 5.2.5 h7b6447c_0 anaconda zlib 1.2.11 h7b6447c_3 anaconda zope-event 4.5.0 pypi_0 pypi zope-interface 5.2.0 pypi_0 pypi zstd 1.4.4 h0b5b093_3 anaconda

server

I1203 08:37:01.962787 1 model_repository_manager.cc:837] successfully loaded 'yolov5x' version 1 I1203 08:37:01.962801 1 model_repository_manager.cc:622] TriggerNextAction() 'yolov5x' version 1: 0 I1203 08:37:01.962805 1 model_repository_manager.cc:637] no next action, trigger OnComplete() I1203 08:37:01.962846 1 model_repository_manager.cc:492] GetVersionStates() 'yolov5x' Starting endpoints, 'inference:0' listening on I1203 08:37:01.963952 1 grpc_server.cc:494] New request handler for HealthHandler, 1 I1203 08:37:01.963977 1 grpc_server.cc:447] Threads started for HealthHandler I1203 08:37:01.964128 1 grpc_server.cc:584] New request handler for StatusHandler, 2 I1203 08:37:01.964157 1 grpc_server.cc:447] Threads started for StatusHandler I1203 08:37:01.964274 1 grpc_server.cc:683] New request handler for RepositoryHandler, 3 I1203 08:37:01.964319 1 grpc_server.cc:447] Threads started for RepositoryHandler I1203 08:37:01.964416 1 grpc_server.cc:1005] New request handler for InferHandler, 4 I1203 08:37:01.964444 1 grpc_server.cc:447] Threads started for InferHandler I1203 08:37:01.964547 1 grpc_server.cc:1281] New request handler for StreamInferHandler, 6 I1203 08:37:01.964581 1 grpc_server.cc:447] Threads started for StreamInferHandler I1203 08:37:01.964672 1 grpc_server.cc:1611] New request handler for ModelControlHandler, 7 I1203 08:37:01.964713 1 grpc_server.cc:447] Threads started for ModelControlHandler I1203 08:37:01.964798 1 grpc_server.cc:1702] New request handler for SharedMemoryControlHandler, 8 I1203 08:37:01.964849 1 grpc_server.cc:447] Threads started for SharedMemoryControlHandler I1203 08:37:01.964878 1 grpc_server.cc:1939] Started GRPCService at 0.0.0.0:8001 I1203 08:37:01.964903 1 http_server.cc:1411] Starting HTTPService at 0.0.0.0:8000 I1203 08:37:02.006834 1 http_server.cc:1426] Starting Metrics Service at 0.0.0.0:8002 I1203 08:37:02.012313 1 server.cc:224] Polling model repository I1203 08:37:17.012641 1 server.cc:224] Polling model repository I1203 08:37:32.013035 1 server.cc:224] Polling model repository

chiyukunpeng commented 3 years ago

error

(py38torch16) cp@ubuntu:~/project/yolov5/client$ python client.py dummy

Traceback (most recent call last): File "client.py", line 120, in <module> if not triton_client.is_server_live(): File "/home/cp/anaconda3/envs/py38torch16/lib/python3.8/site-packages/tritonclient/grpc/__init__.py", line 217, in is_server_live raise_error_grpc(rpc_error) File "/home/cp/anaconda3/envs/py38torch16/lib/python3.8/site-packages/tritonclient/grpc/__init__.py", line 61, in raise_error_grpc raise get_error_grpc(rpc_error) from None tritonclient.utils.InferenceServerException: [StatusCode.UNIMPLEMENTED]

conda env

absl-py 0.10.0 pypi_0 pypi autopep8 1.5.4 pypi_0 pypi blas 1.0 mkl anaconda ca-certificates 2020.7.22 0 anaconda cachetools 4.1.1 pypi_0 pypi certifi 2020.6.20 py38_0 anaconda chardet 3.0.4 pypi_0 pypi click 7.1.2 pypi_0 pypi cudatoolkit 10.1.243 h6bb024c_0 anaconda cycler 0.10.0 pypi_0 pypi cython 0.29.21 pypi_0 pypi freetype 2.10.2 h5ab3b9f_0 anaconda future 0.18.2 pypi_0 pypi gevent 20.9.0 pypi_0 pypi geventhttpclient 1.4.4 pypi_0 pypi google-auth 1.21.3 pypi_0 pypi google-auth-oauthlib 0.4.1 pypi_0 pypi greenlet 0.4.17 pypi_0 pypi grpcio 1.32.0 pypi_0 pypi idna 2.10 pypi_0 pypi intel-openmp 2020.2 254 anaconda jpeg 9b habf39ab_1 anaconda kiwisolver 1.2.0 pypi_0 pypi lcms2 2.11 h396b838_0 anaconda ld_impl_linux-64 2.33.1 h53a641e_7 anaconda libedit 3.1.20191231 h14c3975_1 anaconda libffi 3.3 he6710b0_2 anaconda libgcc-ng 9.1.0 hdf63c60_0 anaconda libpng 1.6.37 hbc83047_0 anaconda libstdcxx-ng 9.1.0 hdf63c60_0 anaconda libtiff 4.1.0 h2733197_1 anaconda lz4-c 1.9.2 he6710b0_1 anaconda markdown 3.2.2 pypi_0 pypi matplotlib 3.3.2 pypi_0 pypi mkl 2020.2 256 anaconda mkl-service 2.3.0 py38he904b0f_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main mkl_fft 1.2.0 py38h23d657b_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main mkl_random 1.1.1 py38h0573a6f_0 anaconda ncurses 6.2 he6710b0_1 anaconda ninja 1.10.1 py38hfd86e86_0 anaconda numpy 1.19.1 py38hbc911f0_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main numpy-base 1.19.1 py38hfa32c7d_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main nvidia-pyindex 1.0.5 pypi_0 pypi oauthlib 3.1.0 pypi_0 pypi olefile 0.46 py_0 anaconda opencv-python 4.4.0.44 pypi_0 pypi openssl 1.1.1h h7b6447c_0 anaconda packaging 20.4 pypi_0 pypi pillow 7.2.0 py38hb39fc2d_0 anaconda pip 20.2.2 py38_0 anaconda protobuf 3.13.0 pypi_0 pypi pyasn1 0.4.8 pypi_0 pypi pyasn1-modules 0.2.8 pypi_0 pypi pycodestyle 2.6.0 pypi_0 pypi pyparsing 2.4.7 pypi_0 pypi pyqt5 5.15.1 pypi_0 pypi pyqt5-sip 12.8.1 pypi_0 pypi pyqt5-tools 5.15.1.1.7.4 pypi_0 pypi python 3.8.5 h7579374_1 anaconda python-dateutil 2.8.1 pypi_0 pypi python-dotenv 0.15.0 pypi_0 pypi python-rapidjson 0.9.4 pypi_0 pypi pytorch 1.6.0 py3.8_cuda10.1.243_cudnn7.6.3_0 pytorch pyyaml 5.3.1 pypi_0 pypi readline 8.0 h7b6447c_0 anaconda requests 2.24.0 pypi_0 pypi requests-oauthlib 1.3.0 pypi_0 pypi rsa 4.6 pypi_0 pypi scipy 1.5.2 pypi_0 pypi setuptools 49.6.0 py38_0 anaconda sip 5.4.0 pypi_0 pypi six 1.15.0 py_0 anaconda sqlite 3.33.0 h62c20be_0 anaconda tensorboard 2.3.0 pypi_0 pypi tensorboard-plugin-wit 1.7.0 pypi_0 pypi tk 8.6.10 hbc83047_0 anaconda toml 0.10.2 pypi_0 pypi torchvision 0.7.0 py38_cu101 pytorch tqdm 4.49.0 pypi_0 pypi tritonclient 2.5.0 pypi_0 pypi urllib3 1.25.10 pypi_0 pypi werkzeug 1.0.1 pypi_0 pypi wheel 0.35.1 py_0 anaconda xz 5.2.5 h7b6447c_0 anaconda zlib 1.2.11 h7b6447c_3 anaconda zope-event 4.5.0 pypi_0 pypi zope-interface 5.2.0 pypi_0 pypi zstd 1.4.4 h0b5b093_3 anaconda

server

I1203 08:37:01.962787 1 model_repository_manager.cc:837] successfully loaded 'yolov5x' version 1 I1203 08:37:01.962801 1 model_repository_manager.cc:622] TriggerNextAction() 'yolov5x' version 1: 0 I1203 08:37:01.962805 1 model_repository_manager.cc:637] no next action, trigger OnComplete() I1203 08:37:01.962846 1 model_repository_manager.cc:492] GetVersionStates() 'yolov5x' Starting endpoints, 'inference:0' listening on I1203 08:37:01.963952 1 grpc_server.cc:494] New request handler for HealthHandler, 1 I1203 08:37:01.963977 1 grpc_server.cc:447] Threads started for HealthHandler I1203 08:37:01.964128 1 grpc_server.cc:584] New request handler for StatusHandler, 2 I1203 08:37:01.964157 1 grpc_server.cc:447] Threads started for StatusHandler I1203 08:37:01.964274 1 grpc_server.cc:683] New request handler for RepositoryHandler, 3 I1203 08:37:01.964319 1 grpc_server.cc:447] Threads started for RepositoryHandler I1203 08:37:01.964416 1 grpc_server.cc:1005] New request handler for InferHandler, 4 I1203 08:37:01.964444 1 grpc_server.cc:447] Threads started for InferHandler I1203 08:37:01.964547 1 grpc_server.cc:1281] New request handler for StreamInferHandler, 6 I1203 08:37:01.964581 1 grpc_server.cc:447] Threads started for StreamInferHandler I1203 08:37:01.964672 1 grpc_server.cc:1611] New request handler for ModelControlHandler, 7 I1203 08:37:01.964713 1 grpc_server.cc:447] Threads started for ModelControlHandler I1203 08:37:01.964798 1 grpc_server.cc:1702] New request handler for SharedMemoryControlHandler, 8 I1203 08:37:01.964849 1 grpc_server.cc:447] Threads started for SharedMemoryControlHandler I1203 08:37:01.964878 1 grpc_server.cc:1939] Started GRPCService at 0.0.0.0:8001 I1203 08:37:01.964903 1 http_server.cc:1411] Starting HTTPService at 0.0.0.0:8000 I1203 08:37:02.006834 1 http_server.cc:1426] Starting Metrics Service at 0.0.0.0:8002 I1203 08:37:02.012313 1 server.cc:224] Polling model repository I1203 08:37:17.012641 1 server.cc:224] Polling model repository I1203 08:37:32.013035 1 server.cc:224] Polling model repository

I think the reason of that is HTTPService is starting instead of started. My triton is 20.03. how to solve it? thanks.

philipp-schmidt commented 3 years ago

Hi, Triton Inference Server changed its interface recently. Starting from 20.08 they introduced breaking changes in the client in their release of Triton Version 2.x. You need to run at least triton 20.08 for the client example in this repo to work or you need to use the older implementation of the python clients, which you can download as a python wheel package from the releases on their github page. In your case head over to tritons github page and check our the python client examples for their version 20.03 and download the respective client version from the 20.03 release.

When using a mor recent version than 20.08, also make sure to use the respective client package here:

https://github.com/isarsoft/yolov4-triton-tensorrt/blob/4edbe849f111324d58eda5f14fe1e4e9b098d4cd/clients/python/environment.yml#L41