search

triton-inference-server / server

The Triton Inference Server provides an optimized cloud and edge inferencing solution.
https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/index.html
BSD 3-Clause "New" or "Revised" License
8.04k stars 1.44k forks source link

Using Python backend for jetson platform #3111

Closed Levocetirizine closed 3 years ago

Levocetirizine commented 3 years ago

Is your feature request related to a problem? Please describe. Hello, I'm trying to use python backend on Jetson AGX Xavier (L4T 32.5.1, JetPack 4.5). The Triton binary was downloaded from Release 2.11.0, and I noticed that only TensorFlow, TensorRT and ONNX backends are officially supported.

I was not sure if it's possible to run Python backend on Jetson, but I tried to clone the python_backend repo and built everything from source. It looks fine and I got things such as libtriton_python.so, the stub, and utils.py. I then copied them into the backend directory.

When I tried to load the models, these error logs were shown (add_sub model is from the example, yolov4_preprocess model is homemade and works well in x86):

➜  triton-dev tritonserver --model-repository `pwd`/demo-s --backend-directory /home/edge/sd/documents/tritonserver/backends
2021-07-13 19:27:19.634711: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.10.2
I0713 11:27:19.708073 19948 tensorflow.cc:2167] TRITONBACKEND_Initialize: tensorflow
I0713 11:27:19.708193 19948 tensorflow.cc:2180] Triton TRITONBACKEND API version: 1.4
I0713 11:27:19.708243 19948 tensorflow.cc:2186] 'tensorflow' TRITONBACKEND API version: 1.4
I0713 11:27:19.708262 19948 tensorflow.cc:2207] backend configuration:
{}
I0713 11:27:19.711176 19948 onnxruntime.cc:1971] TRITONBACKEND_Initialize: onnxruntime
I0713 11:27:19.711236 19948 onnxruntime.cc:1984] Triton TRITONBACKEND API version: 1.4
I0713 11:27:19.711260 19948 onnxruntime.cc:1990] 'onnxruntime' TRITONBACKEND API version: 1.4
I0713 11:27:19.828110 19948 pinned_memory_manager.cc:240] Pinned memory pool is created at '0x2037ca000' with size 268435456
I0713 11:27:19.828374 19948 cuda_memory_manager.cc:105] CUDA memory pool is created on device 0 with size 67108864
I0713 11:27:19.833049 19948 model_repository_manager.cc:1045] loading: add_sub:1
I0713 11:27:19.934184 19948 model_repository_manager.cc:1045] loading: yolov4_preprocess:1
I0713 11:27:19.947179 19948 python.cc:1494] TRITONBACKEND_ModelInstanceInitialize: add_sub_0 (CPU device 0)
E0713 11:27:19.948497 19948 model_repository_manager.cc:1215] failed to load 'add_sub' version 1: Internal: unexpected no host policy for add_sub_0
I0713 11:27:20.045892 19948 python.cc:1494] TRITONBACKEND_ModelInstanceInitialize: yolov4_preprocess_0 (CPU device 0)
E0713 11:27:20.047622 19948 model_repository_manager.cc:1215] failed to load 'yolov4_preprocess' version 1: Internal: unexpected no host policy for yolov4_preprocess_0
I0713 11:27:20.048036 19948 server.cc:504]

I could not understand the meaning unexpected no host policy , it seem related with instance_group [{ kind: KIND_CPU }] in config.pbtxt. But I have no clue...

The ONNX model can be loaded successfully though.

Describe the solution you'd like I am not sure if this is a bug or a misconfiguration. The Python backend gives large flexibility in ensemble model, is there a plan that it will be officially supported in Jetson ?

Tabrizian commented 3 years ago

The error that you are seeing is related to the way that you have compiled Python backend. You need to specify the TRITON_BACKEND_REPO_TAG, TRITON_COMMON_REPO_TAG, and TRITON_CORE_REPO_TAG to match the release version that you are using.

mkdir build
cd build
cmake -DTRITON_ENABLE_GPU=ON -DCMAKE_INSTALL_PREFIX:PATH=`pwd`/install -DTRITON_BACKEND_REPO_TAG=r21.05 -DTRITON_CORE_REPO_TAG=r21.05 -DTRITON_COMMON_REPO_TAG=r21.05 .. 
make install

The Python backend gives large flexibility in ensemble model, is there a plan that it will be officially supported in Jetson ?

@msalehiNV / @deadeyegoodwin Is there a plan to support PB in Jetson?

deadeyegoodwin commented 3 years ago

@Levocetirizine the build steps given above should fix your issue. Please reopen if it does not. We will consider adding the python backend to the jetson release.

Levocetirizine commented 3 years ago

@deadeyegoodwin @Tabrizian I just tested your cmake script and it works fine ! Thank you a lot for your reply.