search

NVIDIA-AI-IOT / deepstream_triton_model_deploy

How to deploy open source models using DeepStream and Triton Inference Server
Apache License 2.0
73 stars 15 forks source link

Jetpack 4.5 triton 2.10 - deepstream 5.1 on Jetson Xavier NX #16

Open htran170642 opened 3 years ago

htran170642 commented 3 years ago

triton 2.10 supports onnx but still got error when loading model. Release link: [(https://github.com/triton-inference-server/server/releases)]

Input or output layers are empty

image

image

ghost commented 3 years ago

This repo is tracking only FasterRCNN and CenterFace model for the blog https://developer.nvidia.com/blog/deploying-models-from-tensorflow-model-zoo-using-deepstream-and-triton-inference-server/. Please create a thread on the DeepStream forum https://forums.developer.nvidia.com/c/accelerated-computing/intelligent-video-analytics/deepstream-sdk/15 listing the hardware(Jetson Xavier) and software(container vs .deb install of 5.1, JP version, model) you're using. From a glance, it seems like the log you've provided doesn't have anything to do with ONNX. When creating the thread on the forum provide a .tar with the model(if it's small enough, else cloud hosted link), and your config file along with the errors you've observed.

htran170642 commented 3 years ago

I just run one of the samples in deepstream 5.1 container, not mine. Those samples worked fine with triton 2.5 but when I tar Jetpack 4.5(it has triton 2.10 inside), all samples are down with same error above.

ghost commented 3 years ago

I see. Looks like the model is a TF1 model. So either the config file which is most probably at/near $DEEPSTREAM_DIR/samples/trtis_model_repo//config.pbtxt is different from Triton 2.5 and incorrect or Triton 2.10 is unable to infer input/output dimensions/layer names while 2.5 was able to. I'm on x86 so I can't repro this on a Jetson, but I can try running the 5.1 container and this model on x86 to see if I get something similar.

htran170642 commented 3 years ago

this is ds image I used: nvcr.io/nvidia/deepstream-l4t:5.1-21.02-samples (Deepstream 5.1 and Triton 2.5). Could you please try it? Thank you so much.