I'm trying to setup transformer-deploy on tritonserver:22.08. tritonserver:22.07 had this bug in TensorRT.
I changed the following things to install transformer-deplot on 22.08 image
Changed base image in Dockerfile to FROM nvcr.io/nvidia/tritonserver:22.08-py3
Updated the onnx, onnxruntime and tensorrt versions in the requirements to the latest versions that is compatible with tensorrt 8.6
I'm able to convert a huggingface model using the convert script, however I see an error like this when deploying it on triton inference server. The solution mentioned in the script of importing tensorrt doesn't work.
Any pointers where I can begin to use transformer-deploy with tritonserver:22.08 and tensorrt:8.6.1?
I'm trying to setup transformer-deploy on tritonserver:22.08. tritonserver:22.07 had this bug in TensorRT.
I changed the following things to install transformer-deplot on 22.08 image
FROM nvcr.io/nvidia/tritonserver:22.08-py3
I'm able to convert a huggingface model using the
convert
script, however I see an error like this when deploying it on triton inference server. The solution mentioned in the script of importingtensorrt
doesn't work.Any pointers where I can begin to use transformer-deploy with tritonserver:22.08 and tensorrt:8.6.1?