AI-Serving not utilizing GPU

[ajertec]

Hello, I got AI Serving Server up and running:

• pulled autodeployai/ai-serving:0.9.0-cuda image,
• started server: docker run --rm -it -v $(pwd):/opt/ai-serving -p 9090:9090 -p 9091:9091 IMAGE_ID (even with docker run --gpus all --rm -it -v $(pwd):/opt/ai-serving -p 9090:9090 -p 9091:9091 IMAGE_ID)

However, when trying out mnist example (AIServingMnistOnnxModel.ipynb notebook), or modifying this example and deploying my custom models with gpu optimizations (https://pypi.org/project/onnxruntime-tools/) server is not utilizing the GPU, only CPU.

My models metadata: (this is actually TinyBERT transformer model)

Can you provide full example of using the AI-Serving with GPU utilization?

Thank you.

[scorebot]

@ajertec Thanks for your issue. It's a defect of the AI-Serving, we will fix it as soon as possible, and let you know once it's done.

[scorebot]

@ajertec We have fixed the issue of GPU. Please, try the image autodeployai/ai-serving:0.9.1-cuda that if resolve your issue.

[ajertec]

@scorebot Thank you, now GPU is being utilized.

[scorebot]

Cool. I close it now.