search

cyrusbehr / tensorrt-cpp-api

TensorRT C++ API Tutorial
MIT License
596 stars 74 forks source link

half precision returns nan for the feature values #47

Closed sammilei closed 8 months ago

sammilei commented 8 months ago

Hi, I am adapting the code for the half precision model. The code runs without error with configuration with fp16. Dimension of input and output are correct too so not buffer issues. However, the feature printout are all nan. Do you have any ideas? Thank you!

cyrusbehr commented 8 months ago

If the model takes FP16 inputs and outputs, then you need to convert your IO buffers from FP32 to FP16 with a CUDA kernel.