Too many false detections with Custom dataset FP16 TFRT inference

[Allamrahul]

Dataset: I am using a custom dataset with npy files and annotations. I just have a single class of objects to detect. I followed all steps required for custom dataset preparation and I am able to get great results during the evaluation phase using pytorch (90% mAP).

Post that, I am able to convert the model to onnx format and perform inference using TFRT. I am getting more or less same detections as what I got during the evaluation phase.

However, when I perform TFRT inference using FP16, I am getting too many false detections(200's) as shown below:

I observed the same behavior on other examples as well. When I disable FP16, my detections using TFRT are approximately close to my evaluation numbers. But something about fp16 makes it go out of whack.

Please let me know if I am missing something here

[Allamrahul]

I see that MAX_VOXELS is hard coded to 10000 in the export script exporter.py. But when I examine the pointpillar.yaml, I see this: MAX_NUMBER_OF_VOXELS: { 'train': 16000, 'test': 40000 } So, should'nt MAX_VOXELS in the export script be 40000?

I tried this out: when I gave 40000 for MAX_VOXELS and exported the onnx file, the multiple false positives I get in FP16 inference TFRT goes away. Can anyone confirm what I did makes sense?