Open nathanielrindlaub opened 1 year ago
From Dan:
Sometimes, if we're still missing animals, but one or both models look close, try again using YOLOv5's test-time augmentation tools via this alternative (but compatible) MD inference script.
Also according to Dan, "inference takes ~1.7x longer with TTA turned on". That's not as bad a hit as I was imagining, so very much worth evaluating.
Just a heads up that I got try running MDV5a compile dwith tensorrt and it was blazing fast. Example here: https://github.com/rbavery/animal_detector/blob/master/mdv5app/torchscript_to_tensorrt.py
It sped up inference something like ~10x on my GPU compared to running the torchscript model without tensorrt.
This might be the most cost effective option for bulk inference without requiring a change in architecture. still uses torchserve and virtually the same handler code paths.
@rbavery had a few ideas for future optimizations of the Megadetector v5 endpoint that I wanted to document:
@rbavery - anything else to add here??