AIRLegend
commented
3 years ago Hi! Thanks for suggesting that :)
The two main problems I see with it is that it's limited to Nvidia's GPUs and, also, I don't know how it would impact the performance of the game constatnly firing requests with batch_size=1, which doesn't like sound the most efficient use of the GPU for me.
Actually, the main bottleneck is the camera image polling (even if not, most cameras run at 30fps) while the used OpenSeeFace models already run well at about 60fps on one core over onnxruntime. Improving the image capture part has higher priority now in order to improve the user experience.
Right now the inference is done on the CPU with the small-ish models. Latency is dramatically improved in many tasks, even at small batch sizes, by moving the network over to the GPU. Exploration of using https://developer.nvidia.com/tensorrt might reduce input lag and/or allow for more complex models to be used with similar latency.