tano297
commented
4 years ago Hi there,
I don't think there is a way to fix this if you re-run the script every time, because the first time the inference happens, Pytorch has to send the model to the GPU, and do some benchmarking of the ops being run in the model.
Just to be clear, what you would like is that every time a new scan shows up in the pointed directory it gets inferred and sends the result to a different directory? If this is the case, then the best way to do it is to have a separate thread analyzing the directory, and if you have new frames or not and pushing to a queue, and then have the for loop for the inference popping from the queue, and blocking when there are no scans. Something KIND OF like this.
I am a big fan of TCP sockets, this may be a good way to do it (kind of how ROS works). You can make an infer script that loads the pytorch model and waits for TCP messages, and then have another, much simpler script that basically does the check for new scans and pushes them to the other script through TCP, and returns the result. The advantage of this is that you could even run the inference in a remote computer and have multiple clients sending requests. This is how cloud inference works in a way, and it would even allow you to batch scans from different users, and fully utilize the GPU, for faster overall processing. This may be like killing a bird with a cannon, but it could be a nice solution for scaling up afterwards :)
ivannson
Hi,
When running inference, the first scan always takes a lot longer (0.5-0.7 sec) compared to all other scans (0.01 sec), is there a way to fix this?
If not, I was wondering if you have any suggestions on how to implement live inference? For example, continuously infer new scans in the data directory. Currently I'm just running the inference script over an over, but because the first scan alway takes so long, the performance isn't great.