search

isl-org / MiDaS

Code for robust monocular depth estimation described in "Ranftl et. al., Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer, TPAMI 2022"
MIT License
4.43k stars 619 forks source link

iOS Demo app is slowing down over time, and the first inference seems much slower #262

Closed TimYao18 closed 8 months ago

TimYao18 commented 9 months ago

Running the iOS Demo app, I found the 1st inference in CPU and GPU will slower than 2nd inference and later. When running inference with the CPU, it becomes noticeably slower. GPU and ANE also become slower but not very noticeable. And if switching the delegates to and from each other, it will become noticeably slower, not only CPU but GPU and ANE, too.

Is this an issue with the MiDaS model, or is it a problem with the TensorFlow Lite Delegate?

The log is as the attached file. I only log the first 105 times inference without switching the delegates. iphone_15_pro_max_midas.csv

heyoeyo commented 8 months ago

I haven't tried the iOS/tensorflow demo myself, but I can say from using pytorch on desktop machines that it's very common for the first run of large models (not just midas) to be slower than follow up runs. Presumably this is due to some initialization/compilation/caching step that occurs on the first run, but I don't know any of the details.

As for slow down observed on later runs, it could be a thermal effect (i.e. phone/cpu overheating). The cpu takes much longer to finish the tests (and is probably more power hungry) which would explain why the effect is more obvious on cpu (though the ane has strange inconsistencies... may just be in use by something else at the same time?). It might be worth repeating the testing with longer gaps between runs (or active cooling) to see if that improves the pattern.

TimYao18 commented 8 months ago

I think this should be a behavior of the Model. So I close the issue since it is not an issue.