search

DeepLabCut / DeepLabCut-live-GUI

GUI to run DeepLabCut on live video feed
GNU General Public License v3.0
58 stars 19 forks source link

Sudden slow down in performance #10

Closed darioringach closed 3 years ago

darioringach commented 3 years ago

I am experiencing a sudden slow-down in performance in DLC Live, and I was wondering if anyone has seen this before and knows how to fix it.

I am running a simple python script that receives imagery data over memory mapped files:

Capture

The pose estimation takes normally about 12-14msec... but the, all of sudden, it jumps to ~140ms:

Capture2

Any ideas what this could be? It happens after this processing loops has been working for 2-3 minutes. It always starts Ok and then it slows down. This is running on a Quadro M6000 24GB.

Thank you!

Dario

darioringach commented 3 years ago

...and if you keep running, it will jump again and slow even more:

Capture3

MMathisLab commented 3 years ago

Hi @darioringach i can confirm I’ve never had this in our dlc-live code or our dlc GUI, or bonsai or autopilot (to the extend we did a lot of testing for our paper).

darioringach commented 3 years ago

Hi @darioringach i can confirm I’ve never had this in our dlc-live code or our dlc GUI, or bonsai or autopilot (to the extend we did a lot of testing for our paper).

Thanks. I guess I will first try with a different GPU card.

MMathisLab commented 3 years ago

I doubt it’s the gpu; can you run the dlc-live benchmarking code, that profiles the system.

darioringach commented 3 years ago

I doubt it’s the gpu; can you run the dlc-live benchmarking code, that profiles the system.

How long does the benchmarking code runs for? It's been looping for over 2 hours now doing Run #i out of 6.

darioringach commented 3 years ago

I can confirm that this was a specific problem with the Quadro M6000.
Everything works fine on an RTX 2080 Ti.

MMathisLab commented 3 years ago

crazy! Thanks for testing this @darioringach - super valuable to know :)