search

spoonsso / dannce

MIT License
214 stars 30 forks source link

Bug (?) and Inquiry of preparing DANNCE data #104

Closed EvenGu closed 2 years ago

EvenGu commented 2 years ago

Hi, I am banging my head trying to prepare the label_3d_file in the Matlab that can be used to train a DANNCE model. I use DeepLabCut to get 2D labels, and trained a COM network. The COM predictions look alright. Unfortunately, my DANNCE model (finetuned from one of yours that takes 3 cameras) makes terrible predictions. I am therefore examing the code to see how the label_3d_file is used.

I find this line of code buggy: https://github.com/spoonsso/dannce/blob/cefd782aee1fce2044c7cef6237dfc207034fc68/dannce/engine/serve_data_DANNCE.py#L84 Here data is the 2D labeled data in the shape of (nFrames, 2, nKeypoints). The above line will subtract 1 from all elements which will result in a shift in all the 2D positions. It has nothing to do with the Matlab "1" indexing as suggested by the comment one line above. For indexing, one should subtract 1 from data_sampleID and/or data_frame (?). This line might not harm the model much because shifting by 1 pixel is no big deal. But I would like to point it out and make sure I understand the code correct.

In addition, my current data_3d comes from simple triangulation. I don't use the labdel3D gui. How many 3d labels should I provide to train DANNCE? My trainning video has about 12000 frames, and I have about 6000 trianglated 3D points. -- I only triangluated from 2D points whose confidence (using DLC) is about 0.95. Is that enough? Am I doing this corrected?

Looking forward to your reply and thank you in advance!

spoonsso commented 2 years ago
  1. We are using python/opencv coordinate system, with the top-left pixel at (0,0), whereas Matlab uses (1,1) because they use "1" indexing.
  2. This 2d data is not actually being used for dannce training, only for com.
  3. You should have more than enough data, assuming the 3D labels cover a wide variety of poses (more on this below) and are accurate, so I suspect there is something else wrong. Can you visualize your labels with Label3D? This will allow you to verify that your (1) required dannce variables are set up correctly, (2) your camera calibrations are solid, (3) you cameras are synchronized properly, and (4) your triangulated 3D labels are good.

I am a little confused about your 3d labeling strategy, though. From the sound of it, there are certain poses that DLC is tracking well (although I recommend visualizing them to be sure, as I would with any labels I were using in general), and you are only using those as training frames for DANNCE with the hope that DANNCE can learn to track the poses DLC is not tracking well. But by construction DANNCE will not have access to the hard poses whose tracking you would like to improve because you have excluded them from the training set. And having DANNCE learn to track the poses that DLC is already working well on is superfluous, since DLC is already tracking those well.