Question: How best to use DANNCE for multiple instances?

[kdquine]

So we're trying to get DANNCE working for two animals, and are running into a couple of problems...

• So we initially did COM training for n_instances = 2, and now have COM predictions generated for two animals. We used a 2-element skeleton for COM and it worked great.
• We have DANNCE labels generated for our training frames. We used a 10 element skeleton (concatenation of two 5-element skeletons), using the same line of thought we used for COM.

We're not sure what parameters we should specify for the dannce-train config and what sort of processing we need to do on the dannce.mat to get things in the proper shape to work.

We're at this point running into some issues during the voxel grid construction, I think because our parameters aren't quite right, or the strucutre of the dannce.mat isn't right for our needs. I think we're not quite sure how to actually approach getting dannce to predict multi-instance, and so general guidance would be helpful.

Here are two approaches we've considered for the config.yaml:

n_channels_in: 3 # because video is RGB n_channels_out: 20 # because fine-tuning new_n_channels_out: 5 # because our model has 5 points per animal n_instances: 2 # because two instances -- does this do anything in dannce-train the way it does for COM?

If we do this, do we have to ensure that the dannce training labels as correctly associated with each com-predicted instance?

Another possibility:

we merge the COM predictions into the mean point as the anchor, set a large ROI volume, and:

n_channels_in: 3 # because video is RGB n_channels_out: 20 # because fine-tuning new_n_channels_out: 10 # predicting all points for 2 animals n_instances: 1

Is there a better solution we missed?

Thanks so much!

[spoonsso]

A proper multi-instance DANNCE implementation is in the works, but the best way to do it now (and what we did for the PAIR-R24M paper) is treat each instance as completely independent. So each instance would have its own label3d_dannce.mat file with its own labels, and you would associate each of these label3d_dannce.mat files with their associated COMs (i.e., don't average them) -- either by using the com_file parameter in your exp block inside io.,yaml or by adding each COM to the appropriate label3d_dannce.mat file -- you can see what this com variable looks like by inspecting demo/markerless_mouse_1/label3d_dannce.mat.

As of now, n_instances does not do anythign for dannce-train or dannce-predict.