Ideal sampling rate for training data

[mshallow]

I'm looking for advice on the ideal frame rate of video/ keypoint to use for the training and applying of this package. Our acquisition rate for videos is much higher than the default set frame rate in all of the functions, and I'm wondering if this could be changing the quality of the syllables being found. We are looking to attempt to downsample, or resample, the data to a lower frame rate before training and could do this after the keypoints are imported, but was wondering if it would be possible to add to the load_keypoints function an option to downsample the data?

[calebweinreb]

Hello,

We've run kpms on 120fps videos and it works well. Is your frame rate a lot higher than that? If you wanted to downsample, then I would just do so directly in the code, e.g.

coordinates = {k:v[::rate] for k,v in coordinates.items()}

We could potentially support downsampling in the code but it would have to be more complicated than just downsampling on import because that would have to be propagated when loading video frames at subsequent steps. Let's discuss your current framerate and what the issues are with it before going down that rate...

[mshallow]

I am currently working with data that was collected at 200fps, and in the past we have downsampled that significantly, which sometimes has led to longer hidden states in other types of modeling we have attempted. In the past however, we were basing our code off of code for datasets acquired at 30-60fps so significantly lower than what your data are collected at. At this point, I have had relative success in isolating interpretable syllables on a small test dataset from my data without downsampling, but I’ve been looking for options to potentially increase the mean syllable length. I’ve done some minimal messing around with the kappa values, but if you think that would be more powerful than downsampling, I can dedicate more time to that. I have been running into a few issues with training new checkpoints of the model either on the same test training data, or on a different test training dataset. I have had a few runs that have trained for all 500 iterations, but the past couple of training runs have terminated early around 300 iterations because of NaNs. I’ve attached a screenshot of the early termination message. I’m wondering if this comes from using a location_aware version of the model or if there is something else I am missing. Thanks for all the help! Best, Molly [A screen shot of a computer program Description automatically generated]

[A computer screen with white text Description automatically generated] From: Caleb Weinreb @.> Date: Thursday, February 29, 2024 at 5:07 AM To: dattalab/keypoint-moseq @.> Cc: Molly Shallow @.>, Author @.> Subject: Re: [dattalab/keypoint-moseq] Ideal sampling rate for training data (Issue #136)

Hello,

We've run kpms on 120fps videos and it works well. Is your frame rate a lot higher than that? If you wanted to downsample, then I would just do so directly in the code, e.g.

coordinates = {k:v[::rate] for k,v in coordinates.items()}

We could potentially support downsampling in the code but it would have to be more complicated than just downsampling on import because that would have to be propagated when loading video frames at subsequent steps. Let's discuss your current framerate and what the issues are with it before going down that rate...

— Reply to this email directly, view it on GitHubhttps://urldefense.com/v3/__https:/github.com/dattalab/keypoint-moseq/issues/136*issuecomment-1971102423__;Iw!!C5qS4YX3!B2WH8dpRFtP5WvNhDQZMHnqHb9PJRhLLwQneTNyDkT4bRsNMrnbTDKS4GGce7NtwBZyPjEXsUfqu__dwiTsSMc9iKkQ$, or unsubscribehttps://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/AJ6GK2Q722VCEYWIWHXBJMLYV4TYTAVCNFSM6AAAAABD6ZAW5KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNZRGEYDENBSGM__;!!C5qS4YX3!B2WH8dpRFtP5WvNhDQZMHnqHb9PJRhLLwQneTNyDkT4bRsNMrnbTDKS4GGce7NtwBZyPjEXsUfqu__dwiTsStXklHAU$. You are receiving this because you authored the thread.Message ID: @.***>

[calebweinreb]

Hi Molly,

I haven't used the location-aware version much so its possible that it's more prone to NaNs. Have you tried any of these tips? https://keypoint-moseq.readthedocs.io/en/latest/FAQs.html#nans-during-fitting

In any case, it seems like downsampling is something that would be useful to include (I actually downsampled recently because modeling with 120fps was using so much GPU memory and felt unnecessary). I'll add it as an experimental feature and post here when thats done. In the meantime, you can just do the hacky version of downsampling I suggested above and that should be forward-compatible with the official downsampling feature.

[mshallow]

I tried those methods previously when I had issues with the NaNs from the format of my multi animal DLC files which was fixed when you updated the loading step to exclude certain individuals, but I haven’t tried them with this particular instance of NaNs yet. I’ll try the hacky version of downsampling for now as well since part of my concern is the amount of memory being used and the fact that we probably don’t need 200fps resolution for what we are recording. Thanks for the help as always! Best, Molly

From: Caleb Weinreb @.> Date: Thursday, February 29, 2024 at 12:46 PM To: dattalab/keypoint-moseq @.> Cc: Molly Shallow @.>, Author @.> Subject: Re: [dattalab/keypoint-moseq] Ideal sampling rate for training data (Issue #136)

Hi Molly,

I haven't used the location-aware version much so its possible that it's more prone to NaNs. Have you tried any of these tips? https://keypoint-moseq.readthedocs.io/en/latest/FAQs.html#nans-during-fittinghttps://urldefense.com/v3/__https:/keypoint-moseq.readthedocs.io/en/latest/FAQs.html*nans-during-fitting__;Iw!!C5qS4YX3!HN3qbb0BE91Y2DpC2OFKR_AURKDYF1k3SRc1eMbvsbnUj90OaPx-sTWy5Isa2DGYMahbKyjVuB4rBd0bll1Mxo1lh1o$

In any case, it seems like downsampling is something that would be useful to include (I actually downsampled recently because modeling with 120fps was using so much GPU memory and felt unnecessary). I'll add it as an experimental feature and post here when thats done. In the meantime, you can just do the hacky version of downsampling I suggested above and that should be forward-compatible with the official downsampling feature.

— Reply to this email directly, view it on GitHubhttps://urldefense.com/v3/__https:/github.com/dattalab/keypoint-moseq/issues/136*issuecomment-1971929699__;Iw!!C5qS4YX3!HN3qbb0BE91Y2DpC2OFKR_AURKDYF1k3SRc1eMbvsbnUj90OaPx-sTWy5Isa2DGYMahbKyjVuB4rBd0bll1M0xd84Os$, or unsubscribehttps://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/AJ6GK2WSCGIS3RBUGX5XSQTYV6JPJAVCNFSM6AAAAABD6ZAW5KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNZRHEZDSNRZHE__;!!C5qS4YX3!HN3qbb0BE91Y2DpC2OFKR_AURKDYF1k3SRc1eMbvsbnUj90OaPx-sTWy5Isa2DGYMahbKyjVuB4rBd0bll1MpN1ThP0$. You are receiving this because you authored the thread.Message ID: @.***>

[calebweinreb]

FYI downsampling is now available in the main branch https://keypoint-moseq.readthedocs.io/en/latest/advanced.html#temporal-downsampling