Restart training in COLAB

[CarmenWinters]

I currently am training a model in COLAB. However, I have reached my user limits. Can I restart training from the last saved epoch?

Carmen

[postpop]

Hi Carmen,

sorry about the late reply. Yes, restarting training is possible. See this notebook. Let me know if this works for or if you have more questions.

Jan

[CarmenWinters]

Hi Jan,

I am having some other difficulties and I am not sure anymore how to solve them.

So I am working on mouse pup ultrasonic vocalizations (sample rate of 250 kHz). They are sampled in a rather noisy environment unfortunately (see https://doi.org/10.1038/s41598-022-05641-w), so I high-pass filtered them at 40 kHz.

I than have labeled more than 2000 USVs (with varying amplitudes, ages, sex...) to train. I tried to use the same parameters for training as the mouse model explained in your publication, but prediction is not going too well. Here are some issues I seem to have

1. If I restart training or when I reach the limits of Colab, no _results.h5-file is created, whereas I cannot perform some visualizations

2. As I labeled the USV's as segments, I wanted to delete detections <1 ms, which gives me following error code: @.***

3. And this might be linked to 2, the predictions that are generated are looking like events (so start and end time are the exact same).

4. The predictions that are made do not trace back the USVs even in files which I used for training.

5. I now just trained on the segments and not the on- or offset, should I include that?

I am so sorry for the many questions, but do you have any suggestions of what I might do wrong?

Kind regards, Carmen

From: Jan Clemens @.> Sent: dinsdag 7 juni 2022 15:49 To: janclemenslab/das @.> Cc: Carmen Winters @.>; Author @.> Subject: Re: [janclemenslab/das] Restart training in COLAB (Issue #30)

Hi Carmen,

sorry about the late reply. Yes, restarting training is possible. See this notebookhttps://gist.github.com/postpop/98c348ccf970093f4f4e71e18b9a4bb6. Let me know if this works for or if you have more questions.

Jan

- Reply to this email directly, view it on GitHubhttps://github.com/janclemenslab/das/issues/30#issuecomment-1148702806, or unsubscribehttps://github.com/notifications/unsubscribe-auth/APM37JOVP2WJKJSTNO7PZYTVN5HMDANCNFSM5XM6SAAQ. You are receiving this because you authored the thread.Message ID: @.**@.>> [ { @.": "http://schema.org", @.": "EmailMessage", "potentialAction": { @.": "ViewAction", "target": "https://github.com/janclemenslab/das/issues/30#issuecomment-1148702806", "url": "https://github.com/janclemenslab/das/issues/30#issuecomment-1148702806", "name": "View Issue" }, "description": "View this Issue on GitHub", "publisher": { @.": "Organization", "name": "GitHub", "url": "https://github.com" } } ]

[postpop]

Sorry you run into so many issues and happy to help!

I re-fitted a model to our mice data just to make that we didn't introduce any bugs recently. Still works as before - the post processing (fill short gaps, delete short segments) is crucial to get smooth detections. The defaults (0.02 for both parameters) or the values from the paper should produce reasonable annotations.

1. The results file is created at the end of the das.train.train function. If training is interrupted, the file will not be created. I'll update the gist to also generate the results file. Probably a good idea to implement picking up training from an existing model in a future update. 2./3./4. I think there is sth fundamentally off - maybe some bug in the code. Looks like you delete all segments with this and that casues the error. Could you send the model and example data? That would help identifying the source of the error.
2. Including the on- and offsets rarely helps - I think the problem lies elsewhere.

[postpop]

The notebook is updated to now also test the model.

[CarmenWinters]

Hi Jan,

Thank you for your suggestions! I really would like to use DAS to analyze my data :)

In the following link I have two of my recordings (one recorded on PND5 and one on PND7), the last model and the .yaml-file. https://filesender.belnet.be/?s=download&token=0c5a4deb-4235-47bb-9bc9-d82d683e356d

I'll retry to fit the model and will let you know whether it worked.

Again, thank you for the help!

From: Jan Clemens @.> Sent: donderdag 9 juni 2022 12:29 To: janclemenslab/das @.> Cc: Carmen Winters @.>; Author @.> Subject: Re: [janclemenslab/das] Restart training in COLAB (Issue #30)

Sorry you run into so many issues and happy to help!

I re-fitted a model to our mice data just to make that we didn't introduce any bugs recently. Still works as before - the post processing (fill short gaps, delete short segments) is crucial to get smooth detections. The defaults (0.02 for both parameters) or the values from the paper should produce reasonable annotations.

1. The results file is created at the end of the das.train.train function. If training is interrupted, the file will not be created. I'll update the gist to also generate the results file. Probably a good idea to implement picking up training from an existing model in a future update. 2./3./4. I think there is sth fundamentally off - maybe some bug in the code. Looks like you delete all segments with this and that casues the error. Could you send the model and example data? That would help identifying the source of the error.
2. Including the on- and offsets rarely helps - I think the problem lies elsewhere.

- Reply to this email directly, view it on GitHubhttps://github.com/janclemenslab/das/issues/30#issuecomment-1150950900, or unsubscribehttps://github.com/notifications/unsubscribe-auth/APM37JL2HVC7NH6NGCAQJ4LVOHBP5ANCNFSM5XM6SAAQ. You are receiving this because you authored the thread.Message ID: @.**@.>> [ { @.": "http://schema.org", @.": "EmailMessage", "potentialAction": { @.": "ViewAction", "target": "https://github.com/janclemenslab/das/issues/30#issuecomment-1150950900", "url": "https://github.com/janclemenslab/das/issues/30#issuecomment-1150950900", "name": "View Issue" }, "description": "View this Issue on GitHub", "publisher": { @.": "Organization", "name": "GitHub", "url": "https://github.com" } } ]

[postpop]

Cool - the recordings are not too noisy and the USVs look pretty clear - DAS should work much better, in particular given your large training dataset. Predictions look a bit better when filling gaps <20ms and deleting short segments <1ms but the model still misses many of the USVs. It looks like the USVs are relatively rare in you recordings (in particular for one of the files). You could force the network to focus more on the rare USVs by balancing the data. To do so, check the "Balance classes" box in the training dialog.

[CarmenWinters]

Did you detect separate USVs when you run it? When I run it with these prediction parameters the whole recording is seen as ‘USV’?

From: Jan Clemens @.> Sent: donderdag 9 juni 2022 13:26 To: janclemenslab/das @.> Cc: Carmen Winters @.>; Author @.> Subject: Re: [janclemenslab/das] Restart training in COLAB (Issue #30)

Cool - the recordings are not too noisy and the USVs look pretty clear - DAS should work much better, in particular given your large training dataset. Predictions look a bit better when filling gaps <20ms and deleting short segments <1ms but the model still misses many of the USVs. It looks like the USVs are relatively rare in you recordings (in particular for one of the files). You could force the network to focus more on the rare USVs by balancing the data. To do so, check the "Balance classes" box in the training dialog.

— Reply to this email directly, view it on GitHubhttps://github.com/janclemenslab/das/issues/30#issuecomment-1151002448, or unsubscribehttps://github.com/notifications/unsubscribe-auth/APM37JPFXMII67IWQ7AGCNDVOHIFPANCNFSM5XM6SAAQ. You are receiving this because you authored the thread.Message ID: @.**@.>> [ { @.": "http://schema.org", @.": "EmailMessage", "potentialAction": { @.": "ViewAction", "target": "https://github.com/janclemenslab/das/issues/30#issuecomment-1151002448", "url": "https://github.com/janclemenslab/das/issues/30#issuecomment-1151002448", "name": "View Issue" }, "description": "View this Issue on GitHub", "publisher": { @.": "Organization", "name": "GitHub", "url": "https://github.com" } } ]

[postpop]

Yes, I get ~20 detected USVs for the P7... file using these settings. Maybe you're on an older version of DAS that has some bugs? Best to re-install DAS to ensure you are on the latest version. P7_Mother3_pup1_annotations.csv

[CarmenWinters]

I reinstalled (version 0.25.10) In which .py-script are the segment+’_proposals’ labels created? After looking at your file, I know realize that does not happen in my csv files, even though it is selected. I also saw there is an argument in your window to select time argument (see below). [predict] @.***

Can that underlie the problem? From: Jan Clemens @.> Sent: donderdag 9 juni 2022 15:17 To: janclemenslab/das @.> Cc: Carmen Winters @.>; Author @.> Subject: Re: [janclemenslab/das] Restart training in COLAB (Issue #30)

Yes, I get ~20 detected USVs for the P7... file using these settings. Maybe you're on an older version of DAS that has some bugs? Best to re-install DAS to ensure you are on the latest version. P7_Mother3_pup1_annotations.csvhttps://github.com/janclemenslab/das/files/8870622/P7_Mother3_pup1_annotations.csv

— Reply to this email directly, view it on GitHubhttps://github.com/janclemenslab/das/issues/30#issuecomment-1151107057, or unsubscribehttps://github.com/notifications/unsubscribe-auth/APM37JJLMXITCXIYTRYKUQLVOHVDZANCNFSM5XM6SAAQ. You are receiving this because you authored the thread.Message ID: @.**@.>> [ { @.": "http://schema.org", @.": "EmailMessage", "potentialAction": { @.": "ViewAction", "target": "https://github.com/janclemenslab/das/issues/30#issuecomment-1151107057", "url": "https://github.com/janclemenslab/das/issues/30#issuecomment-1151107057", "name": "View Issue" }, "description": "View this Issue on GitHub", "publisher": { @.": "Organization", "name": "GitHub", "url": "https://github.com" } } ]