Training hyperparameters optimization

[Hobart10]

Hi, thank you so much for developing such a nice framework! I'm new to model training and I'm trying to train the model on a dataset with about 6000 vocalizations, consider there are around 10 types and the dataset size is still small, I want to find the optimal hyperparameter combination to get a high performance model, yet there are several issues in my training tests:

1. Onset detection is more accurate than offset, the offsets can be really shifted.
2. Higher chance for relative short calls(200-500ms) to be omitted.
3. Sometimes val f1 and recall value are not increasing.
4. It seems "omit empty clip prob" doesn't influence the performance much, maybe just related to the dataset size? Several values from 0.0001 to 0.9 was tested.

The following example figures are from one training with:

lr=0.00005
batch-size=4
omit-empty-clip-prob=0.02

And the inference result will be omitting many more vocalizations. Could you kindly help me in resolving the issues and gaining a deeper understanding of the model? Great appreciation for any suggestions you may have : )

[benjaminsshoffman]

Hi, sorry for a slow reply!

To answer your question more generally, how to do a good hyperparameter optimization: I would start by sweeping a few values of learning rate and batch size. You might also try to train for more epochs -- it looks like you have 10 currently.

1. Onset detection is more accurate than offset, the offsets can be really shifted.

This is something we've observed as well, and it is probably because offsets and onsets are handled differently by the model. The model first predicts the onset, and then predicts the duration of the box to get the offset. You might try to increase the value of the hyperparameter --lamb, which controls the strength of the regression loss.

In a future version, we might adjust how this is handled. It's good to know this is an issue that's important to users.

2. Higher chance for relative short calls(200-500ms) to be omitted.

We have found that it is possible to increase model performance on short calls by changing how fast the audio is played back to the model. To do this, change the --sr from the default value 16000 to 32000. Everything else should work as usual.

3. Sometimes val f1 and recall value are not increasing.

This is tricky to troubleshoot--try some of the hyperparameter adjustments above. If your val set has few vocalizations in it, it may be difficult to see when the model is improving.

4. It seems "omit empty clip prob" doesn't influence the performance much, maybe just related to the dataset size? Several values from 0.0001 to 0.9 was tested.

This is an option we added for datasets that have lots and lots of silence in them. It omits empty clips from each training epoch, just to increase training speed. If your dataset is relatively small and doesn't include lots of silence, then you should set this to 0.

Please let us know if you have more questions, I will try to pay closer attention to github issues.