search

speechbrain / benchmarks

This repository contains the SpeechBrain Benchmarks
Apache License 2.0
83 stars 35 forks source link

Early Stopping for HParam Optimization when model is not converging #22

Closed Drew-Wagner closed 6 months ago

Drew-Wagner commented 6 months ago

I'm running the run_hparam_optimization.sh in the MOABB benchmarks using a variation of EEGNet that I developed (for the BNCI2014001 dataset).

Often the hyperparameters chosen by orion lead to a model which does not converge, and which has an accuracy which is no better than random guessing. However, the program still runs through all the epochs even though it is obviously not converging.

Is there a way to prevent this with the currently available options (other than changing changing the orion flags to adjust the search space)? If not is this a feature that might be useful to add?

mravanelli commented 6 months ago

Hi, I suggest writing yourself a piece of code that stops the execution if NaN or Inf is detected. Note that this is not a healthy condition, especially if it happens very frequently. It might be due to an unideal setting of the ranges of the hparams set in Orion or a bug in the model. Before running the hparam tuning phase, my suggestion is to gain some insight into meaningful ranges for your hyperparameters by manually running single experiments of single subjects.

On Tue, Feb 20, 2024 at 12:08 PM Drew Wagner @.***> wrote:

I'm running the run_hparam_optimization.sh in the MOABB benchmarks using a variation of EEGNet that I developed (for the BNCI2014001 dataset).

Often the hyperparameters chosen by orion lead to a model which does not converge, and which has an accuracy which is no better than random guessing. However, the program still runs through all the epochs even though it is obviously not converging.

Is there a way to prevent this with the currently available options (other than changing changing the orion flags to adjust the search space)? If not is this a feature that might be useful to add?

— Reply to this email directly, view it on GitHub https://github.com/speechbrain/benchmarks/issues/22, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEA2ZVV5V6EJAFEYHHFES3TYUTKA7AVCNFSM6AAAAABDRRD7TOVHI2DSMVQWIX3LMV43ASLTON2WKOZSGE2DIOBWG42TENY . You are receiving this because you are subscribed to this thread.Message ID: @.***>

Drew-Wagner commented 6 months ago

Ok, thanks! I will look into adjusting the search space.

Drew-Wagner commented 6 months ago

I just wanted to follow up and say that it did end up being a small bug in the model. Once fixed, the hyper parameter optimization started working great, and appears to be showing some positive results so far. I look forward to sharing the results once the experiment is complete!