AutoResearch / EEG-GAN

Other
22 stars 3 forks source link

Adding a Training Exit Function #39

Open chadcwilliams opened 1 year ago

chadcwilliams commented 1 year ago

A user of our commented:

Hi Chad, I did have one idea. It seems based on my review of the toolbox that the user needs to determine the number of epochs that get the losses to converge to zero. This kind of reminds me of ICA training, except in the case of ICA training (at least in the eeg analysis toolboxes I use) the training ceases only when some convergence criteria has been met. As someone who does not work with GANs regularly, figuring out what is “adequate” convergence could be a point of confusion. It might be useful if there is some way to have a convergence criteria (i.e., losses hover near 0 for x epochs), and once this is met the training automatically ceases. There may be other issues with that approach that I don’t foresee though and I know y’all have a training continuation option included to help address the problem of needing more training after training for a certain amount of epochs. Just a thought!

whyhardt commented 7 months ago

Added KeyboardBoard-Interrupt exit method to GAN and AE Training

chadcwilliams commented 7 months ago

That's a good feature to add, but I do not believe that is what the issue means. The comment is asking for a way to automatically stop training early if training performance stops improving.

In normal losses that converge towards 0, we could create an easy threshold exit function where if the loss has not decreased by a threshold t within the last n epochs, then we consider training complete and the model training ends.

With the GAN, we don't have a convergence towards 0 in this way, but instead we have our losses oscillating around zero. We could maybe take the absolute of the loss functions and implement the threshold method above? Alternatively, we could develop another method that relies on variability in the loss as variability in losses seem to decrease as training becomes successful.

Both of these methods would need us to determine an appropriate default though, which may take some time.

chadcwilliams commented 7 months ago

I am going to change this to priority 3: optional because it's not a necessary feature.