search

bdzyubak / tensorflow-sandbox

A repository for studying applications of Deep Learning across fields, and demonstrating samples of my code and project managment
0 stars 0 forks source link

Implement module to try to bypass local minima #12

Closed bdzyubak closed 2 years ago

bdzyubak commented 2 years ago

A model optimizer can get stuck in a local minimum yielding sub-optimal accuracy. Some solutions to find a better local minimum are: 1) Retrain from scratch many times. - slow but easy to do; not implementing dedicated functionality for now. 2) Increase the learning rate by a large amount after the model appears to have converged. 3) Decrease batch size which will make training more random and can bypass a local minimum.

This request is to implement methods 2 and/or 3 as a method callable on a trained model. Several iterations should be done and, if any of them yield a better local minimum, the originally trained model should be overwritten.

bdzyubak commented 2 years ago

The default batch size has been set to 32 examples, so decreasing it further runs a risk of being too small and diverging. The learning rate, on the other hand, already uses a scheduler which will progressively decrease. So, going with option 2 for the method to bypass minima.

bdzyubak commented 2 years ago

This module has been implemented in: 9617a644cf2b610a15e2a3c3294e940bc69f7255

The usage is as follows: model = InitializeModel(model_name,dataset,model_path) history = model.run_model() model.try_bypass_local_minimum(n_times=3)