learning_rate probblem

[lucaSartore]

hi the project is cool but for lower loss i suggest to use a "cosine total steps" similar to the total steps, for example if you are doing 30k steps you should add this code

pipeline.train_config.optimizer.momentum_optimizer.learning_rate.cosine_decay_learning_rate.warmup_steps=1000 pipeline.train_config.optimizer.momentum_optimizer.learning_rate.cosine_decay_learning_rate.total_steps=29000

[Namburger]

@lucaSartore MRs are welcome!! :) FYI: I'd love to have a drop of explanation or some reference I can read about on this

[lucaSartore]

the learning rate is a value that decide how much edit the value of your neural network, if the learning rate is high you will train your neural network quicker but less precise, if is low you are going to get a precise neural network but in a lot of time ( this is not 100% correct, check this for more informations: https://www.jeremyjordan.me/nn-learning-rate/). in order to get a good result in a decent amount of time you need to decrease the learning rate while you advance with steps. the "cosine decay learning_rate.total_steps=29000" value decide how many steps the learning rate need for pass from the initial value (0.8 in your case) to 0. if you set 10k steps you need to set the learning rate for reaching 0 just before the turning ending. by default this value is set to 400k that is too high if you train it for only 10-30k steps. you can check the learning rate in tensorboard as shown in the picture (with a total learning rate steps of 5k)