search

google / automl

Google Brain AutoML
Apache License 2.0
6.19k stars 1.45k forks source link

How to use adaptive learning rate? #1030

Open ChulanZhang opened 3 years ago

ChulanZhang commented 3 years ago

Hi, I am trying to use adaptive learning rates for the training process. Here is my config file: ''' num_classes: 31 label_map: {1: airplane, 2: antelope, 3: bear, 4: bicycle, 5: bird, 6: bus, 7: car, 8: cattle, 9: dog, 10: demestic_cat, 11: elephant, 12: fox, 13: giant_panda, 14: hamster, 15: horse, 16: lion, 17: lizard, 18: monkey, 19: motorcycle, 20: rabbit, 21: red_panda, 22: sheep, 23: snake, 24: squirrel, 25: tiger, 26: train, 27: turtle, 28: watercraft, 29: whale, 30: zebra} learning_rate: 0.008 ''' What should I add to use adaptive learning rates for the training? Should it be: moving_average_decay: 0.9998? Or it is already a default setting?

Any information would be greatly appreciated.

Thanks

AnranXu commented 3 years ago

Did you solve that? I also wonder about the right setting for this. I found a param named h.lr_decay_method = 'cosine'. How to set other types of decay methods?

ChulanZhang commented 3 years ago

Did you solve that? I also wonder about the right setting for this. I found a param named h.lr_decay_method = 'cosine'. How to set other types of decay methods?

I haven't figured it out. It seems that the existing code can automatically decrease the learning rate during the training. However, I am not so sure.