search

CSTR-Edinburgh / merlin

This is now the official location of the Merlin project.
http://www.cstr.ed.ac.uk/projects/merlin/
Apache License 2.0
1.31k stars 440 forks source link

Learning rate with keras #384

Closed theabc123 closed 5 years ago

theabc123 commented 6 years ago

I switched to Keras backend, and I noticed that changing the learning_rate in the config files has no effect on the training process (tried 0.000001 to 0.9). I always get the exact same validation error values for the different epochs. How can I pass a given learning_rate value to Merlin when I use ? [Architecture] hidden_layer_size : [ 2048, 1024, 512] hidden_layer_type : [ 'BLSTM', 'BLSTM', 'BLSTM'] switch_to_tensorflow: False switch_to_keras: True

m-toman commented 6 years ago

That's because the learning rate isn't passed to the optimizer here https://github.com/CSTR-Edinburgh/merlin/blob/master/src/keras_lib/model.py#L206

theabc123 commented 6 years ago

@m-toman thanks a lot, I came to the same conclusion, for now I changed line 70 in src/keras_lib/model.py :
self.optimizer = keras.optimizers.Adam(lr=0.00002, beta_1=0.9, beta_2=0.999, epsilon=None, decay=0.0, amsgrad=False)

And it works.

m-toman commented 6 years ago

Yes, I did the same. But as that ignores the "optimizer" configuration setting I never made a PR :). I've also attached https://github.com/bckenstler/CLR with good results.