issue 21 Add AdamW to the optimization search space

PathologyDataScience / glimr

A simplified wrapper for hyperparameter search with Ray Tune.

Apache License 2.0

1 stars 0 forks source link

issue 21 Add AdamW to the optimization search space #25

Closed cooperlab closed 1 year ago

cooperlab commented 1 year ago

Newer tf.keras.optimizers incorporate exponential moving averaging of weights.

Using this requires bumping the TF version to 2.11. Also, many existing optimizers are moved temporarily to tf.keras.optimizers.experimental.

cooperlab commented 1 year ago

Note - the steps/iterations in ema_overwrite_frequency refers to batches or application of gradients. So this parameter determines whether averaging should be applied after so many batches (int), or only once per epoch (None).

review-notebook-app[bot] commented 1 year ago

Check out this pull request on

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

cooperlab commented 1 year ago

This is a complete draft that adds moving averaging for all previously supported optimizers. The example notebook runs.