search

openai / weightnorm

Example code for Weight Normalization, from "Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks"
https://arxiv.org/abs/1602.07868
MIT License
365 stars 118 forks source link

Compatibility with Keras 2.2.3 and Tensorflow 1.11.0 #13

Open krasserm opened 6 years ago

krasserm commented 6 years ago

This PR contains the required changes to get SGDWithWeightnorm and AdamWithWeightnorm running with Keras 2.2.3 and Tensorflow 1.11.0. I tested cifar10_cnn.py with both updated optimizers and training shows reasonable convergence. For testing the constraint branch in the optimizer code I temporarily added a kernel_constraint to the first Dense layer in cifar10_cnn.py (not part of the PR though) with reasonable results. I'm happy to include the test code in the PR if needed.

Update: It seems that this project is not active any more. I'll therefore continue maintaining the Keras 2.x version here. For more information about weight normalization implementation options for Keras see this blog post.

krasserm commented 6 years ago

@TimSalimans instead of changing the original implementation for Keras 1 I created a new keras_2 directory containing all changes needed for compatibility with Keras 2. The old keras directory was renamed to keras_1. Would appreciate your feedback on the PR.

krasserm commented 6 years ago

@TimSalimans you can see the actual changes from Keras 1 to Keras 2 in the second commit.