[Feature Request] Weight Normalization

[p-ruediger]

Hi VinF,

your library is very helpful. Thank you!

Weight normalization might be a way to make SGD-based algorithms suitable for a wider range of environments without the need to manually scale observation vectors and finely tune hyper-parameters. Moreover, the training process might be accelerated considerably for certain environments.

Maybe there is a straightforward way to apply weight normalization to your implementation of actor-critic learning like example code by OpenAI suggests. It appears that only the initialization of the critic would need to be adapted. The example code provides the adaptions for the SGD and Adam optimizers of Keras and the initialization of the critic's parameters based on a single minibatch of data.

The two challenges I see are the following:

• creating the minibatch for the data-dependent initialization of the critic's parameters prior to training
• adapting the RMSprop optimizer of Keras (Should be similar to Adam, though.)

I would really appreciate your thoughts on this.

EDIT: url of the links fixed

[VinF]

Hi,

To introduce weight normalization (or any other form of normalization such as layer normalization or the usage of SeLu), you would need to modify the neural network (e.g. deer/q_networks/NN_keras.py if you use AC_net_keras). You can then also choose appropriate hyper-parameters that are well suited for the given neural network when you instantiate one of the learning algorithms (or even write your own modifications of the provided learning algorithms).

Hope this helps.

PS: the links you provided do not have the right url.

[p-ruediger]

I appreciate your quick response!

Honestly, I was hoping that applying weight normalization to the neural network model provided by NN_keras.py would only require to make changes to AC_net_keras.py in ll. 75-82 similar to the changes made in ll. 63-69 of cifar10_cnn.py in the aforementioned example. As you probably noticed, I'm new to Deep RL. Maybe you can point out the reason(s) for why one would have to change NN_keras.py in order to introduce weight normalization.

[VinF]

Right, in the code you provided they have integrated the normalization in the optimizer instead of modifying the neural network definition. So it should be possible to directly take their optimizers and use it in the learning algorithm (replacing the simple SGD+momentum in AC_net_Keras as you pointed out)

[p-ruediger]

Thanks for the clarification.

Is normalization and weight normalization in particular something you would like to add to your library at this point? If it is not, you may, of course, feel free to close this issue. Otherwise maybe someone else or I will propose an appropriate implementation and post it here at some point.

Anyway, thanks for your help!

[VinF]

We have to be sure that the benefit of adding this outweigh the additional complexity in the code and for the user. If it's only a matter of adding an update rule in AC_net_keras.py in ll. 75-82, then maybe ok. But I guess we would need to add the whole weight normalization code in this library...

[p-ruediger]

Yes, I thought about that, as well. Overall, Weight Normalization might not be worth the additional complexity. One would not only have to add the whole code by OpenAI but also find a solution for the data-dependent initialization. The latter would probably make Weight Normalization only applicable to certain environments or even leave the solution of the initialization problem completely to the user.

Therefore, I decided to go with Batch Normalization for now. The implementation for Keras models is very straightforward and training seems to improve. You are welcome to take a look at my implementation of Batch Normalization for NN_keras.py.

PS: I am not sure if feeding the "training=0" argument to the layer is correct but it seems to work.