RES-863: Use 'Layer Norm' instead of 'Batch Norm' on linear networks

numenta / htmresearch

Experimental algorithms. Unsupported.

GNU Affero General Public License v3.0

224 stars 144 forks source link

RES-863: Use 'Layer Norm' instead of 'Batch Norm' on linear networks #967

Open lscheinkman opened 5 years ago

lscheinkman commented 5 years ago

@subutai Please review. The main change in this PR is the switch from BatchNorm to LayerNorm on LinearSDR module. See https://arxiv.org/abs/1607.06450

subutai commented 5 years ago

This doesn’t affect any of our existing sparse networks right?

lscheinkman commented 5 years ago

If affects LinearSDR. We can now use batch_size=1

lscheinkman commented 5 years ago

However we cannot use LayerNorm on CNNSDR2d. It is only good for Linear and RNN networks