XtractOpen / Meganet.jl

A fresh approach to deep learning written in Julia
http://www.xtract.ai/
Other
14 stars 9 forks source link

speedup singlelayer #57

Closed lruthotto closed 6 years ago

lruthotto commented 6 years ago
coveralls commented 6 years ago

Coverage Status

Coverage decreased (-0.3%) to 78.543% when pulling 11e91a249891d517f7aae99f79a582e348261527 on speedupSingleLayer into dfcb44275ec9b9c66f3d3845808514837c4d73c2 on dev.

klensink commented 6 years ago

Could you submit this to the dev branch?

DavidBegert commented 6 years ago

@lruthotto not sure if you can see my other comments, but what exactly do we need to store when running batch norm? In the backward pass functions we are running batchNormNN with doDerivatives=true which uses a lot of memory storing the 3 Ys at each point of batch norm (one at start, one after norm layer, one after affine scaling layer). Causing memory footprint to grow a lot.