Remove momentum from Layers

HuwCampbell / grenade

Deep Learning in Haskell

BSD 2-Clause "Simplified" License

1.44k stars 84 forks source link

Open HuwCampbell opened 7 years ago

HuwCampbell commented 7 years ago

Momentum shouldn't be stored in the layers any more. This will free us up to use a broader set of optimisation algorithms. We will however need to provide a class for fast updates and manipulations of learnable parameters.
Gradient associated type family shouldn't exist, we'll just return a Network with gradient weights.
randomNetwork shouldn't exist. Networks where all layers have a Random instance will also have a Random instance.

claudeha commented 5 years ago

Moreover, I noticed momentum is not serialized, which means saving/resuming training between sessions may be problematic.

HuwCampbell commented 5 years ago

Indeed.