ECMWFCode4Earth / ml_drought

Machine learning to better predict and understand drought. Moving github.com/ml-clim
https://ml-clim.github.io/drought-prediction/
90 stars 18 forks source link

Add variational dropout for recurrent models #103

Closed gabrieltseng closed 4 years ago

gabrieltseng commented 5 years ago

As in https://arxiv.org/pdf/1512.05287.pdf

The motivation is that dropout between hidden timesteps makes it hard for the models to learn patterns across time, since activations are randomly masked.

The solution is to apply the same dropout mask to each timestep in a minibatch.

gabrieltseng commented 5 years ago

Experiments with this are not so promising - chucking on a WIP to see if I can figure out why

tommylees112 commented 5 years ago

For reference (diagram from the paper):

Screenshot 2019-08-12 at 11 49 43

gabrieltseng commented 4 years ago

Closing, since we won't add it at the moment. Can be re-opened if necessary