Closed gabrieltseng closed 4 years ago
Experiments with this are not so promising - chucking on a WIP to see if I can figure out why
For reference (diagram from the paper):
Closing, since we won't add it at the moment. Can be re-opened if necessary
As in https://arxiv.org/pdf/1512.05287.pdf
The motivation is that dropout between hidden timesteps makes it hard for the models to learn patterns across time, since activations are randomly masked.
The solution is to apply the same dropout mask to each timestep in a minibatch.