Open cosmozhang opened 9 years ago
Thanks! Good catch. Just fixed a typo in the ipynb.
Hi Jonathan,
I am trying to implement dropout in the theano.scan, but without
using your package, since I have written most part of the rnn. Do you have any suggestions? I read your codes and instructions, but still have some questions. E.g, since masks is non_sequences, what will be the input in the theano.function? Thanks very much!
Cosmo
On 04/20/2015 10:18 PM, Jonathan Raiman wrote:
Thanks! Good catch. Just fixed a typo in the ipynb.
— Reply to this email directly or view it on GitHub https://github.com/JonathanRaiman/theano_lstm/issues/11#issuecomment-94615352.
Best, Cosmo Zhang
Hey Cosmo,
To get dropout to work I've done the following:
Get some random number generator:
import theano, theano.tensor as T
srng = theano.tensor.shared_randomstreams.RandomStreams(1234)
Then create the mask you want to use:
drop_prob = T.scalar()
shape = (300, 300)
mask = T.cast(
srng.binomial(n=1, p=1-drop_prob, size=shape),
theano.config.floatX
)
And then in your scan:
result, updates = theano.scan(fn = step,
sequences = [x],
outputs_info = etc...,
non_sequences = [mask])
error = etc...
Your theano function can now take as inputs:
func = theano.function([x, target, drop_prob], (result[-1] - target)**2)
As long as the random variables aren't generated within the scan op you should be fine. I've had problems whenever I tried created a new set of binomials on every time step or something else that's fancy. But the frozen dropout values as shown above worked for me.
Thank you very much, I solved it last night in another way:
import theano, theano.tensoras T srng= theano.tensor.shared_randomstreams.RandomStreams(1234)
drop_prob= T.scalar() shape= (shape1, 300,300) #shape1 is the length of input sequence mask= T.cast( srng.binomial(n=1,p=1-drop_prob,size=shape), theano.config.floatX )
and then
|result, updates = theano.scan(fn = step, sequences = [x, mask], outputs_info = etc..., non_sequences = None)|
Do you think it is also plausible?
And in your setting, how did yo write step function?
Thank you very much!
On 4/21/2015 10:47 PM, Jonathan Raiman wrote:
Hey Cosmo,
To get dropout to work I've done the following:
Get some random number generator:
import theano, theano.tensoras T srng= theano.tensor.shared_randomstreams.RandomStreams(1234)
Then create the mask you want to use:
drop_prob= T.scalar() shape= (300,300) mask= T.cast( srng.binomial(n=1,p=1-drop_prob,size=shape), theano.config.floatX )
And then in your scan:
|result, updates = theano.scan(fn = step, sequences = [x], outputs_info = etc..., non_sequences = [mask])
error = etc... |
Your theano function can now take as inputs:
func = theano.function([x, target, drop_prob], (result[-1] - target)**2) As long as the random variables aren't generated within the scan op you should be fine. I've had problems whenever I tried created a new set of binomials on every time step or something else that's fancy. But the frozen dropout values as shown above worked for me.
— Reply to this email directly or view it on GitHub https://github.com/JonathanRaiman/theano_lstm/issues/11#issuecomment-95006889.
The ipynb was not fixed..