Backprop in evaluate mode

[ruotianluo]

I know it is memory efficient in evaluate mode. However, I need to backprop in evaluate model. I tried use remember('both'), however it doesn't help. What's the right way?

[sudongqi]

evaluate mode doesn't store the hidden state during forward (which is needed for backprop), so why don't use training mode?

[ruotianluo]

Consider nn.Sequencer(nn.Dropout()). How can I make it work correctly.

[JoostvDoorn]

If the only non deterministic module is Dropout for which you need to calculate gradients then you could disable Dropout and use training mode instead.

[ruotianluo]

@JoostvDoorn It should work. Thank you for your advice.

[ruotianluo]

But, still, there should be a switch to turn on and off the memory reduction. However I can't find where it is.

[sudongqi]

If you look into the Dropout implementation, there is a train flag. You can set a handle when you define the model, and call x.train = false to turn it off.

local model = nn.Sequential()
model:add(nn.Linear(50,50))
local x = nn.Dropout(0.5)
model:add(x)
x.train = false
print (x.train) --> should print false
model:training()
print(x.train) --> should print true

[ywelement]

Thanks, @JoostvDoorn, @sudongqi. You're right, setting the train flag is a quick way to enable backprop.

[ywelement]

Looks like we got a solution here. I'm closing this ticket.

[ruotianluo]

@ywelement is there a simple way to turn off the memory sharing?

[ruotianluo]

@JoostvDoorn @sudongqi Just turn dropout module into evaluate mode doesn't work. The memory sharing is in the wrapper. The way I did, is manually turn the p to 0 in evaluation so that the sharing won't change the result.