search

learnables / learn2learn

A PyTorch Library for Meta-learning Research
http://learn2learn.net
MIT License
2.59k stars 348 forks source link

maml.training=True but maml.module.training=False #396

Closed jkang1640 closed 1 year ago

jkang1640 commented 1 year ago

I've noticed that when using a maml wrapper, without explicitly doing maml.train(), maml.training returns True but maml.module.training returns False. Is this an expected behavior..?

I am not sure if dropout is activated or not during the training. To make sure, I just specfy maml.train() (this updates maml.module.training) but I don't see explcit maml.train() in most of the example codes and I don't know if this is necessary to do so.

Can anybody clariy this point please?

seba-1511 commented 1 year ago

Hello @jkang1640,

There's no clear consensus on how dropout should be used with MAML. The standard architecture (CNN4) doesn't use it, but I think you're safe if being explicit about whether you want the model in train or eval mode.

Note that this issue was also raised in the context of batch norm. Chelsea Finn stated the original paper keeps batch norm in train mode. (See: https://github.com/cbfinn/maml/issues/9)