Closed nitish1295 closed 1 year ago
By default, a Module has training=True
. I think HuggingFace sets the model to eval
automatically?
If we call eval on the resulting model we get what you expect (see below).
We should probably assign training
base on the status of the old Module. Good catch!
import torch
from transformers import BertForSequenceClassification
from baal.bayesian.dropout import patch_module,unpatch_module
pretrained_weights = 'bert-base-uncased'
use_cuda = torch.cuda.is_available()
model = BertForSequenceClassification.from_pretrained(pretrained_model_name_or_path=pretrained_weights)
print(f"Droputs enabled: {model.dropout.training}") # False here
model = patch_module(model,inplace=False)
print(f"Droputs enabled: {model.dropout.training}") # True here
model = unpatch_module(model,inplace=False).eval()
print(f"Droputs enabled: {model.dropout.training}") # True here
Describe the bug
unpatch_module
does not seem to be working as expected. Once dropouts are enabled I can't switch them off usingunpatch_module
To Reproduce
Run this in Colab
Expected behavior
Once
unpatch_module
is run dropouts should not be in training modeVesion (please complete the following information):
Additional context So based on my understanding while using MCdropout with vanilla Neural nets I frequently used
model.dropout.training
to make sure if dropouts are enabled or not. If this holds for HF BERT models(which I think it does since it is NN based) then essentially this is a bug.