Closed ishandutta0098 closed 7 months ago
Update_1: fix typo of method name inputs_labels_from_batch
, input
should be plural here
Update_2: missing invocation of super()
in the subclass of nn.Module
Hi @ishandutta0098,
You should return only 2 values from the method inputs_labels_from_batch()
, you can try this snippet to see whether it works or not:
class CustomTrainIter(TrainDataLoaderIter):
def inputs_labels_from_batch(self, batch_data):
inputs = batch_data["ids"], batch_data["mask"], batch_data["token_type_ids"]
labels = batch_data["targets"]
return inputs, labels
And the following code shows why we have to format the output like that: https://github.com/davidtvs/pytorch-lr-finder/blob/acc5e7ee7711a460bf3e1cc5c5f05575ba1e1b4b/torch_lr_finder/lr_finder.py#L371-L378
TrainDataLoaderIter
. Content of inputs
and labels
are determined to the implementation of TrainDataLoaderIter.inputs_labels_from_batch()
.inputs
as input data of model.forward()
. In your case, signature of model.forward()
should be something like this:
class Model(nn.Module):
def forward(self, ids=None, mask=None, token_type_ids=None):
# ...
Since the method invocation self.model(inputs)
(in L377) takes only 1 variable, we have to pack all custom inputs into 1 value. And that's why the inputs
is written as inputs = batch_data["ids"], batch_data["mask"], batch_data["token_type_ids"]
in the snippet above.
Besides, since your model.forward()
takes only keyword arguments, you might want to guarantee the order of inputs. To achieve this, you can create a wrapper for your model, and use it in lr_finder.range_test()
:
class ModelWrapper(nn.Module):
def __init__(self, model):
super(ModelWrapper, self).__init__()
self.model = model
def forward(self, inputs):
# You can organize the order of inputs here, or even handle more complicated forward logic for model.
ids, mask, token_type_ids = inputs
return self.model(ids=ids, mask=mask, token_type_ids=token_type_ids)
# ... code for utilizing lr_finder
my_model = Model(...) # your original model
model_wrapper = ModelWrapper(my_model)
lr_finder = LRFinder(
model_wrapper,
optimizer,
criterion,
device = config.DEVICE
)
If there is any further question, please feel free to let me know!
Hi @NaleRaphael, I tried as you said but I now I am getting two different errors.
Original Model
class BERTModel(nn.Module):
def __init__(self):
super(BERTModel, self).__init__()
self.bert = transformers.BertModel.from_pretrained("bert-base-multilingual-uncased")
self.bert_drop = nn.Dropout(0.3)
self.out = nn.Linear(768 * 2, 1) # *2 since we have 2 pooling layers
def forward(self, ids, mask, token_type_ids):
o1, _ = self.bert(
ids,
attention_mask=mask,
token_type_ids=token_type_ids
)
mean_pooling = torch.mean(o1, 1)
max_pooling, _ = torch.max(o1, 1)
cat = torch.cat((mean_pooling, max_pooling), 1)
bo = self.bert_drop(cat)
output = self.out(bo)
return output
Model Wrapper
class ModelWrapper(nn.Module):
def __init__(self, model):
self.model = model
def forward(self, inputs):
ids, mask, token_type_ids = inputs
return self.model(
ids = ids,
mask = mask,
token_type_ids = token_type_ids
)
CustomTrainerIter
class CustomTrainIter(TrainDataLoaderIter):
def input_labels_from_batch(self, batch_data):
inputs = batch_data["ids"], batch_data["mask"], batch_data["token_type_ids"]
labels = batch_data["targets"]
return inputs, labels
lr_finder Implementation
criterion = nn.BCEWithLogitsLoss()
lr_finder = LRFinder(
model_wrapper,
optimizer,
criterion,
device = config.DEVICE
)
custom_train_iter = CustomTrainIter(train_data_loader)
lr_finder.range_test(
custom_train_iter,
end_lr = 10,
num_iter = 100,
step_mode = "exp"
)
lr_finder.plot()
Error
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
<timed exec> in <module>
<timed exec> in run()
<ipython-input-64-162d67c07fe1> in __init__(self, model)
1 class ModelWrapper(nn.Module):
2 def __init__(self, model):
----> 3 self.model = model
4
5 def forward(self, inputs):
/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py in __setattr__(self, name, value)
803 if modules is None:
804 raise AttributeError(
--> 805 "cannot assign module before Module.__init__() call")
806 remove_from(self.__dict__, self._parameters, self._buffers, self._non_persistent_buffers_set)
807 modules[name] = value
AttributeError: cannot assign module before Module.__init__() call
Model Wrapper
class ModelWrapper(nn.Module):
def __init__(self, model):
super(ModelWrapper, self).__init__()
self.model = model
def forward(self, inputs):
ids, mask, token_type_ids = inputs
return self.model(
ids = ids,
mask = mask,
token_type_ids = token_type_ids
)
CustomTrainerIter
class CustomTrainIter(TrainDataLoaderIter):
def input_labels_from_batch(self, batch_data):
inputs = batch_data["ids"], batch_data["mask"], batch_data["token_type_ids"]
labels = batch_data["targets"]
return inputs, labels
lr_finder Implementation
criterion = nn.BCEWithLogitsLoss()
lr_finder = LRFinder(
model_wrapper,
optimizer,
criterion,
device = config.DEVICE
)
custom_train_iter = CustomTrainIter(train_data_loader)
lr_finder.range_test(
custom_train_iter,
end_lr = 10,
num_iter = 100,
step_mode = "exp"
)
lr_finder.plot()
Error
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<timed exec> in <module>
<timed exec> in run()
/opt/conda/lib/python3.6/site-packages/torch_lr_finder/lr_finder.py in range_test(self, train_loader, val_loader, start_lr, end_lr, num_iter, step_mode, smooth_f, diverge_th, accumulation_steps, non_blocking_transfer)
318 train_iter,
319 accumulation_steps,
--> 320 non_blocking_transfer=non_blocking_transfer,
321 )
322 if val_loader:
/opt/conda/lib/python3.6/site-packages/torch_lr_finder/lr_finder.py in _train_batch(self, train_iter, accumulation_steps, non_blocking_transfer)
369 self.optimizer.zero_grad()
370 for i in range(accumulation_steps):
--> 371 inputs, labels = next(train_iter)
372 inputs, labels = self._move_to_device(
373 inputs, labels, non_blocking=non_blocking_transfer
/opt/conda/lib/python3.6/site-packages/torch_lr_finder/lr_finder.py in __next__(self)
57 try:
58 batch = next(self._iterator)
---> 59 inputs, labels = self.inputs_labels_from_batch(batch)
60 except StopIteration:
61 if not self.auto_reset:
/opt/conda/lib/python3.6/site-packages/torch_lr_finder/lr_finder.py in inputs_labels_from_batch(self, batch_data)
34 "Your batch type is not supported: {}. Please inherit from "
35 "`TrainDataLoaderIter` or `ValDataLoaderIter` and override the "
---> 36 "`inputs_labels_from_batch` method.".format(type(batch_data))
37 )
38
ValueError: Your batch type is not supported: <class 'dict'>. Please inherit from `TrainDataLoaderIter` or `ValDataLoaderIter` and override the `inputs_labels_from_batch` method.
Hi, @ishandutta0098
Sorry for the late response, things are busy here.
That's my fault, there were some typos in previous comment, and the error raised in your case 1 indicating the missing invocation of super()
. The model wrapper should be like this:
class ModelWrapper(nn.Module):
def __init__(self, model):
super(ModelWrapper, self).__init__()
self.model = model
def forward(self, inputs):
ids, mask, token_type_ids = inputs
return self.model(ids=ids, mask=mask, token_type_ids=token_type_ids)
And the typos locate at the method name input_labels_from_batch
, it should be inputs_labels_from_batch
instead. (input
should be plural)
I'll update the that comment in case any one copy those incorrect code. Hope it work this time, and please do let me know if there is any further problem.
Closing due to inactivity
I am training a multilingual-bert model for a sentiment classification task. My torch dataset returns a dictionary. I tried to run
lr_finder.range_test(....)
with and withoutTrainDataLoaderIter
but I get the sameValueError
both times.Torch Dataset
Run Function
%%time
def run():
if name == "main": run()
Error