Closed alcidesmig closed 3 years ago
Hi @alcidesmig ,
The command you tried to run will not work because the BERT-CRF checkpoint has weights for 1) BERT, 2) Linear classifier
layer of shape (21, 768)
and 3) CRF layer, while the BERT-LSTM-CRF model would have weights for 1) BERT, 2) LSTM, 3) Linear layer named classifier
of shape (21, 200)
and 3) CRF layer. So there is a shape mismatch for the classifier
layer, and the LSTM layer would have random weights.
If I understand your question correctly, you want to load a BERT-CRF checkpoint (trained on FirstHAREM) and continue training on another dataset. Is it right? I think you can do that but you will have to modify the checkpoint to delete the weights of the layers that have different shapes using the same name. I believe this code would solve it (I haven't tested it):
import torch
state_dict = torch.load('path/to/pytorch_weights.bin')
del state_dict['classifier'] # if using LSTM or number of classes are different
del state_dict['crf'] # if number of classes are different
torch.save(state_dict, 'path/to/pytorch_weights.bin') # overwrite weights
This way the model will load the available weights in the checkpoint and leave the new layers with random weights.
Hi, @fabiocapsouza!
I got it by doing:
del state_dict['crf.transitions']
del state_dict['crf.end_transitions']
del state_dict['crf.start_transitions']
del state_dict['crf.classifier.bias']
del state_dict['classifier.bias']
del state_dict['classifier.weight']
Thank you!
Hi,
I tried to delete those state_dict keys, as suggested by @alcidesmig, but the error persists: "RuntimeError: Error(s) in loading state_dict for BertLSTMCRF: size mismatch for classifier.weight: copying a param with shape torch.Size([21, 768]) from checkpoint, the shape in current model is torch.Size([21, 200])."
But: ... bert.encoder.layer.11.output.LayerNorm.bias: torch.Size([768]) bert.pooler.dense.weight: torch.Size([768, 768]) bert.pooler.dense.bias: torch.Size([768]) classifier.weight: torch.Size([21, 768]) classifier.bias: torch.Size([21]) crf.start_transitions: torch.Size([21]) crf.end_transitions: torch.Size([21]) crf.transitions: torch.Size([21, 21])
Can not see this [21,200] layer.
Any clues?
Thanks.
Hi, @CarlosEduardoSaMotta. Can you post the code you used to delete the keys in state_dict?
Hi, @alcidesmig.
import torch
state_dict = torch.load('/content/portuguese-bert/ner_evaluation/bertimbau-base_bert-crf_total/pytorch_model.bin') #'path/to/pytorch_weights.bin')
print('\n'.join([k + ': ' + str(state_dict[k].shape) for k in state_dict.keys()]))
And I've got:
... bert.encoder.layer.11.output.LayerNorm.bias: torch.Size([768]) bert.pooler.dense.weight: torch.Size([768, 768]) bert.pooler.dense.bias: torch.Size([768]) classifier.weight: torch.Size([21, 768]) classifier.bias: torch.Size([21]) crf.start_transitions: torch.Size([21]) crf.end_transitions: torch.Size([21]) crf.transitions: torch.Size([21, 21])
Then:
del state_dict['crf.transitions']
del state_dict['crf.end_transitions']
del state_dict['crf.start_transitions']
#del state_dict['crf.classifier.bias']
del state_dict['classifier.bias']
del state_dict['classifier.weight']
!python run_bert_harem.py \
--bert_model /content/portuguese-bert/ner_evaluation/bertimbau-base_bert-crf_total \
--labels_file data/classes-total.txt \
--do_train \
--train_file data/FirstHAREM-total-train.json \
--valid_file data/FirstHAREM-total-dev.json \
--freeze_bert \
--pooler sum \
--num_train_epochs 50 \
--per_gpu_train_batch_size 2 \
--gradient_accumulation_steps 8 \
--do_eval \
--eval_file data/MiniHAREM-total.json \
--output_dir output_bert-lstm-crf_total
And the error message:
RuntimeError: Error(s) in loading state_dict for BertLSTMCRF: size mismatch for classifier.weight: copying a param with shape torch.Size([21, 768]) from checkpoint, the shape in current model is torch.Size([21, 200]).
Hi again, @CarlosEduardoSaMotta.
What have you used to save the model after deleting classifier.weight?
Oops! I am loading from disk ... Thanks.
Hello!
Talking about NER, is possible to use
BERTimbau Base - BERT-CRF (total scenario, 10 classes)
available inner_evaluation/
to Fine-tuning experiments or is this expected to be done with {mBERT, clean BERTimbau, ...}?Trying to do what i said in the title, I executed:
The result was:
Thank you for your attention.