This PR adds functionality to use a pretrained PyTorch-IE model that allows updating weights of base transformer model. Additionally, it provides an optional parameter, pretrained_pie_model_prefix_mapping, to filter and specify which model layers should be updated. Furthermore, this adds the optional parameter pretrained_pie_model_prefix_mapping that allows to restrict the loading to a subset of parameters via their prefixes.
For instance, to load the weights from a PIE token classification model for further training on conll2003:
Note that we use model.model:model.model as prefix mapping which will result in only loading the weights from the base transformer model, but not the classification head. This is useful to fine-tune on data with a different set of labels.
In addition, this also adds imports for all models and task modules from the pie_modules package to the train script because these are required when loading a such a model vie pytorch_ie.AutoModel.
This PR adds functionality to use a pretrained PyTorch-IE model that allows updating weights of base transformer model. Additionally, it provides an optional parameter,
pretrained_pie_model_prefix_mapping
, to filter and specify which model layers should be updated. Furthermore, this adds the optional parameterpretrained_pie_model_prefix_mapping
that allows to restrict the loading to a subset of parameters via their prefixes.For instance, to load the weights from a PIE token classification model for further training on conll2003:
Note that we use
model.model:model.model
as prefix mapping which will result in only loading the weights from the base transformer model, but not the classification head. This is useful to fine-tune on data with a different set of labels.In addition, this also adds imports for all models and task modules from the pie_modules package to the train script because these are required when loading a such a model vie
pytorch_ie.AutoModel
.