Open stefan-it opened 2 years ago
Hi @alexeib , sorry for bothering you again, but could you a look at the missing lm_head
. I could also share the pretrained checkpoint if necessary :hugs:
Me either have that problem, too.
I am trying to convert own pretrained data2vec-text to Hugginface form, but it looks there are many parameter mismatches on code. (Because of AttributeError, i import the model(Data2vecTextModel) using data2vec-text in fairseq library, not in transformers library as suggested)
It seems many troubleshooting parts are included in 'convert_data2vec_text_original_pytorch_checkpoint_to_pytorch.py'
Anyway, i'd like to know how i transfer 'regression_head' and deal with 'lm_head' like @stefan-it asked
Hi,
after trying to convert an own pretrained model checkpoint into Transformers library with the following code snippet:
it seems that
lm_head
is missing in the encoder with the current implementation.Here's a model diff between the original released text model and an own pretrained one:
Official text model:
Pretrained one:
After looking at the
Data2VecTextEncoder
it seems thatbuild_lm_head
is never called:https://github.com/facebookresearch/fairseq/blob/5307a0e078d7460003a86f4e2246d459d4706a1d/examples/data2vec/models/data2vec_text.py#L280-L323
However, here's the corresponding call in the
RobertaEncoder
:https://github.com/facebookresearch/fairseq/blob/5307a0e078d7460003a86f4e2246d459d4706a1d/fairseq/models/roberta/model.py#L555-L564
My questions are now:
lm_head
is really missing in the current implementationregression_head
(that does not exist in released text model)Many thanks in advance!