Closed drunkinlove closed 3 years ago
Please give more info about your code and transformers version. this error can be detected if the version of the transformers is incorrect
The DialoGPT repo uses some old version of the GPT2LMHeadModel (possibly custom): https://github.com/microsoft/DialoGPT/blob/master/lsp_model/modeling_gpt2.py#L72
These weights have a fixed value of -10000, what operation are they a part of? I'm able to finetune the model after I delete the weights, but I can't help wondering how much it'll affect the learned structure of the model
We support only megatron and huggingface interfaces. For consistency use code in or repo
I'm trying to use the small GPT-3 checkpoint for finetuning using the DialoGPT repository, but it says the "transformer.h.N.attn.masked_bias" are unexpected. What layers do these weights refer to?