search

yangkevin2 / naacl-2021-fudge-controlled-generation

MIT License
91 stars 16 forks source link

Perplexity Evaluation #2

Open HappyGu0524 opened 3 years ago

HappyGu0524 commented 3 years ago

Hi, I'm interested in your paper. I encountered a problem when calculating the GPT perplexity. As in eval_topic_metrics.py, you used the pre-trained model openai-gpt as the LM. When I run this code, I got such warnings: Some weights of OpenAIGPTLMHeadModel were not initialized from the model checkpoint at openai-gpt and are newly initialized: ['lm_head.weight'] You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.

Whether it means that the GPT PPL is kind of a random number?

yangkevin2 commented 3 years ago

Hi,

I seem to get that warning whenever initializing one of these models. If you actually use that pretrained model directly to generate text, it should give reasonable results, though; let me know if that's not the case.