search

openai / finetune-transformer-lm

Code and model for the paper "Improving Language Understanding by Generative Pre-Training"
https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf
MIT License
2.14k stars 499 forks source link

Unable to adapt language model #7

Closed chiayewken closed 6 years ago

chiayewken commented 6 years ago

Thank you for the great research and code! Regarding this section in the research paper:

"For CoLA (linguistic acceptability), examples are scored as the average token log-probability the generative model assigns and predictions are made by thresholding. For SST-2 (sentiment analysis), we append the token very to each example and restrict the language model’s output distribution to only the words positive and negative and guess the token it assigns higher probability to as the prediction. For RACE (question answering), we pick the answer the generative model assigns the highest average token log-probability when conditioned on the document and question. For DPRD [46] (winograd schemas), we replace the definite pronoun with the two possible referrents and predict the resolution that the generative model assigns higher average token log-probability to the rest of the sequence after the substitution."

I have tried to adapt the language model part of the model in the code to perform the above mentioned tasks. For instance, for CoLA, I fed in the encoded sentences and evaluated the results by thresholding the lm_losses output from the language model. However, the best Matthews correlation coefficient obtained is 0.015, far short of the 0.479 achieved in the paper. How exactly was the model configured to perform the task? Was it purely through the language model output, or was the supervised classification head used?

Newmu commented 6 years ago

Supervised classification was used. You should be able to achieve ~0.2 with the LM thresholding, however. Are you sure you are computing the average token log-prob?