search

r-tensorflow / gpt2

R Interface to OpenAI's GPT-2 model
https://openai.com/blog/better-language-models/
MIT License
27 stars 6 forks source link

Possible values for top_p #9

Open keruiduo opened 3 years ago

keruiduo commented 3 years ago

Hello,

Thanks a lot for making this available. I have a question regarding possible values for top_p. From my understanding reading the paper "The curious case of neural text degeneration" (Holtzman et al., 2020), top_p should be a probability between 0 and 1. Yet when I check the code in 'model.R', you turn the value into an integer with as.integer() before providing it to Python. I tried various values between 0 and 1 with the same seed, and as I expected from the code, the outputs are the same. I get different outputs with values greater than 1, but I can't figure out how things work exactly. Could you please provide some more explanations?

Thanks a lot in advance. Best,

Christophe C.

keruiduo commented 3 years ago

To be more specific: For a given prompt, gpt2(prompt = prompt, batch_size = 1, model = "355M", total_tokens = 150, top_p = top_p, seed = 3) will give me the same output for top_p = 0.1, top_p = 0.5 and top_p = 0.8. I get another output, always the same, for top_p = 1, top_p = 2, top_p = 10 and top_p = 100

Thanks a lot!