XLNet Generation appears to reference padding text in run_generation script

[thesamuel]

When generating with XLNet in the run_generation.py script, the outputs seem to reference the context from the padding text. For instance, given the prompt "We propose a", XLNet generates "We propose a boy Go Ya Ya, a young Iriel Farg, to be named Rasputin."

This seems to reference the padding text: https://github.com/huggingface/transformers/blob/0866669e751bef636fa693b704a28c1fea9a17f3/examples/text-generation/run_generation.py#L62-L71

From what I understand, this padding text should not influence the generation, since the padding ends with an end of sentence token. Is this behavior expected?

Full command I used for reference:

python -m examples.run_generation --model_type xlnet --model_name_or_path xlnet-base-cased --prompt "We propose a"

[patrickvonplaten]

Hi @thesamuel,

Ideally, the padding text should not influence the outcome, but this is more a hack to make XLNet work with short prompts, than actual science.

Also note that it is recommended now to use the TextGeneration Pipeline instead of the run_generation script:

from transformers import pipeline
generator = pipeline("text-generation", model="xlnet-base-cased")
print(generator("We propose a "))

Note: This works well for XLNet only after merging PR: #4686. So for pipelines to work, you either have to wait a bit or work on the branch of the PR: #4686.

As a default the pipeline employs sampling instead of greedy search. You might also want to play around with the generation hyperparameters here a bit for better results. To learn more about how to effectively use the many parameters for text generation, you might want to take a look at: https://huggingface.co/blog/how-to-generate