search

huggingface / transformers

🤗 Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX.
https://huggingface.co/transformers
Apache License 2.0
134.25k stars 26.84k forks source link

How to make a new line when using gpt2 to generate lyrics? #1002

Closed tine8899 closed 5 years ago

tine8899 commented 5 years ago

❓ Questions & Help

I use pre-trained gpt2 to generate lyrics (text generation). I can generate a long string of lyrics. While how to break the line, I try to add "\n" into it. But it seems that is not a good idea.

Thanks!

LysandreJik commented 5 years ago

Did you pre-train your model while keeping all the line returns or did you remove them? You can keep them during the training so that the model learns to predict them.

If you remove them during training and wish to apply them later on, I guess you can always just create the long string of lyrics and split them with line returns.

tine8899 commented 5 years ago

Actually, I remove them during. Do you think keeping all the line returns during training is a better way? I mean input the whole song (use encode(text) method) instead of splitting each line into echo tokens -> ids .

By the way, Can I input POS of text into the model? Is it necessary?

LysandreJik commented 5 years ago

I think keeping the line returns during your training is a good idea. The model is very likely to learn their position and frequency.

You can input the position_ids in your forward, but it is not necessary. If no position information is provided, the model will create it on its own.

stale[bot] commented 5 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.