Open matthew-jurewicz opened 4 years ago
Can you give more details on what you're trying to do? What is the shape of your training data? How long is long? The readme suggests that you want to turn on random truncation when you're going to be generating long text, as the transformer doesn't generalize well to different text sizes. I'd highly recommend that.
I'd like something coherent with as many as 1 million tokens, and Google says Reformer can ingest something this long. Also, you're saying I can't just pad to maximum sequence length?
I have a long piece of text where the end depends on the start. Should I pass the whole text in one batch for training? Also, does this affect generating the same thing?