Training-test data contamination in the essay example

[mgrankin]

I was excited to see outstanding perplexities (8 and 3) for essays generation. It’s so good that I decided to check for data leakage. Unfortunately, your training set data contains all validation set data. The resulting model could be overfit if you did more than one epoch (it’s impossible to know without a proper validation set).

[king-menin]

You are right. It was only example with very small dataset. In our other experiments perplexity is around 13-18.