ListOps performance

[dido1998]

On running the ListOps task as-is from the repo, I got a validation performance similar to that reported in the paper but the test performance on results.json is very low:

{"accuracy": 0.17500001192092896, "loss": 3.032956123352051, "perplexity": 20.758506774902344}

I saw that the code is saving the model from the last checkpoint as compared to the model with the best validation performance. Could you detail the evaluation setup used in the paper i.e. for the paper do you evaluate the model from the last checkpoint of from the best validation checkpoint?

Thank you very much! :-)

[sihyun-yu]

Have you solved this problem? I have a similar issue.

[dido1998]

Hi @sihyun-yu, I was not able to solve it

[apuaaChen]

I got a similar issue with the transformer_base. The evaluation accuracy curve is a little bit weird. The highest accuracy reaches 0.3359 at step 2.5k, then it drops to < 0.2. I directly use the default configurations.

[jinfengr]

It seems the issue is fixed with the latest code push. Please add comment if the issue still comes up.

[renebidart]

I found either lowering the learning rate or increasing the batch size was useful for this task. I think their hyperparameters are for a large effective batch size because they run on TPU.

[BalloutAI]

I am still getting the same problem, my validation during training is high on the listops, but when running test_only option, I am getting very low accuracy!

[BalloutAI]

The problem is that the data is shuffled every time the code is ran, so the tokens are changed when running the test script giving a random accuracy.

[yuzhenmao]

The problem is that the data is shuffled every time the code is ran, so the tokens are changed when running the test script giving a random accuracy.

@BalloutAI Hi, I also found this issue: high training accuracy, low test accuracy; I also found if I run training process multiple times, sometimes the model cannot even converge. Could you explain your idea a little bit more? Thank you.