Closed mlpen closed 3 years ago
Is there a plan to release this? A benchmark dataset without reproducibility seems quite useless.
Or am I making a mistake with training using the existing configs? The README says: "Currently we have released all the necessary code to get started and run our benchmarks on vanilla Transformers."
Hi,
Thanks for the question and sorry for the issue. Please use the config files in the code for each model to reproduce the results as they are the most updated ones and the final results reported in the paper are based on them. [Some hps changed (we did more tuning) after the paper was submitted. We will update the paper on arxiv to make sure there is no discrepancy here.]
Also let us know if you got any issue with reproducing the results using the the hps in the config folders for each model.
Oh, I just realized that the automatic system that syncs the github repo with the internal code has missed the config files. Really sorry for that. We'll work on this and make sure that the configs for each model on each task will be out today.
Alright! All configs are now available! Let us know if you had any questions...
Thanks for fixing it so quickly!
Hi,
I am looking for the hyperparameters of each task to reproduce table 1. The hyperparameters in base_*_config.py do not match what is reported in Appendix, but when I tried using the hyperparameters reported in the paper, the result does not seem to match the scores in table 1. Could you provide the hyperparameters you used to produce table 1?
Thanks!