search

google-research / long-range-arena

Long Range Arena for Benchmarking Efficient Transformers
Apache License 2.0
710 stars 77 forks source link

Hyperparameters of each task to reproduce table 1 in paper #17

Closed mlpen closed 3 years ago

mlpen commented 3 years ago

Hi,

I am looking for the hyperparameters of each task to reproduce table 1. The hyperparameters in base_*_config.py do not match what is reported in Appendix, but when I tried using the hyperparameters reported in the paper, the result does not seem to match the scores in table 1. Could you provide the hyperparameters you used to produce table 1?

Thanks!

renebidart commented 3 years ago

Is there a plan to release this? A benchmark dataset without reproducibility seems quite useless.

Or am I making a mistake with training using the existing configs? The README says: "Currently we have released all the necessary code to get started and run our benchmarks on vanilla Transformers."

MostafaDehghani commented 3 years ago

Hi,

Thanks for the question and sorry for the issue. Please use the config files in the code for each model to reproduce the results as they are the most updated ones and the final results reported in the paper are based on them. [Some hps changed (we did more tuning) after the paper was submitted. We will update the paper on arxiv to make sure there is no discrepancy here.]

Also let us know if you got any issue with reproducing the results using the the hps in the config folders for each model.

MostafaDehghani commented 3 years ago

Oh, I just realized that the automatic system that syncs the github repo with the internal code has missed the config files. Really sorry for that. We'll work on this and make sure that the configs for each model on each task will be out today.

MostafaDehghani commented 3 years ago

Alright! All configs are now available! Let us know if you had any questions...

renebidart commented 3 years ago

Thanks for fixing it so quickly!