search

mlpen / Nystromformer

Apache License 2.0
361 stars 41 forks source link

Results on Long Range Arena #1

Closed joaogui1 closed 3 years ago

joaogui1 commented 3 years ago

Hi, I saw the tweet about this paper and wanted to know the performance of Nystromformer on each individual task in Long Range Arena. (Also what is the "Standard" in the graph posted? Standard transformer does much worse in LRA)

yyxiongzju commented 3 years ago

Hi @joaogui1,

Thanks for your interest. We did not release the score on each individual task in LRA benchmark. When we update the draft by adding it, I will ping you here. Standard denotes the standard self-attention. Since the LRA contains several tasks, the average score, 58.95 vs 58.77, indicates Nystr\"{o}mformer is competitive to standard self-attention on LRA tasks.

joaogui1 commented 3 years ago

Thanks for the answers @yyxiongzju, but on the original LRA Transformer scores 54.39, not 58.77, why the difference? Also, any predictions on when will you release the next version?

yyxiongzju commented 3 years ago

Hi @joaogui1, here are the scores on each individual task in LRA benchmark. LRA.

The difference is from the performance of running all models on the Retrieval task in LRA benchmark. We followed the LRA procedure to generate tokens/labels to run experiments on the Retrieval task. 1, download LRA dataset; 2, use their code generate tokens/labels. With the generated tokens/labels, we ran all the models on the Retrieval task and it showed higher performance than the original LRA reported. We will release our PyTorch code for running all tasks of LRA benchmark on 02/17.

We will update our paper before 03/02, which is the camera-ready submission deadline of AAAI 2021.