Reproducibility of ICLR Figure 2

HazyResearch / zoology

Understand and test language model architectures on synthetic tasks.

Apache License 2.0

162 stars 27 forks source link

Reproducibility of ICLR Figure 2 #28

Open zhan8855 opened 2 months ago

zhan8855 commented 2 months ago

Thank you so much for your awesome work!

Currently, I am facing difficulty to reproduce the Figure 2 of the ICLR paper. I followed the guidelines at https://github.com/HazyResearch/zoology/tree/main/zoology/experiments/iclr24_zoology_figure2. However, my results seem to differ from those reported in the paper:

Would it be possible for you to offer some guidance or advice? I really appreciate your help and look forward to your insights. Thank you again in advance!

simran-arora commented 1 month ago

These look highly similar to me! What's the issue

zhan8855 commented 1 month ago

Well, it seems that my reproduced Hyena and H3 are better than the results reported in the paper (e.g. H3 reaches ~0.5 with dim=64, while the corresponding results reported in the paper is only ~ 0.25). At the same time, my reproduced BaseConv is slightly worse (e.g. Reproduced BaseConv with dim=256 is less than 0.75 in reproduced results, while in paper it is more than 0.75).