Request for checkpoints and hyperparameters for NAR models

[Sazan-Mahbub]

Hi @chaitjo, thank you for this awesome work!

Could you kindly share the checkpoints and the hyperparameters for the NAR models reported in Table-1 of your paper (image attached)? I tried to reproduce the scores using the configs/default.yaml file, with 'model' set to 'NARv1', and 'num_layers' set to '8', as discussed in the paper. But I think I am missing something here, because the sequence recovery scores I got were much lower (~0.52 vs 0.584).

Thank you for your help!

-Sazan

[chaitjo]

Hi @Sazan-Mahbub, thanks again for your interest!

Sorry that I still haven't found the time to look into this in detail. I did look at my wandb logs for the experiments I reported in the paper, and can confirm that I got the results when using 8 layers. I can send you specific checkpoints or logs if you think that would be helpful?

Can I check whether you're:

• Able to reproduce the autoregressive results?
• Using the correct version of the python libraries?
• Using the same dataset?

I will try to retrain and reproduce the results when I can. I generally did not find them to be so sensitive (eg. I was able to consistently see same performance with multiple seeds and on different hardware such as GPUs and XPUs). Sorry for the late reply.

[chaitjo]

I've also emailed you some checkpoints and logs for invariant and equivariant models on the Single-state split.

[Sazan-Mahbub]

Hi Chaitanya,

Thank you very much for your kind help!

Regarding the questions:

1. Yes I could reproduce the auto regressive results on the single-state split. I have not tried the multi-state yet.
2. I created a conda environment using the instructed steps in the readme.
3. Yes I downloaded the dataset from the google drive link shared in the readme.

The checkpoints and the logs you shared will be very helpful. Thank you again!