Open Sazan-Mahbub opened 2 months ago
Hi @Sazan-Mahbub, thanks again for your interest!
Sorry that I still haven't found the time to look into this in detail. I did look at my wandb logs for the experiments I reported in the paper, and can confirm that I got the results when using 8 layers. I can send you specific checkpoints or logs if you think that would be helpful?
Can I check whether you're:
I will try to retrain and reproduce the results when I can. I generally did not find them to be so sensitive (eg. I was able to consistently see same performance with multiple seeds and on different hardware such as GPUs and XPUs). Sorry for the late reply.
I've also emailed you some checkpoints and logs for invariant and equivariant models on the Single-state split.
Hi Chaitanya,
Thank you very much for your kind help!
Regarding the questions:
The checkpoints and the logs you shared will be very helpful. Thank you again!
Hi @chaitjo, thank you for this awesome work!
Could you kindly share the checkpoints and the hyperparameters for the NAR models reported in Table-1 of your paper (image attached)? I tried to reproduce the scores using the configs/default.yaml file, with 'model' set to 'NARv1', and 'num_layers' set to '8', as discussed in the paper. But I think I am missing something here, because the sequence recovery scores I got were much lower (~0.52 vs 0.584).
Thank you for your help!
-Sazan