Provide Datasets from Paper

[hnisonoff]

Hi can you clarify which dataset is in the chain_set_splits.json you provided and also provide the json files for all of the datasets used in the paper for reproducibility? Thank you!

[hnisonoff]

Can you also provide the code you used to compute recovery and perplexity. It is not clear to me what number of ensembles you used and what step size you used. Thank you!

[ykiiiiii]

Hi @hnisonoff,

The dataset in chain_set_splits.json file is CATH 4.2 used in Generative models for graph-based protein design.

For how to compute the recovery rate and perplexity, I show an example on the notebook. In order to reproduce the paper result, you can set the step size as 100 and I usually sample 50 times and do ensembles.

Hope it is clear to you now.

[hnisonoff]

Hi @ykiiiiii thank you from the reply. I am still running into some issues. I trained the CATH model using the hyper parameters provided for CATH and then evaluated on the test set set as you described above (step size 100, ensembles 50). I am getting a perplexity of 6.26 and a recovery of 53.5%. Can you help me reproduce the paper results?

[ykiiiiii]

Hi @hnisonoff, could you check whether the diverse mode is switched on? Just set diverse = True in the ddim_sample function. This will maintain the same recovery rate but yield a more flat distribution, which results in a higher likelihood value (lower perplexity).

You might also consider using larger step sizes, such as 250, which can further reduce perplexity.

Hope it fix your issue.

[ykiiiiii]

Hi @hnisonoff, I also update more comprehensive results for different parameter chosen in README.md. Hope it helps!

Cheers, Kai

[hnisonoff]

Hi @ykiiiiii thank you for the updates. I was using diversity_model and I am still unable to reproduce the results with both your provided model weights (BLOSUM_3M_small.pt) and with my own trained model using the code provided in gradeif.py. My perplexity scores are much higher >6 for step=100.

Can you please provide: 1) The exact command needed to train the model. 2) The exact code to reproduce the numbers in the readme from a trained model

[hnisonoff]

@ykiiiiii I actually am able to much more closely match your reported results if I use the non-EMA model. It would still be helpful to have the actual code used to reproduce the results though! Thank you!

[hnisonoff]

One more question: I'm confused why none of the numbers in the README correspond to what is reported in the paper? It isn't clear what parameter settings you actually used to get the numbers in the paper. Thank you.

[ykiiiiii]

Hi @hnisonoff, I have added the script to compute recovery rate and perplexity in test_rr.py. Hope it helps.

The improvements were primarily due to a lower learning rate and the use of the AdamW optimizer. I didn't try too many hyperparameters when I caught the paper DDL. You can replicate the improved results using the configuration in our training script.

[hnisonoff]

Thank you this is helpful!