New script for reproducing paper results directly from files

[KruskalLin]

I follow all the instructions in README and cloned and evaluated it three times using the checkpoint you provided. The evaluating instruction is the same as in README. The dataset I am using is Version 2 in zenodo. The evaluated RMSD < 2% is always 17%, filtered RMSD < 2% is among 32%-34%, and top-5 filtered RMSD < 2% is among 38-40%. I see the paper the results are all higher than these. The whole results are as below

run_times_std 21.73
run_times_mean 47.04
steric_clash_fraction 8.37
self_intersect_fraction 0.58
mean_rmsd 19.127588073855097
rmsds_below_2 17.09366391184573
rmsds_below_5 50.874655647382916
rmsds_percentile_25 2.68
rmsds_percentile_50 4.91
rmsds_percentile_75 9.19
mean_centroid 16.57
centroid_below_2 54.66
centroid_below_5 77.22
centroid_percentile_25 0.8
centroid_percentile_50 1.75
centroid_percentile_75 4.3
top5_steric_clash_fraction 6.61
top5_self_intersect_fraction 0.0
top5_rmsds_below_2 31.68
top5_rmsds_below_5 69.97
top5_rmsds_percentile_25 1.68
top5_rmsds_percentile_50 3.17
top5_rmsds_percentile_75 5.73
top5_centroid_below_2 67.77
top5_centroid_below_5 85.4
top5_centroid_percentile_25 0.59
top5_centroid_percentile_50 1.28
top5_centroid_percentile_75 2.59
top10_steric_clash_fraction 6.89
top10_self_intersect_fraction 0.0
top10_rmsds_below_2 38.29
top10_rmsds_below_5 73.83
top10_rmsds_percentile_25 1.52
top10_rmsds_percentile_50 2.74
top10_rmsds_percentile_75 5.21
top10_centroid_below_2 70.52
top10_centroid_below_5 86.78
top10_centroid_percentile_25 0.51
top10_centroid_percentile_50 1.09
top10_centroid_percentile_75 2.37
filtered_self_intersect_fraction 0.55
filtered_steric_clash_fraction 2.48
filtered_rmsds_below_2 32.78
filtered_rmsds_below_5 59.5
filtered_rmsds_percentile_25 1.63
filtered_rmsds_percentile_50 3.48
filtered_rmsds_percentile_75 7.91
filtered_centroid_below_2 62.26
filtered_centroid_below_5 79.89
filtered_centroid_percentile_25 0.55
filtered_centroid_percentile_50 1.26
filtered_centroid_percentile_75 3.3
top5_filtered_self_intersect_fraction 4.68
top5_filtered_steric_clash_fraction 4.68
top5_filtered_rmsds_below_2 39.94
top5_filtered_rmsds_below_5 73.0
top5_filtered_rmsds_percentile_25 1.45
top5_filtered_rmsds_percentile_50 2.55
top5_filtered_rmsds_percentile_75 5.25
top5_filtered_centroid_below_2 69.97
top5_filtered_centroid_below_5 86.5
top5_filtered_centroid_percentile_25 0.46
top5_filtered_centroid_percentile_50 1.05
top5_filtered_centroid_percentile_75 2.43
top10_filtered_self_intersect_fraction 4.13
top10_filtered_steric_clash_fraction 4.13
top10_filtered_rmsds_below_2 42.7
top10_filtered_rmsds_below_5 75.76
top10_filtered_rmsds_percentile_25 1.4
top10_filtered_rmsds_percentile_50 2.43
top10_filtered_rmsds_percentile_75 4.78
top10_filtered_centroid_below_2 72.45
top10_filtered_centroid_below_5 87.6
top10_filtered_centroid_percentile_25 0.45
top10_filtered_centroid_percentile_50 0.96
top10_filtered_centroid_percentile_75 2.15
no_overlap_run_times_std 21.73
no_overlap_run_times_mean 47.04
no_overlap_steric_clash_fraction 12.05
no_overlap_self_intersect_fraction 0.49
no_overlap_mean_rmsd 13.51542289628426
no_overlap_rmsds_below_2 6.09375
no_overlap_rmsds_below_5 32.29166666666667
no_overlap_rmsds_percentile_25 4.09
no_overlap_rmsds_percentile_50 7.82
no_overlap_rmsds_percentile_75 20.88
no_overlap_mean_centroid 11.02
no_overlap_centroid_below_2 33.66
no_overlap_centroid_below_5 56.94
no_overlap_centroid_percentile_25 1.48
no_overlap_centroid_percentile_50 3.52
no_overlap_centroid_percentile_75 19.7
no_overlap_top5_steric_clash_fraction 9.03
no_overlap_top5_self_intersect_fraction 0.0
no_overlap_top5_rmsds_below_2 13.19
no_overlap_top5_rmsds_below_5 52.08
no_overlap_top5_rmsds_percentile_25 2.62
no_overlap_top5_rmsds_percentile_50 4.82
no_overlap_top5_rmsds_percentile_75 8.93
no_overlap_top5_centroid_below_2 47.22
no_overlap_top5_centroid_below_5 70.14
no_overlap_top5_centroid_percentile_25 1.11
no_overlap_top5_centroid_percentile_50 2.24
no_overlap_top5_centroid_percentile_75 5.99
no_overlap_top10_steric_clash_fraction 9.03
no_overlap_top10_self_intersect_fraction 0.0
no_overlap_top10_rmsds_below_2 16.67
no_overlap_top10_rmsds_below_5 55.56
no_overlap_top10_rmsds_percentile_25 2.45
no_overlap_top10_rmsds_percentile_50 4.19
no_overlap_top10_rmsds_percentile_75 8.05
no_overlap_top10_centroid_below_2 49.31
no_overlap_top10_centroid_below_5 72.22
no_overlap_top10_centroid_percentile_25 0.96
no_overlap_top10_centroid_percentile_50 2.06
no_overlap_top10_centroid_percentile_75 5.64
no_overlap_filtered_self_intersect_fraction 0.0
no_overlap_filtered_steric_clash_fraction 4.86
no_overlap_mean_filtered_rmsds 12.111509165359777
no_overlap_filtered_rmsds_below_2 15.28
no_overlap_filtered_rmsds_below_5 38.89
no_overlap_filtered_rmsds_percentile_25 2.73
no_overlap_filtered_rmsds_percentile_50 6.79
no_overlap_filtered_rmsds_percentile_75 16.45
no_overlap_mean_filtered_centroid 9.79313355364492
no_overlap_filtered_centroid_below_2 41.67
no_overlap_filtered_centroid_below_5 61.11
no_overlap_filtered_centroid_percentile_25 0.94
no_overlap_filtered_centroid_percentile_50 2.82
no_overlap_filtered_centroid_percentile_75 14.3
no_overlap_top5_filtered_self_intersect_fraction 6.25
no_overlap_top5_filtered_steric_clash_fraction 6.25
no_overlap_top5_filtered_rmsds_below_2 22.92
no_overlap_top5_filtered_rmsds_below_5 56.94
no_overlap_top5_filtered_rmsds_percentile_25 2.11
no_overlap_top5_filtered_rmsds_percentile_50 4.28
no_overlap_top5_filtered_rmsds_percentile_75 9.13
no_overlap_top5_filtered_centroid_below_2 51.39
no_overlap_top5_filtered_centroid_below_5 72.22
no_overlap_top5_filtered_centroid_percentile_25 0.8
no_overlap_top5_filtered_centroid_percentile_50 1.86
no_overlap_top5_filtered_centroid_percentile_75 6.24
no_overlap_top10_filtered_self_intersect_fraction 6.94
no_overlap_top10_filtered_steric_clash_fraction 6.94
no_overlap_top10_filtered_rmsds_below_2 25.69
no_overlap_top10_filtered_rmsds_below_5 60.42
no_overlap_top10_filtered_rmsds_percentile_25 1.98
no_overlap_top10_filtered_rmsds_percentile_50 3.96
no_overlap_top10_filtered_rmsds_percentile_75 7.83
no_overlap_top10_filtered_centroid_below_2 54.86
no_overlap_top10_filtered_centroid_below_5 73.61
no_overlap_top10_filtered_centroid_percentile_25 0.8
no_overlap_top10_filtered_centroid_percentile_50 1.74
no_overlap_top10_filtered_centroid_percentile_75 5.46

Could you please take a look? I am not sure if this is because of changes of original codes.

[HannesStark]

Sorry for the inconvenience and thanks for pointing this out. I have updated the instructions in the README and provided a file to evaluate inference outputs when running inference straight from input files.

Please let me know if there are any further issues!

[KruskalLin]

I also retried this twice. The total result is fine right now. One of results is shown below

steric_clash_fraction 4.9
self_intersect_fraction 0.69
rmsds_below_2 34.98622589531681
rmsds_below_5 60.05509641873278
rmsds_percentile_25 1.56
rmsds_percentile_50 3.5
rmsds_percentile_75 8.2
mean_centroid 5.21
centroid_below_2 62.81
centroid_below_5 79.06
centroid_percentile_25 0.52
centroid_percentile_50 1.23
centroid_percentile_75 3.48
top5_steric_clash_fraction 2.75
top5_self_intersect_fraction 0.83
top5_rmsds_below_2 44.9
top5_rmsds_below_5 76.58
top5_rmsds_percentile_25 1.36
top5_rmsds_percentile_50 2.44
top5_rmsds_percentile_75 4.79
top5_centroid_below_2 75.48
top5_centroid_below_5 88.98
top5_centroid_percentile_25 0.43
top5_centroid_percentile_50 0.85
top5_centroid_percentile_75 1.96
top10_self_intersect_fraction 0.83
top10_steric_clash_fraction 2.48
top10_rmsds_below_2 45.45
top10_rmsds_below_5 81.54
top10_rmsds_percentile_25 1.28
top10_rmsds_percentile_50 2.26
top10_rmsds_percentile_75 4.32
top10_centroid_below_2 76.58
top10_centroid_below_5 90.36
top10_centroid_percentile_25 0.39
top10_centroid_percentile_50 0.78
top10_centroid_percentile_75 1.83
no_overlap_steric_clash_fraction 6.94
no_overlap_self_intersect_fraction 0.56
no_overlap_mean_rmsd 11.631793423691793
no_overlap_rmsds_below_2 19.444444444444443
no_overlap_rmsds_below_5 40.97222222222222
no_overlap_rmsds_percentile_25 2.88
no_overlap_rmsds_percentile_50 6.59
no_overlap_rmsds_percentile_75 16.69
no_overlap_mean_rmsd_mean 11.164879602084547
no_overlap_rmsds_mean_below_2 6.944444444444445
no_overlap_rmsds_mean_below_5 30.555555555555557
no_overlap_rmsds_mean_percentile_25 4.1
no_overlap_rmsds_mean_percentile_50 8.06
no_overlap_rmsds_mean_percentile_75 14.33
no_overlap_mean_centroid 9.52
no_overlap_centroid_below_2 44.44
no_overlap_centroid_below_5 60.42
no_overlap_centroid_percentile_25 0.92
no_overlap_centroid_percentile_50 2.53
no_overlap_centroid_percentile_75 15.99
no_overlap_mean_centroid_mean 8.7
no_overlap_centroid_mean_below_2 31.94
no_overlap_centroid_mean_below_5 53.47
no_overlap_centroid_mean_percentile_25 1.65
no_overlap_centroid_mean_percentile_50 3.81
no_overlap_centroid_mean_percentile_75 13.24
no_overlap_top5_steric_clash_fraction 1.39
no_overlap_top5_self_intersect_fraction 0.0
no_overlap_top5_rmsds_below_2 30.56
no_overlap_top5_rmsds_below_5 62.5
no_overlap_top5_rmsds_percentile_25 1.87
no_overlap_top5_rmsds_percentile_50 3.49
no_overlap_top5_rmsds_percentile_75 7.29
no_overlap_top5_centroid_below_2 63.89
no_overlap_top5_centroid_below_5 77.08
no_overlap_top5_centroid_percentile_25 0.67
no_overlap_top5_centroid_percentile_50 1.42
no_overlap_top5_centroid_percentile_75 3.84
no_overlap_top10_self_intersect_fraction 0.69
no_overlap_top10_steric_clash_fraction 2.08
no_overlap_top10_rmsds_below_2 30.56
no_overlap_top10_rmsds_below_5 70.83
no_overlap_top10_rmsds_percentile_25 1.83
no_overlap_top10_rmsds_percentile_50 3.09
no_overlap_top10_rmsds_percentile_75 5.77
no_overlap_top10_centroid_below_2 63.19
no_overlap_top10_centroid_below_5 80.56
no_overlap_top10_centroid_percentile_25 0.65
no_overlap_top10_centroid_percentile_50 1.36
no_overlap_top10_centroid_percentile_75 3.18

[luwei0917]

I got steric_clash_fraction 4.55 self_intersect_fraction 0.8 mean_rmsd 8.32244847626614 rmsds_below_2 32.231404958677686 rmsds_below_5 61.43250688705234

The paper reported rmsds_below_2 = 38.2 +/- 1.0 which is larger than 32.23.

Did I miss something? Or is "Top-1 RMSD" not the same as "rmsds_below_2"?

[KruskalLin]

@luwei0917 Yep, you are correct. The script matches DiffDock (40) and I thought it was for DiffDock (10). It should have a better rmsds_below_2.

[gcorso]

Hi @luwei0917 and @KruskalLin

Apologies for the confusion there was a further small imprecision caused by some incomprehension between me and Hannes. The correct script for evaluation is the following (similar for inference):

python -m inference --protein_ligand_csv data/testset_csv.csv --out_dir results/user_predictions_testset --inference_steps 20 --samples_per_complex 40 --batch_size 10 --actual_steps 18 --no_final_step_noise

As @KruskalLin correctly pointed out 38% refers to the 40 samples. We also forgot to add --actual_steps 18 --no_final_step_noise which just means that the inference procedure is stopped early and no noise is added after the final step (these steps improve validation performance by about 1%).

I corrected the README accordingly.

Thanks for raising the issue and let us know if you have any further problems reproducing the results!

[KruskalLin]

Hi @gcorso,

Thanks for your kind reply. The added script seems not the main solution for the inconsistent results of rmsds_below_2. My new evaluated result of rmsds_below_2 is 32.5. It is still a margin compared to 38.2 in the paper. Below I show all my output

steric_clash_fraction 4.49
self_intersect_fraction 0.74
rmsds_below_2 32.5068870523416
rmsds_below_5 60.05509641873278
rmsds_percentile_25 1.66
rmsds_percentile_50 3.53
rmsds_percentile_75 8.15
mean_centroid 5.04
centroid_below_2 61.71
centroid_below_5 82.09
centroid_percentile_25 0.52
centroid_percentile_50 1.3
centroid_percentile_75 3.09
top5_steric_clash_fraction 3.03
top5_self_intersect_fraction 0.83
top5_rmsds_below_2 42.7
top5_rmsds_below_5 74.93
top5_rmsds_percentile_25 1.38
top5_rmsds_percentile_50 2.49
top5_rmsds_percentile_75 4.96
top5_centroid_below_2 73.28
top5_centroid_below_5 88.43
top5_centroid_percentile_25 0.45
top5_centroid_percentile_50 0.94
top5_centroid_percentile_75 2.13
top10_self_intersect_fraction 0.55
top10_steric_clash_fraction 3.58
top10_rmsds_below_2 44.63
top10_rmsds_below_5 81.54
top10_rmsds_percentile_25 1.33
top10_rmsds_percentile_50 2.4
top10_rmsds_percentile_75 4.35
top10_centroid_below_2 76.86
top10_centroid_below_5 90.08
top10_centroid_percentile_25 0.43
top10_centroid_percentile_50 0.87
top10_centroid_percentile_75 1.92
no_overlap_steric_clash_fraction 6.94
no_overlap_self_intersect_fraction 0.62
no_overlap_mean_rmsd 11.035061458594955
no_overlap_rmsds_below_2 18.75
no_overlap_rmsds_below_5 40.97222222222222
no_overlap_rmsds_percentile_25 2.44
no_overlap_rmsds_percentile_50 6.04
no_overlap_rmsds_percentile_75 14.24
no_overlap_mean_rmsd_mean 11.161503241954968
no_overlap_rmsds_mean_below_2 6.25
no_overlap_rmsds_mean_below_5 30.555555555555557
no_overlap_rmsds_mean_percentile_25 4.38
no_overlap_rmsds_mean_percentile_50 7.87
no_overlap_rmsds_mean_percentile_75 13.6
no_overlap_mean_centroid 8.66
no_overlap_centroid_below_2 45.14
no_overlap_centroid_below_5 66.67
no_overlap_centroid_percentile_25 0.82
no_overlap_centroid_percentile_50 2.35
no_overlap_centroid_percentile_75 11.88
no_overlap_mean_centroid_mean 8.68
no_overlap_centroid_mean_below_2 29.86
no_overlap_centroid_mean_below_5 54.17
no_overlap_centroid_mean_percentile_25 1.71
no_overlap_centroid_mean_percentile_50 3.54
no_overlap_centroid_mean_percentile_75 11.87
no_overlap_top5_steric_clash_fraction 6.25
no_overlap_top5_self_intersect_fraction 0.69
no_overlap_top5_rmsds_below_2 27.78
no_overlap_top5_rmsds_below_5 60.42
no_overlap_top5_rmsds_percentile_25 1.96
no_overlap_top5_rmsds_percentile_50 4.04
no_overlap_top5_rmsds_percentile_75 6.75
no_overlap_top5_centroid_below_2 56.94
no_overlap_top5_centroid_below_5 77.08
no_overlap_top5_centroid_percentile_25 0.63
no_overlap_top5_centroid_percentile_50 1.64
no_overlap_top5_centroid_percentile_75 4.34
no_overlap_top10_self_intersect_fraction 0.69
no_overlap_top10_steric_clash_fraction 4.86
no_overlap_top10_rmsds_below_2 30.56
no_overlap_top10_rmsds_below_5 70.83
no_overlap_top10_rmsds_percentile_25 1.82
no_overlap_top10_rmsds_percentile_50 3.39
no_overlap_top10_rmsds_percentile_75 5.59
no_overlap_top10_centroid_below_2 63.19
no_overlap_top10_centroid_below_5 80.56
no_overlap_top10_centroid_percentile_25 0.64
no_overlap_top10_centroid_percentile_50 1.35
no_overlap_top10_centroid_percentile_75 3.15

[HannesStark]

Hi @KruskalLin ,

Thanks for figuring this issue out with us. Would you be able to rerun evaluate_files.py with --num_predictions 40 ? That would be this:

python evaluate_files.py --results_path results/user_predictions_testset --file_to_exclude rank1.sdf --num_predictions 40

Sorry for the oversight where we did not include the --num_predictions 40 in the readme. Also, are you using version 2 of the dataset on Zenodo?

[KruskalLin]

Thanks for you all. I successfully reproduced the results by recloning the repo. One is shown below.

steric_clash_fraction 7.71
self_intersect_fraction 0.63
mean_rmsd 6.81229025712359
rmsds_below_2 37.1900826446281
rmsds_below_5 63.63636363636363
rmsds_percentile_25 1.54
rmsds_percentile_50 3.25
rmsds_percentile_75 7.2
mean_centroid 4.78
centroid_below_2 64.46
centroid_below_5 81.54
centroid_percentile_25 0.52
centroid_percentile_50 1.09
centroid_percentile_75 2.78
top5_steric_clash_fraction 2.75
top5_self_intersect_fraction 0.55
top5_rmsds_below_2 45.45
top5_rmsds_below_5 76.86
top5_rmsds_percentile_25 1.35
top5_rmsds_percentile_50 2.22
top5_rmsds_percentile_75 4.75
top5_centroid_below_2 75.76
top5_centroid_below_5 89.81
top5_centroid_percentile_25 0.42
top5_centroid_percentile_50 0.87
top5_centroid_percentile_75 1.94
top10_self_intersect_fraction 1.1
top10_steric_clash_fraction 2.75
top10_rmsds_below_2 46.56
top10_rmsds_below_5 80.17
top10_rmsds_percentile_25 1.3
top10_rmsds_percentile_50 2.07
top10_rmsds_percentile_75 4.39
top10_centroid_below_2 76.58
top10_centroid_below_5 90.36
top10_centroid_percentile_25 0.4
top10_centroid_percentile_50 0.85
top10_centroid_percentile_75 1.87
no_overlap_steric_clash_fraction 10.12
no_overlap_self_intersect_fraction 0.47
no_overlap_mean_rmsd 10.85757694331568
no_overlap_rmsds_below_2 19.444444444444443
no_overlap_rmsds_below_5 43.75
no_overlap_rmsds_percentile_25 2.24
no_overlap_rmsds_percentile_50 6.06
no_overlap_rmsds_percentile_75 14.26
no_overlap_mean_centroid 8.56
no_overlap_centroid_below_2 48.61
no_overlap_centroid_below_5 66.67
no_overlap_centroid_percentile_25 0.83
no_overlap_centroid_percentile_50 2.09
no_overlap_centroid_percentile_75 10.86
no_overlap_top5_steric_clash_fraction 4.86
no_overlap_top5_self_intersect_fraction 0.69
no_overlap_top5_rmsds_below_2 28.47
no_overlap_top5_rmsds_below_5 63.19
no_overlap_top5_rmsds_percentile_25 1.82
no_overlap_top5_rmsds_percentile_50 3.52
no_overlap_top5_rmsds_percentile_75 6.72
no_overlap_top5_centroid_below_2 61.11
no_overlap_top5_centroid_below_5 79.17
no_overlap_top5_centroid_percentile_25 0.65
no_overlap_top5_centroid_percentile_50 1.43
no_overlap_top5_centroid_percentile_75 3.53
no_overlap_top10_self_intersect_fraction 1.39
no_overlap_top10_steric_clash_fraction 3.47
no_overlap_top10_rmsds_below_2 29.86
no_overlap_top10_rmsds_below_5 68.75
no_overlap_top10_rmsds_percentile_25 1.78
no_overlap_top10_rmsds_percentile_50 3.29
no_overlap_top10_rmsds_percentile_75 5.86
no_overlap_top10_centroid_below_2 61.11
no_overlap_top10_centroid_below_5 79.86
no_overlap_top10_centroid_percentile_25 0.62
no_overlap_top10_centroid_percentile_50 1.32
no_overlap_top10_centroid_percentile_75 3.34