Lower results for forward-synthesis on USPTO-MIT

[AslantheAslan]

I have been reproducing following the same instructions you gave in the readme file, however, I am unable to reach the same top-k accuracy results as mentioned in the paper. Please note that I am training on USPTO-MIT (mixed version). Results are as above:

I did not set external training parameters, just followed the same way mentioned in repository. Is that because I should pass additional parameters to match your implementation? Any comment would be appreciated.

[mikolajsacha]

The results reported in the paper should be acquired by running the exact scripts from the readme file - the default parameters are already the ones that should give the best result. Some slight deviation is expected due to random seed, but your results seem much worse than those reported by us. First, I would check whether the training ran until the end. Other than that, I don't have any ideas off the top of my head. Unfortunately, I currently don't have access to our trained model checkpoint on USPTO-MIT. Mikołaj

[AslantheAslan]

Thanks for your rapid reply. Just to confirm, the model trains with max_epoch=200 and early_stopping=16 as given in the config file for USPTO-MIT_mixed. Around epoch 100, the early stopping interrupts and training terminates there. Even when I drop early stopping and train until the end of 200 epochs, the model did not improve. At this point, I am unsure if this is because the evaluation script (I suspect of different RDKit versions may give different outputs, but this is very unlikely). By the way, in the readme it was mentioned that preprocessed data and trained models are provided in v1.1 but I could not find those. Please keep the issue opened and I will update here if I can solve the problem I have.