Closed Anri-Lombard closed 3 weeks ago
If I generate 1000 molecules and 952 are valid, but of the generated molecules running it through rdkit shows 100% of them are valid, is validity 0.952 or 1.0? The reason I ask is because after training the model from scratch and replicating your de_novo generation results, the fragment generation results do not seem to have as high a validity as the claim. I'm curious if this is my mistake.
The validity would be 0.952. By default, the sanitize option is set to true, so validity can be computed using # valid compounds returned / # compounds asked
. Other metrics were then computed on the valid molecules. You can also just put sanitize to false and compute the metrics the "normal way".
Also note that for the paper we run it in chunks using different seeds, so not all the sampling is done in a single block. I doubt however that it would explain any disparity you might see.
I am quite curious about your results if you could share them. The most important difference is likely the massive amount of data + model capacity. If your training data does not cover a similar chemical space to the benchmark drugs, then you might have some difference in performance which can be fixed by finetuning on compounds from those chemical space.
As a sanity check, I would suggest picking a molecule from your training set, then defining scaffold, motif, fragments, etc from it to make a small scale test. You should normally expect good performance on that molecule.
Also explore the do_no_fragment_further
argument, it can really change performance for some fragment-based tasks.
Thank you @maclandrol; I'd be happy to share the results:
Keep in mind I accidentally multiplied validity by 10, but beyond that these calculations should be taken with a heavy scoop of salt. I'm going to take your advice into account and do some experiments to see if my evaluation is set up correctly.
Just to address a few points you brought up: I am using the benchmark dataset provided by yourselves, although it has a few more molecules than what was used in the paper, if I'm not mistaken. I'll report back after some additional experimentation so we could keep this open for now if you don't mind.
@maclandrol unfortunately I was not able to replicate the exact findings, but I did not follow the exact experimental setup either. It would be awesome to see different architectures of similar sizes to the large model be compared to safe-gpt, so perhaps we could do an experiment on this at a later date 🙌 😄
The paper claims a validity very close to 1 (and in fragment constrained generation exactly 1 on average). Was this calculation done by looking at the validity of generated molecules, or how many molecules were valid from the eventually generated molecules?
For example:
For some more context here is a snippet of my notebook:
(Disclaimer: this is with the small 20M model, so that could be the cause now that I ponder this)