Question about set-wise exact match

[YanjingLiLi]

Hi, could you please explain more about the set-wise exact match here? I am still confused about its difference with the normal top-k accuracy.

[SongtaoLiu0823]

First of all, the goal of retrosynthetic planning is to predict the set of starting materials. Like one-step retrosynthesis prediction, we can predict multiple sets of staring materials, which are ranked by their cost (negative log-likelihood). We compare the predicted sets with the ground truth set and get the top-k accuracy.

[YanjingLiLi]

Thanks for the explanation. Do you mean that each set (containing several starting materials) has one cost, so that there can be multiple ranked sets e.g. 1. A, 2. B ...... Then, the top-1 accuracy is the fraction of correct starting materials in set A?

[SongtaoLiu0823]

No. You can find the details about top-k accuracy at https://github.com/SongtaoLiu0823/FusionRetro/blob/d194b1a8491950e37610e0bba75f9bcbbc373878/FusionRetro/retro_star_0.py#L327.

I also explain it here.

We consider it a hit if the staring materials of a predicted set are the same as those in the ground truth set. Consider we have M ground truth sets G_m (1<=m<=M), for each set, we have multiple predicted sets 1. A_m, 2. B_m ... (1<=m<=M). If A_m = G_m, we think we have a correct prediction. So top-1 accuracy: count(A_m=G_m)/M.

[YanjingLiLi]

I really appreciate your explanation, which is clear.

I have another question about your benchmarking. Based on my understanding, for a given target molecule, other works using top-k accuracy usually generate a bunch of reactant molecules, ranked by a score, which is not grouped into "sets" as you have done in your paper. So how did you calculate the set-wise exact match accuracy of their work?

[SongtaoLiu0823]

Like other one-step retrosynthesis models, our work is indeed a one-step model, which utilizes in-context reactions for prediction. During each step in retrosynthetic planning, we generate a bunch of reactant molecules for backward search until all leaf nodes of synthetic routes are starting materials. You can find more details from retro_star_0.py in each folder.

[YanjingLiLi]

Thanks for your patients. I carefully read the code in retro_star_0.py for Megan evaluation. Let me use this as an example to explain my questions.

In Megan, the beam search will result in a list of list of dictionaries, which can be described as the following structure:

- List for each molecule:
    - List of dictionaries representing finished paths for that molecule
      - Dictionary for each finished path:
        - 'final_smi': Final SMILES representation of the molecule after following the path
        - 'final_smi_unmapped': Final SMILES representation of the molecule without atom mapping numbers
        - 'prob': Probability associated with the path
        - 'actions': List of actions taken to reach the final molecule
        - 'n_steps': Number of steps taken to reach the final molecule
        - 'changed_atoms': Set of atom map numbers that were changed during the path
        - 'mol_graph': Tuple containing the molecular graph (adjacency matrix and node features

The for each molecule here means the product. In other words, for each product, Megan can generate a list of predictions, each of which is a set of starting materials split by ".", denoted by "final_smi". In their code for calculating the top-k score (https://github.com/molecule-one/megan/blob/master/bin/eval.py), they also define a correct set if all the predicted starting materials in that set are the same as the ground truth set. I am thinking if it's just what you mean by "set-wise exact match"?

Based on my understanding of how you evaluate Megan, you still treat it as a one-step prediction, but you combine them into a multi-step prediction by recursively predicting the next reactant based on the current product at this step until all the predicted reactants are marked as the starting materials. But in Megan, it seems like what they predict in their paper is directly considered as the starting materials. Am I making any mistake here?

[SongtaoLiu0823]

“The for each molecule here means the product. In other words, for each product, Megan can generate a list of predictions, each of which is a set of starting materials split by ".", denoted by "final_smi".”

No, we need to check whether the predicted reactants are starting materials.

In the Megan's paper, their task is one-step retrosynthesis prediction on USPTO-50K. So they don't need to check whether the reactants are starting materials. Our benchmark is the first to evaluate different one-step retrosynthesis models on retrosynthetic planning. I think you need to understand the difference between single-step retrosynthetic prediction and retrosynthetic planning tasks.

For Retrosynthetic planning task, please refer to Retro: Learning Retrosynthetic Planning with Neural Guided A Search. For one-step retrosynthesis prediction task, please refer to G2Gs, Megan, GraphRetro, etc.

[YanjingLiLi]

Get it. Thanks a lot for your explanation. So reactants are not necessarily starting materials.