search

jasonkyuyim / se3_diffusion

Implementation for SE(3) diffusion model with application to protein backbone generation
https://arxiv.org/abs/2302.02277
MIT License
305 stars 50 forks source link

One Question about Validation (Intermittent evaluation) #36

Closed cnexah closed 5 months ago

cnexah commented 5 months ago

Hi, thank you for your great work! I have a question about the input data during validation (Intermittent evaluation):

In the following code:

https://github.com/jasonkyuyim/se3_diffusion/blob/53359d71cfabc819ffaa571abd2cef736c871a5d/data/pdb_data_loader.py#L260

In the dataloader, during validation, the input to the model is random noise, but the ground truth is obtained by sampling from data. Is it meaningful to compute the TM-score metric? https://github.com/jasonkyuyim/se3_diffusion/blob/53359d71cfabc819ffaa571abd2cef736c871a5d/experiments/train_se3_diffusion.py#L382

jasonkyuyim commented 5 months ago

Hi! The intermittent TM-score during evaluation is not meaningful for unconditional generation. We were experimenting with ensembles and forward folding performance for which TM-score was useful. You can ignore that metric depending on the task at hand.

cnexah commented 5 months ago

I see. Thank you for your reply!