guidance function fails

[pengzhangzhi]

Hi Jason, I recently worked on guiding the generation of se3 diffusion with a guidance function, similar to classifier guidance. I found the output trajectory Xt satisfies the guidance function very well at t in the middle, when t is close to 0, the Xt is out of control. I think this is because the training of se3 uses many losses to stress on the X0 reconstruction at t close to 0. Not sure if my hypothesisis correct. Do you have any ideas?

[pengzhangzhi]

For your reference, I try to guide the generated structure to obey a template structure. Here is the trajectory. convert txt to pdb. bb_traj_1.txt

[pengzhangzhi]

For those who are interested in guiding se3 diffusion, after a few days of debugging and experiments, my conclusions are:

• The guidance works to a certain extent.
• Due to the bias in the training, the models tend to reconstruct the X0 structure at t close to 0. Thus the model will compete with the guidance function as t is close to 0 so the guidance seems like doesn't work perfectly.

[jasonkyuyim]

Hi, I'm currently busy with some paper deadlines so can't look in detail. I have a few thoughts.

• It seems plausible the losses near t=0 could be a problem for guidance. Have you tried different weights? I can't help debug without knowing your guidance function or how you're implementing it. You should make sure the auxiliary losses are consistent with the minimzers of the guidance.
• Score matching losses have trouble learning near t=0. It's a well-known phenomenon you can read about in the Song SDE paper or NVIDIA's paper on the design space of diffusion models. There's nothing wrong with truncating your diffusion to stop at t=0.1 for instance if that gives the best results (though some theorists would get mad at you). But ideally the optimization would play nicely at t=0.01. There is a option for this in the config.
• One idea is you can try turning off the score matching losses for very low t and only rely on the auxiliary losses and guidance function.