is it possible to provide an example for inpainting?

jasonkyuyim / se3_diffusion

Implementation for SE(3) diffusion model with application to protein backbone generation

https://arxiv.org/abs/2302.02277

MIT License

320 stars 51 forks source link

is it possible to provide an example for inpainting? #2

Closed bzhousd closed 1 year ago

bzhousd commented 1 year ago

Dear se3_diffusion/FrameDiff team,

it is great work on methodology and implementation. I followed the installation instructions and I was able to run the inference example code.

In my current project, I need to redesign a part of existing protein. I wonder if you could provide an example for this inpainting purpose. I did find your code to get atom position from pdb file, and convert these atom positions to Rigid object, and it looks fixed_mask or diffusion_mask is helpful to define no-design part based on rigid objects. I guess it is doable theoretically.

Thanks

jasonkyuyim commented 1 year ago

Hi,

Great to hear installation worked for you! We do have internal inpainting code but it is currently an experimental feature. Unless you're working on model development, I recommend waiting for the RFdiffusion code which will do inpainting much better than what FrameDiff can (currently) accomplish.

The fixed_mask and diffusion_mask are indeed the way to start performing inpainting. Our internal inpainting code works with these objects.

pengzhangzhi commented 1 year ago

Hi. Does the current model checkpoint support inpainting? Have you trained the model in an inpainting setting?

jasonkyuyim commented 1 year ago

Hi! Can I ask what the use case is for inpainting with FrameDiff?

If your aim is to have current best motif-scaffolding method then RFdiffusion if your best bet. We just open sourced the code today.
If your aim is to develop new inpainting methods then we're happy to discuss extending FrameDiff for inpainting but only if there is genuine interest.

pengzhangzhi commented 1 year ago

Yes! Great to see the RFDiffusion open-sourced.
I am also interested in discussing how to extend FrameDiff for inpainting. Is current checkpoint support inpainting?

jasonkyuyim commented 1 year ago

The current checkpoint supports inpainting in the sense any unconditional generative model can support inpainting (see our previous work SMCDiff). We have tried training the model in an inpainting setting. I would feel more comfortable discussing our findings in a private setting, i.e. over email. Please reach out to me (jyim@mit.edu) if you want to discuss further.

jasonkyuyim commented 1 year ago

Closing this due to inactivity.