search

yccyenchicheng / SDFusion

MIT License
404 stars 34 forks source link

About Multiple Modalities #8

Closed yjcaimeow closed 1 year ago

yjcaimeow commented 1 year ago

Hi @yccyenchicheng ,

I have some question about multi-modalities. Which dataset is this portion of the function implemented on? and When the code for this part will be release ?

Best regards, Yingjie CAI

yccyenchicheng commented 1 year ago

The dataset for image-shape is from 3d-r2n2. The text-shape is from text2shape.

This part along with the training of buildingnet and texturization will be released soon!

yjcaimeow commented 1 year ago

Thank you for your patient reply.

So text2shape dataset can support text and partial shape two condition info at the same time?

Best, Yingjie

yccyenchicheng commented 1 year ago

Shape completion is achieved naturally when training the diffusion model. The paired data we train is: (img and/or text, shape), where the and/or is achieved by dropout during training. In the inference, the given partial shape can guide the diffusion process for generating the complete shapes, while using classifier-free guidance to control the strength from image or text conditions.

yjcaimeow commented 1 year ago

Thans a lot!