Image conditioning

[jclarkk]

Hi, thanks for sharing your awesome work!

Will image conditioning be supported at some point?

[xrenaa]

Hi, thanks for your interest.

We do not have a concrete plan to release the image conditioning part, so it may take months to merge into the current XCube codebase. However, it is very easy to extend the current codebase to image conditioning. There are two ways we have tried:

• Use the cross-attention in the diffusion model: for the condition, we use DINOv2 to tokenize the input images and then concatenate with the Plücker coordinates.
• Unproject the images to dense/sparse voxels and then apply pooling. Then concatenate this feature cube to the input of the diffusion model

[jclarkk]

Thanks for the reply and explanation!