search

nv-tlabs / LION

Latent Point Diffusion Models for 3D Shape Generation
Other
735 stars 57 forks source link

How can I use clip-related features ? #36

Closed tky5622 closed 1 year ago

tky5622 commented 1 year ago

Hi, @ZENGXH. I appreciate your excellent work!

I try to use clip-related features equipped with this model, such as single-view reconstruction. I see through the original paper, and it said that feature requires training latent diffusion models by images. I'd like to know how I can realize this.

We render 2D images from the 3D ShapeNet shapes, extracted the images’ CLIP [105] image embeddings, and trained LION’s latent diffusion models while conditioning on the shapes’ CLIP image embeddings.

I guess I need to change clip_forge_enable = 1 when training train_prior. But I needed help understanding how to do it properly. I was wondering if you could instruct how to do it.

thank you in advance !

ZENGXH commented 1 year ago

Hi @tky5622, I update the readme for the prior training with clip feature, see here.

Hope it helps!

tky5622 commented 1 year ago

That's awesome! I'll get started on it right away! Thank you so much for your assistance!