search

postech-ami / Paint-it

[CVPR'24] Official PyTorch Implementation of "Paint-it: Text-to-Texture Synthesis via Deep Convolutional Texture Map Optimization and Physically-Based Rendering"
https://kim-youwang.github.io/paint-it
MIT License
199 stars 6 forks source link

The choise of UNet for reparameterization #23

Closed 07hyx06 closed 3 months ago

07hyx06 commented 4 months ago

Hi, thanks for your work!

Have you tried using a decoder-only architecture (e.g. StyleGAN generator) to reparameterize the UV maps? I have done some experiments to use different architectures to overfit a single image using L1 loss only (optimize all the network parameters to fit a single image). I find the UNet used in Paint-it can do it very well but the StyleGAN generator cannot perform well on this task. Do you have some insights on this experiment?

Thanks in advance.

07hyx06 commented 4 months ago

BTW I think reparameterization is a very neat idea and also recognize your work that applies parameterization technique to 3DMM fitting :)

Youwang-Kim commented 3 months ago

Hi, thanks for your interest in our work.

I guess you're trying a task similar to the Deep Image Prior (DIP) paper. The authors of DIP have extensively compared the architecture choice, and we followed their conclusion (U-Net with skip connection).

We have done experiments about removing the skip connection in our U-Net and observed that the skip connection is crucial for generating high-frequency details. But for other architectures (e.g., decoder-only), unfortunately we haven't done experiments.

It'd be a really interesting direction to explore the optimal neural re-parameterization for various tasks :)

Youwang-Kim commented 3 months ago

Please reopen this issue if you need more help regarding this.