search

astra-vision / MaterialPalette

[CVPR 2024] Official repository of "Material Palette: Extraction of Materials from a Single Real-world Image"
MIT License
233 stars 10 forks source link

Generated texture is not smooth #6

Closed coderSinol closed 3 months ago

coderSinol commented 7 months ago

First of all, this is a great project. Thanks for sharing.

I tried the given sample mansion.zip. However, it generated a texture like this for grass, it is not smooth and contains 4 tile-like areas. Do you have any suggestions to improve this? azertyuiop_1K_t50_wmean_top-view-realistic-texture-of-o_1

wonjunior commented 7 months ago

Hi @coderSinol, I appreciate your comment, I also noticed this happening in some instances. During denoising, noise unrolling only enforces a local constraint such that the content of contiguous patches is aligned after decoding. Since decoding is done in batches, the stitching won't be perfectly continuous. I believe the shift in appearance between patches originates from the separate decoding.

It wasn't made very clear in my code but you can try to apply a renormalization step (at the image level after decoding) to try to counter that. It normalizes the patch means with the mean of the entire image. Something like this:

lmean = image_stack.mean(dim=(-1,-2), keepdim=True)
gmean = image_stack.mean(dim=(0,2,3), keepdim=True)
image_stack = image_stack*gmean/lmean

When there are outliers, this might lead to major artifacts, however. So I recommend you also experiment by aligning the median. Let me know if that helps!