search

compphoto / IntrinsicCompositing

Code for the SIGGRAPH Asia 2023 paper "Intrinsic Harmonization for Illumination-Aware Compositing"
https://yaksoy.github.io/intrinsicCompositing/
Other
39 stars 9 forks source link

Get a blurry result if I keep the same inference resolution as the original image (means scale = 1.0). #6

Open guyleaf opened 1 month ago

guyleaf commented 1 month ago

Hi, thank you for making this great work!

If I keep the same inference resolution as the original image (means scale = 1.0), I will have a chance to get a blurry result. If I set the inference resolution to 1024, the result is still a little blurry.

I think it is because the harmonization model for albedo works on 384 x 384 resolution. Do you know how to solve it?

Result (original resolution, 1145 x 707)

result result_intermediate

Result (1024)

result_1024 result_1024_intermediate

Composite

composite

Background

bg

Mask

mask

CCareaga commented 1 month ago

The blurriness probably comes from the reshading estimation. The albedo harmonization is parametric so it can be applied at any resolution. The reshading network is trained at (384 x 384) but the network will make blurrier predictions at low-resolutions. I think you will find that sending the image through as large as possible will give the sharpest results, but eventually the network could exhibit some global inconsistencies due to the fixed capacity/receptive field. Another trick that can help is to upscale the image, send it through the pipeline, and then downscale the output back to the original size. This can give a sharper result than sending the image through at it's original resolution but it could degrade performance as I said before.

guyleaf commented 1 month ago

Hi, thanks for your response.

Oh, I see. So, the actual problem may come from the re-shading network.

But you say sending the image through as large as possible will give the sharpest results. In this case, the down-scaled version (1024) is less blurry than the original one, especially for the yellow object.

Sorry, I forgot to mention that the original resolution is 1145 x 707.

CCareaga commented 1 month ago

Yeah the estimations definitely look different on the yellow object, but when looking at the shading it's hard to tell which is blurrier. You could try sending the image through at a large resolution (e.g. 1500 or the max your GPU allows) then downscaling the shading back to the original image resolution before multiplying by the albedo. In general, this is a limitation of the model and it's difficult to get a very sharp shading layer partially due to the fact that the input normals aren't always super sharp.