search

Kai-46 / IRON

Inverse rendering by optimizing neural SDF and materials from photometric images
BSD 2-Clause "Simplified" License
299 stars 24 forks source link

Low level of details for synthetic dragon #20

Open arterms opened 2 years ago

arterms commented 2 years ago

Hello! Thank you for the great work! Right now I'm trying to reproduce with your code reconstruction of dragon from 200 frames, rendered by mitsuba in 512x512 resolution. Unfortunately after 50000 steps of the second stage I realized that many details on the mesh are lost (if comparing to those pictures from the paper). Dragon_51200 Dragon_51201 Dragon_51202

Could you please tell me, whether _womaskiron.conf config file is the valid one for training with preserving small details? Which parameters might be tuned for better quality? Should be gamma-correction be applied for exr-files generated by mitsuba-renderer before feeding into networks? How input resolution affects shape precision? Thanks!

Kai-46 commented 2 years ago

Hi -- can you share some of your rendered images?

Note that we do gamma-correct our mitsuba-rendered images (also w/ proper global scaling to reasonable brightness) and save as uint8 png images. Then we will use these png images (no inverse gamma correction) in the first stage; in the second stage, we will first inverse-gamma-correct the png images, as rendering equation is linear.

arterms commented 2 years ago

Hello! Without gamma-correction mitsuba-rendered images looks like under-exposed: mitsuba_rendered For the first stage I used this code for reading *.exr files, which performs gamma-correction to reasonable brightness: https://github.com/Kai-46/IRON/blob/5993e35c19c8338f6e5bed33dc3b852c36d12295/models/dataset.py#L69-L78

Is it flag --inv_gamma_gt supposed to be set during the second stage to perform inverse gamma-correction of the input images? I'm asking, because train_scene.sh script doesn't set this flag for the second stage.

Kai-46 commented 2 years ago

Hi -- the exr images need to be scaled (same scaling factor applied to all your training and test images) and then gamma-corrected to have well-exposed uint8 png images. I can share some example script to achieve this exr to png conversion later.

And yes, --inv_gamma_gt would be needed in the second stage for the synthetic data. (the setting in train_scene.sh was meant for real-world images; somehow I found that without this --inv_gamma_gt, the method recovers better specularity for real-world captures sometimes).

arterms commented 2 years ago

Hello! Thank you for the reply! Could you suggest any reasonable scaling factor to apply for all exr images before gamma-correction (with grade 1/2.2)?

Kai-46 commented 2 years ago

Let's say you put all your training and test images into a tensor of shape [N, H, W, 3]; then a good heuristic might be to scale the 98-percentile (might worth trying other percentiles) of the flattened tensor to be 1; then you clip the scaled tensor to the range (0,1), and perform gamma correction, followed by saving as uint8 png.

arterms commented 2 years ago

Hi! I experimented a bit with the different percentiles for normalization and gamma-correction setups for the second stage. The best result I got is with 95-percentile and disabled _--gammapred and enabled _--inv_gammagt flags. Unfortunately, this result is still doesn't contain small details and sharp edges like that presented in the paper. dragon_1024_00-00 dragon_1024__00-00 Do you have any suggestion how to tune configs/setups to get the shape more accurate? Thanks!