search

half-potato / nmf

Our method takes as input a collection of images (100 in our experiments) with known cameras, and outputs the volumetric density and normals, materials (BRDFs), and far-field illumination (environment map) of the scene.
https://half-potato.gitlab.io/posts/nmf/
MIT License
53 stars 3 forks source link

How to train with known envmap? #11

Closed gkouros closed 1 year ago

gkouros commented 1 year ago

Hi,

Is it possible to train with a known envmap and if so, how can you do that? I'm trying to see how the model performs with regard to material estimation if you reduce the ambiguity by providing known illumination.

I tried to set the fixed_bg argument but it's not working. It seems to expect a torch checkpoint and when I feed it with the backgrounds/forest.th there are some conflicts with resolution (which is easy to fix) and missing keys (not so easy to fix).

Also, is it possible to use the exr files directly?

Thanks.

half-potato commented 1 year ago

You probably need to recreate the environment map checkpoint. It's probably from an older version of the code.

python -m scripts.pano2cube backgrounds/forest.exr --output backgrounds/forest.th

If this is not the case, can you send the list of missing keys? There is a chance you can just skip those keys with strict=false.

gkouros commented 1 year ago

It seems there is an issue with my version of imageio (2.31.6):

ValueError: Could not load bitmap "/path/to/nmf/backgrounds/forest.exr": Cannot read image file "". Unknown compression type in image header.

Switching to opencv for reading the exr file works.

Also, failing to import _bgmodules, but that can be removed since it's not used anymore.

Training with GT envmap seems to be running fine after recreating forest.th.

half-potato commented 1 year ago

Good to hear things are working.

gkouros commented 1 year ago

Is it normal to get worse performance with the GT envmap? I get around 2 PSNR less on the shiny-blender/car scene in this case.

half-potato commented 1 year ago

I suspect that you need to start with a low frequency environment map and steadily increase the frequency through training. Just a hunch though.

gkouros commented 1 year ago

Thanks. I'll give it a try and report back.