Influence of object scale

[MartinSmeyer]

This is really awesome work, thank you!

I got good initial results for an image collection of an industrial object including both metallic and plastic parts:

Since some geometric detail is still hidden in the texture, I was wondering if the small scale (6cm diameter) might be a problem? I tried playing with the mesh_scale parameter but it diverges so I assume it only works for input meshes not for posed image collections?

{
    "ref_mesh": "data/tless_half/obj_000001",
    "random_textures": true,
    "iter": 5000,
    "save_interval": 100,
    "texture_res": [ 512, 512 ],
    "train_res": [280, 280],
    "batch": 32,
    "learning_rate": [0.06, 0.01],
    "dmtet_grid" : 128,
    "mesh_scale" : 1.0,
    "validate" : true,
    "n_samples" : 12,
    "laplace_scale" : 3000,
    "denoiser" : "bilateral",
    "display": [{"latlong" : true}, {"bsdf" : "kd"}, {"bsdf" : "ks"}, {"bsdf" : "normal"}],
    "background" : "black",
    "out_dir": "tless_half_obj_000001_bs_lr_tex512"
}

[MartinSmeyer]

I guess with a little finer subdivisions we could gain a lot:

[MartinSmeyer]

Actually my fault sorry, I used mesh_scale=5 instead of mesh_scale=1/5.

[jmunkberg]

Thanks @MartinSmeyer

Yes, the code is fairly rudimentary for posed image collections. For best results, make sure your object is centered, and that the grid is fairly tight around the object. There is a known issue with mesh_scale < 1, so if possible, it would be better to scale the mv matrix in the dataset loader if that is an option for you. Which loader are you using for this test?

[jmunkberg]

Also, if possible, I would recommend to run in higher image resolution than "train_res": [280, 280] to increase geometry detail.

[MartinSmeyer]

Thank you very much for your insights, @jmunkberg!

Yes, I think I am running into the issue for mesh_scale<1 now:

Which loader are you using for this test?

I am using the DatasetLLFF. Could you please point me to the line where I can scale the mv matrix?

I would recommend to run in higher image resolution than "train_res": [280, 280]

Good point, I have another set of higher resolution images which I will test next.

[jmunkberg]

After line 88 https://github.com/NVlabs/nvdiffrecmc/blob/main/dataset/dataset_llff.py#L88 try adding a line mv = mv @ util.scale(5.0) or mv = mv @ util.scale(1.0/5.0) (I don't remember which way to scale in the LLFF format) Untested, but hopefully something like that can work!

That loader should already auto-center the poses as well https://github.com/NVlabs/nvdiffrecmc/blob/main/dataset/dataset_llff.py#L65

[MartinSmeyer]

Yes thanks, I just figured it out, it looks promising:

[MartinSmeyer]

I did mv[:3,3] *= 5 so mv = mv @ util.scale(5.0)

[MartinSmeyer]

It's getting more details now:

Flat:

Smooth:

The metal material part is unfortunately missing this time:

Thanks for your help!

[jmunkberg]

Great! I would still expect much higher tessellation/triangle count, if you are using ""dmtet_grid" : 128," so ensure that the initial grid is fairly tight around your model (that is usually visible in early training images). Try scaling even more perhaps, like mv = mv @ util.scale(10.0) or higher.

[jmunkberg]

To get better material parameters, you can play a bit with the regularizers. The settings here are a good start if you are working from real photographs

https://github.com/NVlabs/nvdiffrecmc/blob/main/configs/nerd_gold.json

[MartinSmeyer]

Yes, I am using the 128 dmtet_grid, with 10 it already cuts off the top:

Currently trying 8, it looks better. For the material it seems like it still needs a higher loss on missing metallness despite the boundaries. Higher resolution should help, too, will report back.

[MartinSmeyer]

Auto smooth shading with mv scaled by 8:

Got a bit of the metallicness as well:

I still need to preprocess the high resolution data, will close this for now but probably report back results.

Thanks again!

[MartinSmeyer]

As an update, higher resolution data improved geometry, unfortunately the metallic parts are missing completely now although I tried playing around with the parameters. I might try changing the lambda parameters as well. Maybe it's because it was recorded on a turntable, without a constant point light source. Or because there are not so many metallic regions.

Original:

Thanks again!

[jmunkberg]

Maybe it's because it was recorded on a turntable, without a constant point light source.

Yes, that is likely the issue. We assume a fixed lighting condition and a static object, so if the object rotates, that will break our assumption. For a turntable setup, we could likely add a transform to the env map for each frame, to take the rotation into account, but that is not supported atm