search

PJLab-ADG / neuralsim

neuralsim: 3D surface reconstruction and simulation based on 3D neural rendering.
MIT License
582 stars 31 forks source link

Hyperparameters selection (bounding_box) clipping #27

Open awarebayes opened 11 months ago

awarebayes commented 11 months ago

Greetings! I am trying your work on my own indoors colmap dataset.

I am running it with this config: code_single/configs/exps/lotd_neus.colmap_indoor.230829.yaml

The only problem is that I haven't figured out how to use clipping bounding box. My model is accurately scaled so that 1 unit is 1 meter. So

bounding_size: 2.0

Is not really an option in my case.

I can of course downsize my model, I will be trying it in the near future.

When I set bounding_size, and depth_max to be 40, I get this error:

assert gidx_nonempty.numel() > 0, "Occupancy grid becomes empty during training. Your model/algorithm/training settings might be incorrect. Please check configs and tensorboard."

AssertionError: Occupancy grid becomes empty during training. Your model/algorithm/training settings might be incorrect. Please check configs and tensorboard.

It happens on first iteration.

I can send you the colmap model for debug purposes, it's relatively small consisting of 200 images, 1000px each.

It would also be nice to have some explanation for the hyperparameters used, and how they interplay.

I believe it to be crucial you also use some normalized data for your future experiments and hyperparameter selection, so that 1 unit = 1 real world meter

Here is the indoor scene I am trying to reconstruct. image

awarebayes commented 11 months ago

Sorry to bother, I wrote you previous reply about the clip range selection, but for large scale reconstructions, this parameter really needs to be taken into consideration, as well as actual model scale

awarebayes commented 11 months ago

I have downscaled the model, run 25k iterations. My losses look like that (ignore the jump, I launched second training session some time after):

image

https://github.com/PJLab-ADG/neuralsim/assets/42784580/5cd58b20-81e4-4d57-9e95-a62e0f5281d1

I can say it works, but the results are meh, I get this as room mesh, which looks like colmap reconstuction :) image

Can you please suggest me which hyperparams to tune so that the model looks presentable?

I can send you anything, tensorboard, my colmap folder... Thanks in advance, you are the only repository which accepts colmap inputs

awarebayes commented 11 months ago

I am just using colmap output as is, so it is randomly rotated. Maybe if I rotate it in a specific way (axis aligned), it will better allign with the grid? The existance of grid may imply implicit manhattan assumption. I am talking about the multiscale interpolation grid used in conjunction with MLP

awarebayes commented 11 months ago

I will also debug depth maps generated by the omnidata. Will play with the clearance loss, because the main issue are the bulbous blob looking things protruding everywhere in the room, in the air

kevinchan04 commented 1 month ago

Hi @awarebayes , have you exported a good mesh? I am also facing this issue, and could you share how to finetuen the hyperparameters? Thank you!