Get mesh with small aabb

[darui18]

Hi,Thanks for sharing the great work! When I use neus-facto, I can get mesh with default parameters. There is a small object in the middle of the mesh, so I set aabb to [[-0.2,-0.2,-0.2], [0.2,0.2,0.2]], but I can’t get any mesh output, where is the parameter Having trouble setting it up?

[niujinshuchong]

Hi, you don't need to change aabb since it's used for training. You can add --bounding-box-min -0.2 -0.2 -0.2 --bounding-box-max 0.2 0.2 0.2 to the command when you run ns-extract-mesh.

[darui18]

I hope that model can pay attention to the small area, so that I can get a better mesh.

[niujinshuchong]

I see. You can also normalise the camera poses such that the object bbox becomes [-1, 1].

[darui18]

Thank you very much for your reply. I've also tried to zoom in on the area of interest to [-1,1] or narrow down aabb, but I don't get mesh, which is confusing.

In my scene, there is an object in the middle and a background around it. It seems that aabb must include the background to get the mesh. But I want mlp to focus only on the objects in the middle, so I can get more details on output.

[niujinshuchong]

Hi, you need to use a background model in this case. You could try to add --pipeline.model.num-samples-outside 32 --pipeline.model.background-model grid to your command. Also using a larger MLPs (8 layer with 512 hidden dimensions) could help.

[darui18]

Thanks for the feedback. I will giving it a try. By the way, I wonder if there's a more straight-forward way of controlling the scope of the foreground MLP ?To be specific, I have a scene with a foreground and a background object. I would like the MLP to only learn the foreground object without the background. Could you kindly point me to which parameters may be used to achieve this foreground-only reconstruction ?

[MIMNSI]

Thanks for your great work @niujinshuchong. I have a similar doubt. Can we subtract the background while keeping the foreground object?

[niujinshuchong]

@darui18 @MIMNSI If you have the foreground masks of the object, you could create images that all the backgrounds are replaced with white/black color and set the background color with --pipeline.model.background-color white/black accordingly.

[MIMNSI]

Thanks for the quick reply, @niujinshuchong ! I tried it on different objects and it works great on some of them. However there are cases where segmentation masks fail to cover the entire object. In these cases the mesh constructed is incomplete or contains surrounding objects. Sharing a few segmentation examples here to highlight the problem

To mitigate this, maybe a method other than segmentation masks can be used.

[niujinshuchong]

@MIMNSI The mask looks bad. Maybe you could try the model with background modeling --pipeline.model.num-samples-outside 32 --pipeline.model.background-model grid and tune the foreground mask loss weight with --pipeline.model.fg-mask-loss-mult.