Camera poses & coordinate system used

[smandava98]

Hey! Great work. I had a couple questions on the surface reconstruction when the camera in the input video moves a lot.

Does Neuralangelo optimize the implicit surface in a canonical object-centric coordinate system or directly in world coordinates?

For example, say I take a video of the Nvidia HQ from 2 different angles (but still show the whole HQ in both videos), will the reconstructions be the same? Will they be in the same coordinates?

[mli0603]

Hi @smandava98

• Neuralangelo optimizes the surface in a canonical coordinate system, which is the coordinate where camera/object poses are defined.
• Two different video sequences of the same object often do NOT have the same coordinate frame due to how COLMAP initializes the world coordinate. However, it is possible to align the coordinates using a single affine transformation.

[smandava98]

Thanks Max. I have a followup question:

How can I run the neural reconstruction around a particular object in a video? For example, I have a video of a house on a road but a street lamp is blocking parts of the house in some frames which are then more visible in future frames. How can I somehow omit this object in the reconstruction and have missing pieces of that house filled up from later video frames?

[mli0603]

Hi @smandava98

Neuralangelo can handle occlusion and will simply reconstruct both objects (i.e., the lamp and the house). If you don't want the lamp, you have two options:

• remove the lamp after extracting the mesh using mesh editing softwares (such as meshlab or blender)
• remove the lamp from the bounding regions. You can see step 5 in Data Preprocessing on how to adjust the bounding region

[smandava98]

This still seems to be a manual approach. Is there any way to make it automatic? I have a tons of videos of houses but are being obstructed by lamps and I have a instance segmentation model for the lamps that gives me the masks. Is there not a way where I can just feed Neuralangelo a mask that it can ignore automatically?

[mli0603]

Yes, feeding masks is definitely an option. We removed this functionality in the release as we did not feel there will be lots of use cases. Usually people are happy with both lamps and house ;)

I will add this to our todo list

[smandava98]

Thank you! Adding that option will be extremely helpful. My use case is I want to get clear reconstructions of certain objects and lamps and all that are unnecessary elements in that reconstruction I'm trying to create.

Hope this illuminated the use case better.

[smandava98]

Hi @mli0603. Thanks for your detailed and consistent support to the community. It's amazing.

While I wait for this enhancement to be added, can you please point me to which file(s) I might need to modify to have it ignore a mask? Since you mentioned it was included before but it got removed.

[mli0603]

Hi @smandava98

If you look at the code below, it reads an image and generates pixel indexes to optimize the geometry. What you would do is read the corresponding masks and remove the unwanted pixels. If you have something working, feel free to submit a PR :)

https://github.com/NVlabs/neuralangelo/blob/2213bb3733267c021f8cbf7651bcb39b4804d4d9/projects/neuralangelo/data.py#L49-L85

[smandava98]

Awesome, thank you. Will definitely put up a PR if I get it working :)

[tkl2020]

I also encountered the same problem, I added Neus's Mask loss and tried white background, but it didn't work. Ask the author if he has any feasible suggestions.

[chenhsuanlin]

Please also see #58 for related discussions.

[lishaobingdong]

@smandava98 @tkl2020 Hello! I also encountered this problem. I read the mask images and added mask_loss, but it seemed to have no effect. What files did you change to succeed? Thank you! Looking forward to replying

[lishaobingdong]

@smandava98 @tkl2020 Hello! I also encountered this problem. I read the mask images and added mask_loss, but it seemed to have no effect. What files did you change to succeed? Thank you! Looking forward to replying

I succeeded by setting background enabled to false and using image=image*mask

[Davidyao99]

@lishaobingdong Hi, I am trying to do the same task of training neuralangelo on a scene with masks to remove certain objects. Can you elaborate a little on how you mange to make it work!! Thank you so much ;))