jkulhanek
commented
1 month ago Hi, thank you for your interest. I was able to find the issue. 1) camera_ids - this is a trivial bug as you already realized, initializing with zeros fixes the issue. 2) However, after fixing 1), there is a much more sinister bug caused by nerfstudio. In particular, nerfstudio's viewer has two rendering passes for each view - fast and slow. First, the fast pass is used with a lower res to render something to present to the user immediately. This is followed by the second pass which takes much longer for nerfs (could be seconds with nerfacto). For that reason, nerfstudio uses sys.settrace to cancel the running rendering and switch back to faster rendering as soon as use moves the camera. Unfortunately, this has lots of unwanted effects - breaks debuggers, and in our case, it cancels the rendering at a point that breaks the model - causes random errors, or renders the image totally weird.
I will soon fix both 1) and 2). 2) will be fixed by "hacking" nerfstudio to remove sys.settrace, but there is no need to reinstall nerfstudio or anything, it will remove sys.settrace at runtime.
zlaskar
Thanks for the code release! Installation, training and suggested evaluation works as expected.
However, when using viewer and rendering custom camera paths (selected from viewer) returns similar error as below which is related to https://github.com/tobiasfshr/map4d/blob/main/src/map4d/model/gauss_splat.py#L759-L760.
On the other hand, if camera_id and sequence is fixed to 0, the rendered image is incorrect (attached figure)
Attached figure:
File "/home/laskazak/.conda/envs/map4d/lib/python3.10/threading.py", line 1016, in _bootstrap_inner self.run() File "/home/laskazak/.conda/envs/map4d/lib/python3.10/site-packages/nerfstudio/viewer/render_state_machine.py", line 222, in run outputs = self._render_img(action.camera_state) File "/home/laskazak/.conda/envs/map4d/lib/python3.10/site-packages/nerfstudio/viewer/render_state_machine.py", line 168, in _render_img outputs = self.viewer.get_model().get_outputs_for_camera(camera, obb_box=obb) File "/home/laskazak/.local/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context return func(*args, **kwargs) File "/mnt/personal/laskazak/code_iiith/map4d/src/map4d/model/gauss_splat.py", line 1130, in get_outputs_for_camera outs = self.get_outputs(camera.to(self.device)) File "/mnt/personal/laskazak/code_iiith/map4d/src/map4d/model/gauss_splat.py", line 902, in get_outputs colors_crop[visible_mask], opacities_crop[visible_mask] = self.get_colors_opacities( File "/mnt/personal/laskazak/code_iiith/map4d/src/map4d/model/gauss_splat.py", line 760, in get_colors_opacities camera_id = camera.metadata["camera_ids"][0] KeyError: 'camera_ids'