search

ToniRV / NeRF-SLAM

NeRF-SLAM: Real-Time Dense Monocular SLAM with Neural Radiance Fields. https://arxiv.org/abs/2210.13641 + Sigma-Fusion: Probabilistic Volumetric Fusion for Dense Monocular SLAM https://arxiv.org/abs/2210.01276
BSD 2-Clause "Simplified" License
1.14k stars 139 forks source link

NANs in L_inv!! #18

Open Wjt-shift opened 1 year ago

Wjt-shift commented 1 year ago

When I run the code,I got this error.It seems L matrix can't inverse.How should I solve it?Thanks a lot. @ToniRV @jrpowers NANs in L_inv!! Traceback (most recent call last): File "/home/wjt/data/code/NeRF-SLAM/./examples/slam_demo.py", line 200, in <module> run(args) File "/home/wjt/data/code/NeRF-SLAM/./examples/slam_demo.py", line 179, in run and (not slam or slam_module.spin()) \ File "/home/wjt/data/code/NeRF-SLAM/./examples/../pipeline/pipeline_module.py", line 101, in spin output = self.spin_once(input); File "/home/wjt/data/code/NeRF-SLAM/./examples/../slam/slam_module.py", line 11, in spin_once output = self.slam(input) File "/home/wjt/data/software/anaconda3/envs/nerfslam/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl return forward_call(*input, **kwargs) File "/home/wjt/data/code/NeRF-SLAM/./examples/../slam/meta_slam.py", line 30, in forward output = self._frontend(batch["data"], self.state, self.delta) File "/home/wjt/data/code/NeRF-SLAM/./examples/../slam/vio_slam.py", line 114, in _frontend x0_visual, visual_factors, viz_out = self.visual_frontend(batch) # TODO: currently also calls BA, and global BA File "/home/wjt/data/software/anaconda3/envs/nerfslam/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl return forward_call(*input, **kwargs) File "/home/wjt/data/code/NeRF-SLAM/./examples/../slam/visual_frontends/visual_frontend.py", line 326, in forward self.__initialize() File "/home/wjt/data/code/NeRF-SLAM/./examples/../slam/visual_frontends/visual_frontend.py", line 650, in __initialize x0, rcm_factor = self.update(kf0=None, kf1=None, use_inactive=True) File "/home/wjt/data/software/anaconda3/envs/nerfslam/lib/python3.9/site-packages/torch/amp/autocast_mode.py", line 12, in decorate_autocast return func(*args, **kwargs) File "/home/wjt/data/code/NeRF-SLAM/./examples/../slam/visual_frontends/visual_frontend.py", line 439, in update x0, rcm_factor = self.ba(gru_estimated_flow, gru_estimated_flow_weight, damping, File "/home/wjt/data/code/NeRF-SLAM/./examples/../slam/visual_frontends/visual_frontend.py", line 1176, in ba raise RuntimeError: No active exception to reraise

ToniRV commented 1 year ago

Does it always happen? Is it at the beginning of the run?

Wjt-shift commented 1 year ago

Does it always happen? Is it at the beginning of the run?

It happens in the beginning of the code when I run Replica dataset.(I resize the image and depth map get this error).I found the Hession Matrix have Nan element.So the L is not inverseable.It is the error information. _Running with GPUs: 0 Running pipeline in sequential mode. 100it [00:04, 23.39it/s] RaftVisualFrontend.forward ic| k: 0 ic| self.dsf: 8 ic| h: 480 ic| w: 640 ic| h//self.dsf: 60 ic| w//self.dsf: 80 /home/wjt/software/anaconda3/envs/nerfslam/lib/python3.9/site-packages/torch/functional.py:478: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:2894.) return _VF.meshgrid(tensors, kwargs) # type: ignore[attr-defined] viz_index is empty, nothing to visualize ic| os.environ['CUDA_VISIBLE_DEVICES']: '0' 10:27:58 SUCCESS NGP Device set to: 0 10:27:58 INFO Loading network config from: /home/wjt/code/NeRF-SLAM/thirdparty/instant-ngp/configs/nerf/base.json 10:27:58 INFO GridEncoding: Nmin=16 b=1.51572 F=2 T=2^19 L=16 10:27:58 INFO Density model: 3--[HashGrid]-->32--[FullyFusedMLP(neurons=64,layers=3)]-->1 10:27:58 INFO Color model: 3--[Composite]-->16+16--[FullyFusedMLP(neurons=64,layers=4)]-->3 10:27:58 INFO total_encoding_params=13074912 total_network_params=10240 Fusion packet from SLAM module is None... RaftVisualFrontend.forward ic| self.last_kf_idx: 0 Fusion packet from SLAM module is None... RaftVisualFrontend.forward ic| self.last_kf_idx: 0 Fusion packet from SLAM module is None... RaftVisualFrontend.forward ic| self.last_kf_idx: 0 Fusion packet from SLAM module is None... RaftVisualFrontend.forward ic| self.last_kf_idx: 0 Fusion packet from SLAM module is None... RaftVisualFrontend.forward ic| self.last_kf_idx: 0 viz_index is empty, nothing to visualize Fusion packet from SLAM module is None... RaftVisualFrontend.forward ic| self.last_kf_idx: 1 Fusion packet from SLAM module is None... RaftVisualFrontend.forward ic| self.last_kf_idx: 1 Fusion packet from SLAM module is None... RaftVisualFrontend.forward ic| self.last_kf_idx: 1 Fusion packet from SLAM module is None... RaftVisualFrontend.forward ic| self.last_kf_idx: 1 Fusion packet from SLAM module is None... RaftVisualFrontend.forward ic| self.last_kf_idx: 1 Fusion packet from SLAM module is None... RaftVisualFrontend.forward ic| self.last_kf_idx: 1 Fusion packet from SLAM module is None... RaftVisualFrontend.forward ic| self.last_kf_idx: 1 Fusion packet from SLAM module is None... RaftVisualFrontend.forward ic| self.last_kf_idx: 1 Fusion packet from SLAM module is None... RaftVisualFrontend.forward ic| self.last_kf_idx: 1 Fusion packet from SLAM module is None... RaftVisualFrontend.forward ic| self.last_kf_idx: 1 Fusion packet from SLAM module is None... RaftVisualFrontend.forward ic| self.last_kf_idx: 1 Fusion packet from SLAM module is None... RaftVisualFrontend.forward ic| self.last_kf_idx: 1 viz_index is empty, nothing to visualize Fusion packet from SLAM module is None... RaftVisualFrontend.forward ic| self.last_kf_idx: 2 Fusion packet from SLAM module is None... RaftVisualFrontend.forward ic| self.last_kf_idx: 2 Fusion packet from SLAM module is None... RaftVisualFrontend.forward ic| self.last_kf_idx: 2 Fusion packet from SLAM module is None... RaftVisualFrontend.forward ic| self.last_kf_idx: 2 viz_index is empty, nothing to visualize Fusion packet from SLAM module is None... RaftVisualFrontend.forward ic| self.last_kf_idx: 3 Fusion packet from SLAM module is None... RaftVisualFrontend.forward ic| self.last_kf_idx: 3 Fusion packet from SLAM module is None... RaftVisualFrontend.forward ic| self.last_kf_idx: 3 Fusion packet from SLAM module is None... RaftVisualFrontend.forward ic| self.last_kf_idx: 3 viz_index is empty, nothing to visualize Fusion packet from SLAM module is None... RaftVisualFrontend.forward ic| self.last_kf_idx: 4 Fusion packet from SLAM module is None... RaftVisualFrontend.forward ic| self.last_kf_idx: 4 Fusion packet from SLAM module is None... RaftVisualFrontend.forward ic| self.last_kf_idx: 4 Fusion packet from SLAM module is None... RaftVisualFrontend.forward ic| self.last_kf_idx: 4 viz_index is empty, nothing to visualize Fusion packet from SLAM module is None... RaftVisualFrontend.forward ic| self.last_kf_idx: 5 Fusion packet from SLAM module is None... RaftVisualFrontend.forward ic| self.last_kf_idx: 5 Fusion packet from SLAM module is None... RaftVisualFrontend.forward ic| self.last_kf_idx: 5 Fusion packet from SLAM module is None... RaftVisualFrontend.forward ic| self.last_kf_idx: 5 Fusion packet from SLAM module is None... RaftVisualFrontend.forward ic| self.last_kf_idx: 5 Fusion packet from SLAM module is None... RaftVisualFrontend.forward ic| self.last_kf_idx: 5 viz_index is empty, nothing to visualize Fusion packet from SLAM module is None... RaftVisualFrontend.forward ic| self.last_kf_idx: 6 Fusion packet from SLAM module is None... RaftVisualFrontend.forward ic| self.last_kf_idx: 6 Fusion packet from SLAM module is None... RaftVisualFrontend.forward ic| self.last_kf_idx: 6 Fusion packet from SLAM module is None... RaftVisualFrontend.forward ic| self.last_kf_idx: 6 Fusion packet from SLAM module is None... RaftVisualFrontend.forward ic| self.last_kf_idx: 6 Fusion packet from SLAM module is None... RaftVisualFrontend.forward ic| self.last_kf_idx: 6 Fusion packet from SLAM module is None... RaftVisualFrontend.forward ic| self.last_kf_idx: 6 Fusion packet from SLAM module is None... RaftVisualFrontend.forward ic| self.last_kf_idx: 6 Fusion packet from SLAM module is None... RaftVisualFrontend.forward ic| self.last_kf_idx: 6 Fusion packet from SLAM module is None... RaftVisualFrontend.forward ic| self.last_kf_idx: 6 viz_index is empty, nothing to visualize Fusion packet from SLAM module is None... RaftVisualFrontend.forward ic| self.last_kf_idx: 7 Fusion packet from SLAM module is None... RaftVisualFrontend.forward ic| self.last_kf_idx: 7 Fusion packet from SLAM module is None... RaftVisualFrontend.forward ic| self.last_kf_idx: 7 Fusion packet from SLAM module is None... RaftVisualFrontend.forward ic| self.last_kf_idx: 7 Fusion packet from SLAM module is None... RaftVisualFrontend.forward ic| self.last_kf_idx: 7 Fusion packet from SLAM module is None... RaftVisualFrontend.forward ic| self.last_kf_idx: 7 ic| 'BA!' ic| 'ADDING initial prior!' ic| 'ADDING initial prior!' NANs in L_inv!! Traceback (most recent call last): File "/home/wjt/code/NeRF-SLAM/./examples/slam_demo.py", line 200, in run(args) File "/home/wjt/code/NeRF-SLAM/./examples/slam_demo.py", line 179, in run and (not slam or slam_module.spin()) \ File "/home/wjt/code/NeRF-SLAM/./examples/../pipeline/pipeline_module.py", line 101, in spin output = self.spin_once(input); File "/home/wjt/code/NeRF-SLAM/./examples/../slam/slam_module.py", line 11, in spin_once output = self.slam(input) File "/home/wjt/software/anaconda3/envs/nerfslam/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl return forward_call(*input, *kwargs) File "/home/wjt/code/NeRF-SLAM/./examples/../slam/meta_slam.py", line 30, in forward output = self._frontend(batch["data"], self.state, self.delta) File "/home/wjt/code/NeRF-SLAM/./examples/../slam/vio_slam.py", line 114, in _frontend x0_visual, visual_factors, viz_out = self.visual_frontend(batch) # TODO: currently also calls BA, and global BA File "/home/wjt/software/anaconda3/envs/nerfslam/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl return forward_call(input, kwargs) File "/home/wjt/code/NeRF-SLAM/./examples/../slam/visual_frontends/visual_frontend.py", line 326, in forward self.initialize() File "/home/wjt/code/NeRF-SLAM/./examples/../slam/visual_frontends/visual_frontend.py", line 650, in initialize x0, rcm_factor = self.update(kf0=None, kf1=None, use_inactive=True) File "/home/wjt/software/anaconda3/envs/nerfslam/lib/python3.9/site-packages/torch/amp/autocast_mode.py", line 12, in decorate_autocast return func(*args, **kwargs) File "/home/wjt/code/NeRF-SLAM/./examples/../slam/visual_frontends/visual_frontend.py", line 439, in update x0, rcm_factor = self.ba(gru_estimated_flow, gru_estimated_flow_weight, damping, File "/home/wjt/code/NeRF-SLAM/./examples/../slam/visual_frontends/visualfrontend.py", line 1176, in ba raise RuntimeError: No active exception to reraise Calling frontend dtor...

ChenHoy commented 1 year ago

Hi, I am experiencing the same issue. This happens consistently when I run the demo code on the example replica scene.

Any update on this?

ChenHoy commented 1 year ago

Hi, did anyone resolve this?

lihaowei1999 commented 2 weeks ago

Maybe you check if the stereo option in arg is True. This happens in my code when slam is using stereo=True with single view dataset. Change it to false and everything will be good.