search

ToniRV / NeRF-SLAM

NeRF-SLAM: Real-Time Dense Monocular SLAM with Neural Radiance Fields. https://arxiv.org/abs/2210.13641 + Sigma-Fusion: Probabilistic Volumetric Fusion for Dense Monocular SLAM https://arxiv.org/abs/2210.01276
BSD 2-Clause "Simplified" License
1.17k stars 143 forks source link

AttributeError: 'gtsam.gtsam.GaussianFactorGraph' object has no attribute 'optimizeDensely' #5

Open binbin-xu opened 1 year ago

binbin-xu commented 1 year ago

I installed the gtsam 4.2a5 specified in the requirements.txt. However, There is a running error related to the gtsam function:

  File "./examples/slam_demo.py", line 200, in <module>
    run(args)
  File "./examples/slam_demo.py", line 179, in run
    and (not slam or slam_module.spin()) \
  File "/home/bxu/code/NeRF-SLAM/./examples/../pipeline/pipeline_module.py", line 101, in spin
    output = self.spin_once(input);
  File "/home/bxu/code/NeRF-SLAM/./examples/../slam/slam_module.py", line 11, in spin_once
    output = self.slam(input)
  File "/home/bxu/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/bxu/code/NeRF-SLAM/./examples/../slam/meta_slam.py", line 30, in forward
    output = self._frontend(batch["data"], self.state, self.delta)
  File "/home/bxu/code/NeRF-SLAM/./examples/../slam/vio_slam.py", line 114, in _frontend
    x0_visual, visual_factors, viz_out = self.visual_frontend(batch)  # TODO: currently also calls BA, and global BA
  File "/home/bxu/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/bxu/code/NeRF-SLAM/./examples/../slam/visual_frontends/visual_frontend.py", line 324, in forward
    self.__initialize()
  File "/home/bxu/code/NeRF-SLAM/./examples/../slam/visual_frontends/visual_frontend.py", line 648, in __initialize
    x0, rcm_factor = self.update(kf0=None, kf1=None, use_inactive=True)
  File "/home/bxu/.local/lib/python3.8/site-packages/torch/amp/autocast_mode.py", line 12, in decorate_autocast
    return func(*args, **kwargs)
  File "/home/bxu/code/NeRF-SLAM/./examples/../slam/visual_frontends/visual_frontend.py", line 437, in update
    x0, rcm_factor = self.ba(gru_estimated_flow, gru_estimated_flow_weight, damping,
  File "/home/bxu/code/NeRF-SLAM/./examples/../slam/visual_frontends/visual_frontend.py", line 1142, in ba
    gtsam_delta = linear_factor_graph.optimizeDensely() # Calls Eigen Cholesky, without a particularly smart ordering (vs Eigen::SimplicialLLt...)
AttributeError: 'gtsam.gtsam.GaussianFactorGraph' object has no attribute 'optimizeDensely'

It was tested using the following command: python ./examples/slam_demo.py --dataset_dir=./Datasets/Replica/office0 --dataset_name=nerf --buffer=100 --slam --img_stride=2 --fusion='nerf' --gui

When I checked the public functions in the gtsam.gtsam.GaussianFactorGraph class, it shows the following ones. It seems there is no python binder for optimizeDensely function. 

gtsam.gtsam.GaussianFactorGraph.add(                           gtsam.gtsam.GaussianFactorGraph.error(                         gtsam.gtsam.GaussianFactorGraph.optimize(
gtsam.gtsam.GaussianFactorGraph.at(                            gtsam.gtsam.GaussianFactorGraph.exists(                        gtsam.gtsam.GaussianFactorGraph.optimizeGradientSearch(
gtsam.gtsam.GaussianFactorGraph.augmentedHessian(              gtsam.gtsam.GaussianFactorGraph.gradient(                      gtsam.gtsam.GaussianFactorGraph.print(
gtsam.gtsam.GaussianFactorGraph.augmentedJacobian(             gtsam.gtsam.GaussianFactorGraph.gradientAtZero(                gtsam.gtsam.GaussianFactorGraph.printErrors(
gtsam.gtsam.GaussianFactorGraph.clone(                         gtsam.gtsam.GaussianFactorGraph.hessian(                       gtsam.gtsam.GaussianFactorGraph.probPrime(
gtsam.gtsam.GaussianFactorGraph.deserialize(                   gtsam.gtsam.GaussianFactorGraph.jacobian(                      gtsam.gtsam.GaussianFactorGraph.push_back(
gtsam.gtsam.GaussianFactorGraph.dot(                           gtsam.gtsam.GaussianFactorGraph.keyVector(                     gtsam.gtsam.GaussianFactorGraph.saveGraph(
gtsam.gtsam.GaussianFactorGraph.eliminateMultifrontal(         gtsam.gtsam.GaussianFactorGraph.keys(                          gtsam.gtsam.GaussianFactorGraph.serialize(
gtsam.gtsam.GaussianFactorGraph.eliminatePartialMultifrontal(  gtsam.gtsam.GaussianFactorGraph.marginal(                      gtsam.gtsam.GaussianFactorGraph.size(
gtsam.gtsam.GaussianFactorGraph.eliminatePartialSequential(    gtsam.gtsam.GaussianFactorGraph.marginalMultifrontalBayesNet(  gtsam.gtsam.GaussianFactorGraph.sparseJacobian_(
gtsam.gtsam.GaussianFactorGraph.eliminateSequential(           gtsam.gtsam.GaussianFactorGraph.mro(                           
gtsam.gtsam.GaussianFactorGraph.equals(                        gtsam.gtsam.GaussianFactorGraph.negate(
Wjt-shift commented 1 year ago

I also met the same problem,And I try other version gtsam 4.1.1,4.2a8,4.2a7,but can't solve this problem,what public function should instead of optimizeDensely? have you solved it ?

Wjt-shift commented 1 year ago

I also met the same problem,And I try other version gtsam 4.1.1,4.2a8,4.2a7,but can't solve this problem,what public function should instead of optimizeDensely? have you solved it ? But i find the function in code in gtsam c++ 4.2a5,GaussianFactorGraph.h defined the function optimizeDensely.But python api not that function.I don't know what caused it. image

ToniRV commented 1 year ago

Aha! Sorry for that, I have my own fork of gtsam as well. I will add that asap.

Wjt-shift commented 1 year ago

Have you solved this problem?

ToniRV commented 1 year ago

Here is the gtsam fork, and the branch is `feature/nerf_slam'. You need to checkout that branch to make it work.

https://github.com/ToniRV/gtsam-1/tree/feature/nerf_slam

I'll add to that to the Readme shortly.

ToniRV commented 1 year ago

You can see the changes I did here: https://github.com/ToniRV/gtsam-1/pull/1

I might as well try to just remove this dependency altogether, since it is not strictly necessary.

Wjt-shift commented 1 year ago

Here is the gtsam fork, and the branch is `feature/nerf_slam'. You need to checkout that branch to make it work.

https://github.com/ToniRV/gtsam-1/tree/feature/nerf_slam

I'll add to that to the Readme shortly.

Can you add Readme shortly about how to install your gtsam-1?Thank you a lot!

Coral79 commented 1 year ago

Hi, I got this error when 'make python-install'. Are there any hints on how to install gtsam-1(I have checked out the branch) correctly? image

Wjt-shift commented 1 year ago

Hi, I got this error when 'make python-install'. Are there any hints on how to install gtsam-1(I have checked out the branch) correctly? image have you solved this problem?

Coral79 commented 1 year ago

I still have the problem, haven't figured out how to solve it

SeoDU commented 1 year ago

I have solved the problem this way: Just clone the default branch of https://github.com/ToniRV/gtsam-1 and add the following code at line 408 and line 239 of gtsam/gtsam/linear/linear.i.

gtsam::VectorValues optimizeDensely() const; Vector vector(const gtsam::KeyVector& keys) const;

and make gtsam as in README.

It seems that the header of two functions for wrapping is missing.

Wjt-shift commented 1 year ago

I have solved the problem this way: Just clone the default branch of https://github.com/ToniRV/gtsam-1 and add the following code at line 408 and line 239 of gtsam/gtsam/linear/linear.i.

gtsam::VectorValues optimizeDensely() const; Vector vector(const gtsam::KeyVector& keys) const;

and make gtsam as in README.

It seems that the header of two functions for wrapping is missing.

I do that you said,the repo can compile successfully.But meet another problem,AttributeError: type object 'gtsam.gtsam.Pose3' has no attribute 'identity' Have you meet this problem?And how you solved it?

phamquandung commented 1 year ago

I have solved the problem this way: Just clone the default branch of https://github.com/ToniRV/gtsam-1 and add the following code at line 408 and line 239 of gtsam/gtsam/linear/linear.i. gtsam::VectorValues optimizeDensely() const; Vector vector(const gtsam::KeyVector& keys) const; and make gtsam as in README. It seems that the header of two functions for wrapping is missing.

I do that you said,the repo can compile successfully.But meet another problem,AttributeError: type object 'gtsam.gtsam.Pose3' has no attribute 'identity' Have you meet this problem?And how you solved it?

Just need to replace identity by Identity

jrpowers commented 1 year ago

This PR may address it all for you, can you check if it works? https://github.com/ToniRV/NeRF-SLAM/issues/12

szgy66 commented 1 year ago

I have solved the problem this way: Just clone the default branch of https://github.com/ToniRV/gtsam-1 and add the following code at line 408 and line 239 of gtsam/gtsam/linear/linear.i. gtsam::VectorValues optimizeDensely() const; Vector vector(const gtsam::KeyVector& keys) const; and make gtsam as in README. It seems that the header of two functions for wrapping is missing.

I do that you said,the repo can compile successfully.But meet another problem,AttributeError: type object 'gtsam.gtsam.Pose3' has no attribute 'identity' Have you meet this problem?And how you solved it?

Just need to replace identity by Identity

I replace the identity by Identity, but the error "AttributeError: type object 'gtsam.gtsam.Pose3' has no attribute 'Identity' " still remain. How should I do?

Coral79 commented 1 year ago

Hi, I got this error when 'make python-install'. Are there any hints on how to install gtsam-1(I have checked out the branch) correctly? image have you solved this problem?

Hi, I am wondering how did you solve this problem?

szgy66 commented 1 year ago

Hi, I got this error when 'make python-install'. Are there any hints on how to install gtsam-1(I have checked out the branch) correctly? image have you solved this problem?

Hi, I am wondering how did you solve this problem?

You should use: "git clone https://github.com/jrpowers/NeRF-SLAM.git --recurse-submodules". The rest of the operation can follow this.

pgaston commented 1 year ago

@szgy66 I've installed roughly 5 times over the last 3 weeks. Currently close, but the demo fails with a segfault... Ideas?

First, a note - before "python setup.py install" I needed to do "export PATH=/usr/local/cuda/bin:$PATH" - one of the pytorch things override your native nvcc. Also, I'm installing to CUDA 11.6, python 3.10.9, , pytorch 1.12.1. I have done "export CUDA_VISIBLE_DEVICES=0" (after trying without.)

For some reason, when I launch I get a segfault. Note that the os.environ['CUDA_VISIBLE_DEVICES']: '1' seems to have overridden my setting to '0'. Not sure if that is useful.

(nerf) pg@pg-MS-7A59:~/repos/NeRF-SLAM$ python ./examples/slam_demo.py --dataset_dir=./Datasets/Replica/office0 --dataset_name=nerf --buffer=100 --slam --parallel_run --img_stride=2 --fusion='nerf' --multi_gpu --gui
Running with GPUs: 0,1
Running pipeline in parallel mode.
I0118 09:44:08.855345 29211 pipeline_module.py:85] Module: VioSLAM - Spinning.
Aborted (core dumped)
(nerf) pg@pg-MS-7A59:~/repos/NeRF-SLAM$ I0118 09:44:09.368386 29213 pipeline_module.py:85] Module: nerf - Spinning.
I0118 09:44:09.384157 29214 pipeline_module.py:85] Module: nerf - Spinning.
ic| os.environ['CUDA_VISIBLE_DEVICES']: '1'
Process Process-2:
Traceback (most recent call last):
  File "/home/pg/miniconda3/envs/nerf/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/home/pg/miniconda3/envs/nerf/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/home/pg/repos/NeRF-SLAM/./examples/../pipeline/pipeline_module.py", line 88, in spin
    self.initialize_module()
  File "/home/pg/repos/NeRF-SLAM/./examples/../fusion/fusion_module.py", line 25, in initialize_module
    self.fusion = NerfFusion(self.name, self.args, self.device)
  File "/home/pg/repos/NeRF-SLAM/./examples/../fusion/nerf_fusion.py", line 65, in __init__
    self.ngp = ngp.Testbed(mode, 0) # NGP can only use device = 0
RuntimeError: Could not allocate memory: /home/pg/repos/NeRF-SLAM/thirdparty/instant-ngp/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:123 cudaMalloc(&rawptr, n_bytes+DEBUG_GUARD_SIZE*2) failed with error no CUDA-capable device is detected
Loading 1000 images.

Running the demo without --multi_gpu I quickly get a segfault, but at least I get the ngp window. Here are the first 10 lines or so...

(nerf) pg@pg-MS-7A59:~/repos/NeRF-SLAM$ python ./examples/slam_demo.py --dataset_dir=./Datasets/Replica/office0 --dataset_name=nerf --buffer=100 --slam --parallel_run --img_stride=2 --fusion='nerf' --gui
Running with GPUs: 0
Running pipeline in parallel mode.
I0118 09:33:40.423174 27145 pipeline_module.py:85] Module: VioSLAM - Spinning.
Aborted (core dumped)
(nerf) pg@pg-MS-7A59:~/repos/NeRF-SLAM$ I0118 09:33:40.969152 27147 pipeline_module.py:85] Module: nerf - Spinning.
I0118 09:33:40.970565 27148 pipeline_module.py:85] Module: nerf - Spinning.
ic| os.environ['CUDA_VISIBLE_DEVICES']: '0'
Loading 1000 images.
09:33:42 SUCCESS  NGP Device set to: 0
  0%|                                                  | 0/1000 [00:00<?, ?it/s]09:33:42 WARNING  Vulkan error: loaderValidateLayers: Layer 0 does not exist in the list of available layers
09:33:42 WARNING  Could not initialize Vulkan and NGX. DLSS not supported. (/home/pg/repos/NeRF-SLAM/thirdparty/instant-ngp/src/dlss.cu:220 vkCreateInstance(&instance_create_info, nullptr, &vk_instance) failed)
  1%|▎                                         | 6/1000 [00:00<00:20, 47.91it/s]09:33:42 INFO     Loading network config from: /home/pg/repos/NeRF-SLAM/thirdparty/instant-ngp/configs/nerf/base.json

(and keeps running, but no viewer shows.)

btw, Removing the '--multi-gpu' flag I still get the segfault - but at least the viewer starts...

Coral79 commented 1 year ago

Hi, I got this error when 'make python-install'. Are there any hints on how to install gtsam-1(I have checked out the branch) correctly? image have you solved this problem?

Hi, I am wondering how did you solve this problem?

You should use: "git clone https://github.com/jrpowers/NeRF-SLAM.git --recurse-submodules". The rest of the operation can follow this.

Thank you very much! This works for me, now I am able to run the demo

1005452649 commented 1 year ago

Hi,I meet a similar problem with you: AttributeError: 'gtsam.gtsam.GaussianFactorGraph' object has no attribute 'optimizeDensely' could you please tell me how you solved this error finally? @ToniRV @binbin-xu @pgaston

1005452649 commented 1 year ago

have solved this problem myself,but I meet a problem:Traceback (most recent call last): File "./examples/slam_demo.py", line 200, in run(args) File "./examples/slam_demo.py", line 143, in run slam_module.spin() # visualizer should be the main spin, but pytorch has a memory bug/leak if threaded... File "/home/user/linzejun01/linzejun_mutiply_view01/NeRF-SLAM/./examples/../pipeline/pipeline_module.py", line 101, in spin output = self.spin_once(input); File "/home/user/linzejun01/linzejun_mutiply_view01/NeRF-SLAM/./examples/../slam/slam_module.py", line 11, in spin_once output = self.slam(input) File "/home/user/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl return forward_call(*input, kwargs) File "/home/user/linzejun01/linzejun_mutiply_view01/NeRF-SLAM/./examples/../slam/meta_slam.py", line 30, in forward output = self._frontend(batch["data"], self.state, self.delta) File "/home/user/linzejun01/linzejun_mutiply_view01/NeRF-SLAM/./examples/../slam/vio_slam.py", line 119, in _frontend x0_visual, visual_factors, viz_out = self.visual_frontend(batch) # TODO: currently also calls BA, and global BA File "/home/user/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl return forward_call(*input, *kwargs) File "/home/user/linzejun01/linzejun_mutiply_view01/NeRF-SLAM/./examples/../slam/visual_frontends/visual_frontend.py", line 331, in forward self.initialize() File "/home/user/linzejun01/linzejun_mutiply_view01/NeRF-SLAM/./examples/../slam/visual_frontends/visual_frontend.py", line 655, in initialize x0, rcm_factor = self.update(kf0=None, kf1=None, use_inactive=True) File "/home/user/.local/lib/python3.8/site-packages/torch/amp/autocast_mode.py", line 14, in decorate_autocast return func(args, kwargs) File "/home/user/linzejun01/linzejun_mutiply_view01/NeRF-SLAM/./examples/../slam/visual_frontends/visual_frontend.py", line 444, in update x0, rcm_factor = self.ba(gru_estimated_flow, gru_estimated_flow_weight, damping, File "/home/user/linzejun01/linzejun_mutiply_view01/NeRF-SLAM/./examples/../slam/visual_frontends/visual_frontend.py", line 1143, in ba linear_factor_graph.push_back(vision_factors) TypeError: push_back(): incompatible function arguments. The following argument types are supported:

  1. (self: gtsam.gtsam.GaussianFactorGraph, factor: gtsam.gtsam.GaussianFactor) -> None
  2. (self: gtsam.gtsam.GaussianFactorGraph, conditional: gtsam::GaussianConditional) -> None
  3. (self: gtsam.gtsam.GaussianFactorGraph, graph: gtsam.gtsam.GaussianFactorGraph) -> None
  4. (self: gtsam.gtsam.GaussianFactorGraph, bayesNet: gtsam::GaussianBayesNet) -> None
  5. (self: gtsam.gtsam.GaussianFactorGraph, bayesTree: gtsam::GaussianBayesTree) -> None
Tonny24Wang commented 8 months ago

I installed the gtsam 4.2a5 specified in the requirements.txt. However, There is a running error related to the gtsam function:

  File "./examples/slam_demo.py", line 200, in <module>
    run(args)
  File "./examples/slam_demo.py", line 179, in run
    and (not slam or slam_module.spin()) \
  File "/home/bxu/code/NeRF-SLAM/./examples/../pipeline/pipeline_module.py", line 101, in spin
    output = self.spin_once(input);
  File "/home/bxu/code/NeRF-SLAM/./examples/../slam/slam_module.py", line 11, in spin_once
    output = self.slam(input)
  File "/home/bxu/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/bxu/code/NeRF-SLAM/./examples/../slam/meta_slam.py", line 30, in forward
    output = self._frontend(batch["data"], self.state, self.delta)
  File "/home/bxu/code/NeRF-SLAM/./examples/../slam/vio_slam.py", line 114, in _frontend
    x0_visual, visual_factors, viz_out = self.visual_frontend(batch)  # TODO: currently also calls BA, and global BA
  File "/home/bxu/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/bxu/code/NeRF-SLAM/./examples/../slam/visual_frontends/visual_frontend.py", line 324, in forward
    self.__initialize()
  File "/home/bxu/code/NeRF-SLAM/./examples/../slam/visual_frontends/visual_frontend.py", line 648, in __initialize
    x0, rcm_factor = self.update(kf0=None, kf1=None, use_inactive=True)
  File "/home/bxu/.local/lib/python3.8/site-packages/torch/amp/autocast_mode.py", line 12, in decorate_autocast
    return func(*args, **kwargs)
  File "/home/bxu/code/NeRF-SLAM/./examples/../slam/visual_frontends/visual_frontend.py", line 437, in update
    x0, rcm_factor = self.ba(gru_estimated_flow, gru_estimated_flow_weight, damping,
  File "/home/bxu/code/NeRF-SLAM/./examples/../slam/visual_frontends/visual_frontend.py", line 1142, in ba
    gtsam_delta = linear_factor_graph.optimizeDensely() # Calls Eigen Cholesky, without a particularly smart ordering (vs Eigen::SimplicialLLt...)
AttributeError: 'gtsam.gtsam.GaussianFactorGraph' object has no attribute 'optimizeDensely'

It was tested using the following command: python ./examples/slam_demo.py --dataset_dir=./Datasets/Replica/office0 --dataset_name=nerf --buffer=100 --slam --img_stride=2 --fusion='nerf' --gui

When I checked the public functions in the gtsam.gtsam.GaussianFactorGraph class, it shows the following ones. It seems there is no python binder for optimizeDensely function.

gtsam.gtsam.GaussianFactorGraph.add(                           gtsam.gtsam.GaussianFactorGraph.error(                         gtsam.gtsam.GaussianFactorGraph.optimize(
gtsam.gtsam.GaussianFactorGraph.at(                            gtsam.gtsam.GaussianFactorGraph.exists(                        gtsam.gtsam.GaussianFactorGraph.optimizeGradientSearch(
gtsam.gtsam.GaussianFactorGraph.augmentedHessian(              gtsam.gtsam.GaussianFactorGraph.gradient(                      gtsam.gtsam.GaussianFactorGraph.print(
gtsam.gtsam.GaussianFactorGraph.augmentedJacobian(             gtsam.gtsam.GaussianFactorGraph.gradientAtZero(                gtsam.gtsam.GaussianFactorGraph.printErrors(
gtsam.gtsam.GaussianFactorGraph.clone(                         gtsam.gtsam.GaussianFactorGraph.hessian(                       gtsam.gtsam.GaussianFactorGraph.probPrime(
gtsam.gtsam.GaussianFactorGraph.deserialize(                   gtsam.gtsam.GaussianFactorGraph.jacobian(                      gtsam.gtsam.GaussianFactorGraph.push_back(
gtsam.gtsam.GaussianFactorGraph.dot(                           gtsam.gtsam.GaussianFactorGraph.keyVector(                     gtsam.gtsam.GaussianFactorGraph.saveGraph(
gtsam.gtsam.GaussianFactorGraph.eliminateMultifrontal(         gtsam.gtsam.GaussianFactorGraph.keys(                          gtsam.gtsam.GaussianFactorGraph.serialize(
gtsam.gtsam.GaussianFactorGraph.eliminatePartialMultifrontal(  gtsam.gtsam.GaussianFactorGraph.marginal(                      gtsam.gtsam.GaussianFactorGraph.size(
gtsam.gtsam.GaussianFactorGraph.eliminatePartialSequential(    gtsam.gtsam.GaussianFactorGraph.marginalMultifrontalBayesNet(  gtsam.gtsam.GaussianFactorGraph.sparseJacobian_(
gtsam.gtsam.GaussianFactorGraph.eliminateSequential(           gtsam.gtsam.GaussianFactorGraph.mro(                           
gtsam.gtsam.GaussianFactorGraph.equals(                        gtsam.gtsam.GaussianFactorGraph.negate(

i have solved this problem by installing newest gtsam with the command " pip3 install gtsam==4.2a9 " , and you don't need to compile the gtsam source code.