search

isl-org / Open3D

Open3D: A Modern Library for 3D Data Processing
http://www.open3d.org
Other
11.29k stars 2.29k forks source link

How to run dense_slam_gui example with data taken from a stationary camera? #6311

Open halil-karabacak opened 1 year ago

halil-karabacak commented 1 year ago

Checklist

My Question

Hi. I am using open3d 0.17 with python 3.9 using cuda on Windows 10. I can succesfully run the example with the LoungeRGBDImages dataset. My problem is currently I don't have a portable camera, so I tried to copy the pipeline by rotating the object instead of moving the camera and took around 35 frames, which I thought would work. I captured depth maps in 16-bit alongside the rgb images. Finally, I updated the intrinsic matrix and printed it to check if I am going right and printed result is what I expected. I also changed width and height in default_intrinsic.json file. However I am getting 

[Open3D Error] (void __cdecl open3d::core::OPEN3D_LAPACK_CHECK(__int64,const class std::basic_string<char,struct std::char_traits<char>,class std::allocator<char> > &)) ...\Open3D-0.17.0\cpp\open3d/core/linalg/LinalgUtils.h:40: gels failed in SolveCPU: singular condition detected.

[Open3D Error] (void __cdecl open3d::t::pipelines::kernel::DecodeAndSolve6x6(const class open3d::core::Tensor &,class open3d::core::Tensor &,float &,int &)) ...\Open3D-0.17.0\cpp\open3d\t\pipelines\kernel\TransformationConverter.cpp:129: Singular 6x6 linear system detected, tracking failed.

Exception in thread UpdateMain:
Traceback (most recent call last):
  File "...\Python39\lib\threading.py", line 954, in _bootstrap_inner
    self.run()
  File "...\Python39\lib\threading.py", line 892, in run
    self._target(*self._args, **self._kwargs)
  File "...\Open3D-0.17.0\examples\python\t_reconstruction_system\dense_slam_gui.py", line 373, in update_main
    result = self.model.track_frame_to_model(
RuntimeError: [Open3D Error] (void __cdecl open3d::t::pipelines::kernel::DecodeAndSolve6x6(const class open3d::core::Tensor &,class open3d::core::Tensor &,float &,int &)) ...\Open3D-0.17.0\cpp\open3d\t\pipelines\kernel\TransformationConverter.cpp:129: Singular 6x6 linear system detected, tracking failed.

My humble guess is that this error might be related to my stationary camera. However, I am not familiar with the pipeline and this is my first attempt of running & understanding it. I also couldn't understand the meaning of depth_scale Any comment would be appreciated, thanks.

theNded commented 1 year ago

Unfortunately the algorithm only works for a moving camera pointing at static scenes. Tracking heavily relies on this assumption, and if it is violated, failure is inevitable. The meaning of depth_scale is the unit of the depth image. Usually it is 1000.0, meaning that the pixel values are measured in millimeters, and depth_scale converts it to meters.

halil-karabacak commented 1 year ago

Thanks @theNded ! I wonder if there is workaround of this problem? Like maybe passing identity matrix to tracking etc. but this time I think tsdf would blow up, but just to confirm.

theNded commented 1 year ago

I think your best shot would be https://github.com/isl-org/Open3D/blob/master/examples/python/t_reconstruction_system/integrate.py. You can fabricate a path_trajectory file with all identity poses, or simply replace extrinsics in this line to be identity: https://github.com/isl-org/Open3D/blob/master/examples/python/t_reconstruction_system/integrate.py#L53

halil-karabacak commented 1 year ago

Thank you @theNded. I will try it. And do I need to have poses of the camera for each frame if I moved to a portable camera? I guess I will only have depth maps and RGB images, idk even if this is doable. I mean if I need to have poses (.log) files than probably I can't use it because I will have a high FPS ~ 500/600FPS low ROI camera

FraDePace commented 6 months ago

@halil-karabacak can you please share the command you used to compile with CUDA on Windows?

halil-karabacak commented 6 months ago

@halil-karabacak can you please share the command you used to compile with CUDA on Windows?

@FraDePace I don't exactly remember the commands but I build from source as explained in the official doc https://www.open3d.org/docs/latest/compilation.html