Error for "engine: tensor" in "new_reconstruction_system"

[andapka]

Describe the bug I set engine == tensor in config.yaml file and run dense_slam.py I got the "RuntimeError: [Open3D Error]"

RuntimeError: [Open3D Error] (void open3d::t::pipelines::kernel::DecodeAndSolve6x6(const open3d::core::Tensor&, open3d::core::Tensor&, float&, int&)) /root/Open3D/cpp/open3d/t/pipelines/kernel/TransformationConverter.cpp:155: Singular 6x6 linear system detected, tracking failed.

To Reproduce Steps to reproduce the behavior:

• set engine as tensor
• run dense_slam.py
• See error

Expected behavior The process is running without any errors

Screenshots

Environment (please complete the following information): Operating system: Ubuntu 20.04 Python 3.8 Open3D version: development version How did you install Open3D?: pip install --pre https://storage.googleapis.com/open3d-releases-master/python-wheels/open3d-0.13.0+9efe03a-cp38-cp38-manylinux_2_27_x86_64.whl

Additional context I got also the warning (look at screenshot) "[Open3D WARNING] Property not found for color, return an empty tensor!" Maybe the main error connects with this one. What should I pay attention to?

[theNded]

The warning is not critical, it is mainly for the default ray casting. There are many reasons for tracking failure. Did you specify the correct intrinsic matrix or depth scale?

[andapka]

@theNded

I can check when set engine to "legacy" intrinsic.shape = (3, 3) intrinsic = [[3.97051699e+03 0.00000000e+00 1.66432301e+03] [0.00000000e+00 3.95871613e+03 1.61584663e+03] [0.00000000e+00 0.00000000e+00 1.00000000e+00]] It seems that matrix is correct. How can I check when intrinsic.matrix is a Tensor?

depth_scale == 1000.0 by default. Should I play with depth_scale?

Is it necessary to run on GPU when engine==tensor?

[theNded]

The tensor version is printable, just print(intrinsic). It looks good to me. I don't know which sensor you are using given this resolution, so depth_scale could be the most critical part here. GPU is not necessary, CPU also works but is much slower.

[andapka]

I'm using Ipad Pro 3024x4032 depth = 384x576 I received bad result with reconstruction_system, so I decide to try new one.

[theNded]

Then the intrinsic is wrong. It is for RGB image but I would say not for the depth. Currently we don't support RGB/depth with different resolutions/intrinsics, see https://github.com/isl-org/Open3D/issues/4104. We will try to add this feature. Before that, you can disable the color input, figure out the correct depth intrinsics and depth scale, and try again.

[andapka]

I know that RGB and Depth should have one resolution. That why when I preprocess images I upscale depth images to RGB. Any way, afterwards, I create RGBD images and apply o3d.pipelines.integration.ScalableTSDFVolume. I can do it due to transformation data from Apple devices. I get pretty good result but not for all cases. That why I'm trying to find better solution. If I have mistakes in my pipeline I'll appreciate for your advices.

[andapka]

If I use only depth images, what should be the pipeline?

[theNded]

Ok, sorry for the confusion -- it looks like intrinsics and scales are fine, as the reconstruction with o3d.pipelines.integration.ScalableTSDFVolume actually works.

The offline system is the best we have now in terms of accuracy, if it does not work then there is not much space for improvement. However, it would be good to find the actual depth intrinsics, since color and depth intrinsics are not necessarily the same even at the same resolution. Once you find it, rather than upsampling the depth, it might be better to downsample the color. Also, an alignment of depth and color could be important to ensure data association.

As of now, RGB images are not used for tracking, only for rendering, so any dummy color image with the same resolution of depth should work (e.g., a white image) -- I have supported that in the integration pipeline (not the SLAM pipeline) that will appear in a new PR. I should have constructed an interface for that, but unfortunately, I'm running out of time developing new features.

[andapka]

Thanks a lot! So, as far as I have intrinsic matrix for RGB images how can I convert it to intrinsic matrix for depth? They are already aligned (as far as I know). Maybe I don't need to convert and just downscale it according to depth resolution. But I have an issue in that case - how to convert transformation matrix. I think it includes elements according to RGB as well.

[theNded]

Sorry somehow the issue went off my timeline... Yes if they are aligned then rescaling could be sufficient, for fx, cx and fy and cy. Transformations need not to be changed -- intrinsic is in charge of camera-specific projection, and extrinsic is independently located in 3D.

[andapka]

Thanks a lot for your help. I fixed intrinsics matrix and the reconstructon system creates pretty good 3D object, but too smothed. Are there any approaches in open3d (or parameters) for increasing sharpness?

According to the current issue:

• does it make sense to try new_reconstruction_system in tensor mode now? Is it done?
• what parameters and dependencies to pay attention to?

[theNded]

As of now color map optimization may increase the sharpness. Tensor-based reconstruction is only half-done. It will only speed up the system and not improve accuracy, so I think at current sticking to the legacy system should be fine.

[andapka]

I used o3d.pipelines.color_map.ColorMapOptimizationOption() for color optimization. so I tried your proposal and the result is not good. Thanks. I'll try to find some workaround approaches. I also think about depth images quality. Iphone's rear camera returns depth in 256x192 resolution. So maybe it's impossible to increase the quality in this case. In any case, thanks a lot for your help.

[theNded]

Thanks for the feedback! Low resolution could be a problem for reconstruction. I personally would be interested in such hand-held device's data, but my capacity is limited for now. Will close the issue and let you know if I have any update later.