Delay, noise and no output on Oak D Pro Wide depth camera when running specific commands on ROS2 Galactic.

jodusan commented 1 year ago

I am currently using the Oak D Pro Wide depth camera on Ubuntu 20.04 and galactic branch of repo and I am experiencing an issue when running the command

'ros2 launch depthai_examples stereo_inertial_node.launch.py' - the camera image in rviz has significant delay
When attempting to run the command 'ros2 launch depthai_examples stereo.launch.py' on the /stereo/depth image topic, there is a significant amount of noise present in the output, here is an image:
When the topic /stereo/converted_depth camera is chosen, nothing is displayed at all in rviz.

These issue is causing significant difficulties for me and I would greatly appreciate any assistance in resolving it.

Serafadam commented 1 year ago

Hi, few questions:

Is that USB or PoE camera?
stereo.launch.py is deprecated, I'd advise on working with stereo_intertial_node
Are you tied to Galactic distro? On Humble we have a brand new driver, it will be ported soon to Galactic, but take note that there are some caveats with Galactic distro itself (parameter namespacing etc.)

jodusan commented 1 year ago

It is an USB Camera
Ok I will try with stereo_inertial_node
I am because of Nvidia Jetson, but might be able to use Humble docker container. I am trying to use your depthai_ros humble container on my dev PC but it is not building successfuly.

When I try building with:

docker build --build-arg USE_RVIZ=1 -t depthai_ros .

On a Ubunutu 20.04 system (Intel PC), I get the following error

Sending build context to Docker daemon  157.1MB
Step 1/23 : ARG ROS_DISTRO=humble
Step 2/23 : FROM ros:${ROS_DISTRO}-ros-base
 ---> ff722dedce2c
Step 3/23 : ARG USE_RVIZ
 ---> Using cache
 ---> 64fb728ed5bc
Step 4/23 : ARG BUILD_SEQUENTIAL=0
 ---> Using cache
 ---> eb7cf73fa841
Step 5/23 : ENV DEBIAN_FRONTEND=noninteractive
 ---> Using cache
 ---> 34e0b93b3955
Step 6/23 : RUN apt-get update    && apt-get -y install --no-install-recommends software-properties-common git libusb-1.0-0-dev wget zsh python3-colcon-common-extensions
 ---> Running in 75d64a24d5dd
Get:1 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB]
Get:2 http://archive.ubuntu.com/ubuntu jammy InRelease [270 kB]
Get:3 http://packages.ros.org/ros2/ubuntu jammy InRelease [4673 B]
Get:4 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [114 kB]
Get:5 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [729 kB]
Get:6 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [99.8 kB]
Get:7 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [4732 B]
Get:8 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [786 kB]
Get:9 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [670 kB]
Get:10 http://archive.ubuntu.com/ubuntu jammy/multiverse amd64 Packages [266 kB]
Get:11 http://packages.ros.org/ros2/ubuntu jammy/main amd64 Packages [787 kB]
Get:12 http://archive.ubuntu.com/ubuntu jammy/restricted amd64 Packages [164 kB]
Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 Packages [1792 kB]
Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 Packages [17.5 MB]
Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [8978 B]
Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1043 kB]
Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [990 kB]
Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [718 kB]
Get:19 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [8738 B]
Get:20 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [3520 B]
Fetched 26.0 MB in 3s (8229 kB/s)
Reading package lists...
W: http://packages.ros.org/ros2/ubuntu/dists/jammy/InRelease: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details.
E: Problem executing scripts APT::Update::Post-Invoke 'rm -f /var/cache/apt/archives/*.deb /var/cache/apt/archives/partial/*.deb /var/cache/apt/*.bin || true'
E: Sub-process returned an error code

On a Ubuntu 22.04 Intel PC I get:

[+] Building 23.6s (12/19)
 => [internal] load build definition from Dockerfile                                                                                                                                                          0.0s
 => => transferring dockerfile: 38B                                                                                                                                                                           0.0s
 => [internal] load .dockerignore                                                                                                                                                                             0.0s
 => => transferring context: 2B                                                                                                                                                                               0.0s
 => [internal] load metadata for [docker.io/library/ros:humble-ros-base](http://docker.io/library/ros:humble-ros-base)                                                                                                                                        0.6s
 => [internal] load build context                                                                                                                                                                             0.0s
 => => transferring context: 14.54kB                                                                                                                                                                          0.0s
 => [ 1/15] FROM [docker.io/library/ros:humble-ros-base@sha256:187938e57357e2e1180ada88ddeb89bd54c156e444d09a1d92442c804c1cdf96](http://docker.io/library/ros:humble-ros-base@sha256:187938e57357e2e1180ada88ddeb89bd54c156e444d09a1d92442c804c1cdf96)                                                                                0.0s
 => CACHED [ 2/15] RUN apt-get update    && apt-get -y install --no-install-recommends software-properties-common git libusb-1.0-0-dev wget zsh python3-colcon-common-extensions                              0.0s
 => CACHED [ 3/15] RUN apt install -y ros-humble-rmw-cyclonedds-cpp                                                                                                                                           0.0s
 => CACHED [ 4/15] RUN sh -c "$(wget https://raw.github.com/ohmyzsh/ohmyzsh/master/tools/install.sh -O -)"                                                                                                    0.0s
 => CACHED [ 5/15] RUN mkdir -p /ws/src                                                                                                                                                                       0.0s
 => CACHED [ 6/15] COPY ./ ./ws/src/depthai-ros                                                                                                                                                               0.0s
 => CACHED [ 7/15] RUN cd ./ws/ && rosdep install --from-paths src --ignore-src  -y                                                                                                                           0.0s
 => ERROR [ 8/15] RUN cd ./ws/ && . /opt/ros/humble/setup.sh && ./src/depthai-ros/build.sh -s 0 -r 1 -m 1                                                                                                    22.9s
------
 > [ 8/15] RUN cd ./ws/ && . /opt/ros/humble/setup.sh && ./src/depthai-ros/build.sh -s 0 -r 1 -m 1:
#12 0.311 Build type: Release, Install_type: merge-install
#12 0.311 Parallel build
#12 0.660 Starting >>> depthai_ros_msgs
#12 5.522 Finished <<< depthai_ros_msgs [4.86s]
#12 5.523 Starting >>> depthai_bridge
#12 22.73 --- stderr: depthai_bridge
#12 22.73 c++: fatal error: Killed signal terminated program cc1plus
#12 22.73 compilation terminated.
#12 22.73 gmake[2]: *** [CMakeFiles/depthai_bridge.dir/build.make:118: CMakeFiles/depthai_bridge.dir/src/ImuConverter.cpp.o] Error 1
#12 22.73 gmake[2]: *** Waiting for unfinished jobs....
#12 22.73 gmake[1]: *** [CMakeFiles/Makefile2:137: CMakeFiles/depthai_bridge.dir/all] Error 2
#12 22.73 gmake: *** [Makefile:146: all] Error 2
#12 22.73 ---
#12 22.73 Failed   <<< depthai_bridge [17.2s, exited with code 2]
#12 22.79
#12 22.79 Summary: 1 package finished [22.2s]
#12 22.79   1 package failed: depthai_bridge
#12 22.79   1 package had stderr output: depthai_bridge

I will try building from source on the 22.04, but container doesnt seem to be working?

Serafadam commented 1 year ago

That's probably due to RAM limitations, can you try again with BUILD_SEQUENTIAL=1?

jodusan commented 1 year ago

Container build works :+1: on Ubuntu 22.04 host with BUILD_SEQUENTIAL=1 (how much ram is needed?) flag
Should the humble branch build on Ubuntu 20.04 host successfully?

but upon running the command (and sharing host folder /tmp/.X11-unix from Docker ui)

docker run -it -v /dev/:/dev/  --privileged -e DISPLAY -v /tmp/.X11-unix:/tmp/.X11-unix depthai_ros

I get the following error:

docker: Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "/ws/src/depthai_ros/entrypoint.sh": stat /ws/src/depthai_ros/entrypoint.sh: no such file or directory: unknown.
ERRO[0000] error waiting for container: context canceled

jodusan commented 1 year ago

I have managed to build from source on the 22.04 Ubuntu host, it seems to have less delay in Rviz (maybe 0.3-0.5s), but there are still missing points on the depth data. Is it possible to run some neural network on the device to give a more smooth depth surface with no "holes" ?

Serafadam commented 1 year ago

I get the following error: :+1: There was a minor typo, it's fixed now. On changing depth quality I'd suggest using depthai_ros_driver packages as you can set various depth filtering algorithms there ( see here ). As for the "Neural depth`, there are some example, but FPS is much lower. This week we're going to add host-side WLS filter, which improves holes in depth images https://github.com/luxonis/depthai-experiments/tree/master/gen2-wls-filter

jodusan commented 1 year ago

Good to know it was such an easy fix!

How much is FPS lower for neural depth? The reason I've chose this camera is because ZED cameras don't have onboard processing and use host-side comp power. In my case this is not possible since I have 3 luxonis cameras. Can you point to "some examples" where fps is much slower (10-15 would do it in my case). THanks!

Serafadam commented 1 year ago

You can check out some discussion here https://github.com/luxonis/depthai/issues/173, there might be some development here in upcoming weeks, what I can suggest for depth quality improvements is to play with filter settings for your use-case (more information here https://docs.luxonis.com/projects/api/en/latest/components/nodes/stereo_depth/). Also using cameras with IR diodes definitely helps in case of flat surfaces, as demonstrated here param_stereo

luxonis / depthai-ros

Delay, noise and no output on Oak D Pro Wide depth camera when running specific commands on ROS2 Galactic. #200