ZED-X data timestamps not updated in docker image

bmegli commented 8 months ago

Preliminary Checks

[X] This issue is not a duplicate. Before opening a new issue, please search existing issues.
[X] This issue is not a question, feature request, or anything other than a bug report directly related to this project.

Description

I am using:

ZED-X + ZED-XM
on Jetson AGX Orin
from within docker container

Cameras work and output data but with same timestamp all the time. This breaks other parts of the system that are depending on timestamp.

Steps to Reproduce

Start docker container

docker run --runtime nvidia -it --privileged --ipc=host --network=host --pid=host -e DISPLAY   -v /dev/shm:/dev/shm   -v /tmp/:/tmp/ -v /var/nvidia/nvcam/settings/:/var/nvidia/nvcam/settings/   -v ${HOME}/zed_docker_ai/:/usr/local/zed/resources/    CONTAINER_TAG

Start ZED-X node

roslaunch zed_wrapper zedx.launch &

Check up timestamps (also TF warnings)

rostopic echo zedx/zed_node/rgb/image_rect_color --noarr

Expected Result

Timestamp changing for all data (depth, RGB, ...)

Actual Result

Same timestamp published all the time

nsec is interchaning between two close values, otherwise stamp doesn't change

There are also warnings from TF on duplicate timestamps

rostopic echo zedx/zed_node/rgb/image_rect_color --noarr

# ...

header: 
  seq: 192
  stamp: 
    secs: 1698330340
    nsecs: 529478000
  frame_id: "zedx_left_camera_optical_frame"
height: 600
width: 960
encoding: "bgra8"
is_bigendian: 0
step: 3840
data: "<array type: uint8, length: 2304000>"
---
Warning: TF_REPEATED_DATA ignoring data with redundant timestamp for frame base_link at time 1698330340.529479 according to authority unknown_publisher
         at line 278 in /tmp/binarydeb/ros-noetic-tf2-0.7.6/src/buffer_core.cpp
Warning: TF_REPEATED_DATA ignoring data with redundant timestamp for frame odom at time 1698330340.529479 according to authority unknown_publisher
         at line 278 in /tmp/binarydeb/ros-noetic-tf2-0.7.6/src/buffer_core.cpp
header: 
  seq: 193
  stamp: 
    secs: 1698330340
    nsecs: 529479000
  frame_id: "zedx_left_camera_optical_frame"
height: 600
width: 960
encoding: "bgra8"
is_bigendian: 0
step: 3840
data: "<array type: uint8, length: 2304000>"
---
Warning: TF_REPEATED_DATA ignoring data with redundant timestamp for frame base_link at time 1698330340.529478 according to authority unknown_publisher
         at line 278 in /tmp/binarydeb/ros-noetic-tf2-0.7.6/src/buffer_core.cpp
Warning: TF_REPEATED_DATA ignoring data with redundant timestamp for frame odom at time 1698330340.529478 according to authority unknown_publisher
         at line 278 in /tmp/binarydeb/ros-noetic-tf2-0.7.6/src/buffer_core.cpp
header: 
  seq: 194
  stamp: 
    secs: 1698330340
    nsecs: 529478000
  frame_id: "zedx_left_camera_optical_frame"
height: 600
width: 960
encoding: "bgra8"
is_bigendian: 0
step: 3840
data: "<array type: uint8, length: 2304000>"
---
Warning: TF_REPEATED_DATA ignoring data with redundant timestamp for frame base_link at time 1698330340.529479 according to authority unknown_publisher
         at line 278 in /tmp/binarydeb/ros-noetic-tf2-0.7.6/src/buffer_core.cpp
Warning: TF_REPEATED_DATA ignoring data with redundant timestamp for frame odom at time 1698330340.529479 according to authority unknown_publisher
         at line 278 in /tmp/binarydeb/ros-noetic-tf2-0.7.6/src/buffer_core.cpp
header: 
  seq: 195
  stamp: 
    secs: 1698330340
    nsecs: 529479000
  frame_id: "zedx_left_camera_optical_frame"
height: 600
width: 960
encoding: "bgra8"
is_bigendian: 0
step: 3840
data: "<array type: uint8, length: 2304000>"
---
Warning: TF_REPEATED_DATA ignoring data with redundant timestamp for frame base_link at time 1698330340.529478 according to authority unknown_publisher
         at line 278 in /tmp/binarydeb/ros-noetic-tf2-0.7.6/src/buffer_core.cpp
Warning: TF_REPEATED_DATA ignoring data with redundant timestamp for frame odom at time 1698330340.529478 according to authority unknown_publisher
         at line 278 in /tmp/binarydeb/ros-noetic-tf2-0.7.6/src/buffer_core.cpp
header: 
  seq: 196
  stamp: 
    secs: 1698330340
    nsecs: 529478000
  frame_id: "zedx_left_camera_optical_frame"
height: 600
width: 960
encoding: "bgra8"
is_bigendian: 0
step: 3840
data: "<array type: uint8, length: 2304000>"

ZED Camera model

ZED-X

Environment

OS
- Host   - L4T R35 2.1
- Docker - L4T R35 3
  - to be investigated if above mismatch has any impact (but will take hours)

CPU/GPU - Jetson AGX Orin

ZED SDK
- Host - `ZED SDK for JetPack 5.1 (L4T 35.2) 4.0 (Jetson Xavier, Orin AGX/NX 16GB, CUDA 11.4)`
- Docker- image based on [stereolabs/zed:4.0-runtime-l4t-r35.3](https://github.com/stereolabs/zed-docker/blob/fde3fe833859eb78831b0cc4c0a119b745f6ff62/4.X/l4t/runtime/Dockerfile#L20)
  - so likely `ZED SDK for JetPack 5.1.1 (L4T 35.3)4.0 (Jetson Xavier, Orin AGX/NX/Nano, CUDA 11.4)`
  - mismatch to be investigated (but it will take hours)

ROS Noetic installed from apt

zed_ros_wrapper tested
- current master 71eb2bb434f059e17191503b707267938f5a1b7f (as of 27 Oct 2023)
- but also own fork based on 50dda077a5ea72b7f8c00faad5746938331e46c2 (10th May 2023)

GMSL driver (Docker host)

sudo dpkg -i stereolabs-zedx_0.5.1-MAX96712-L4T35.2_arm64.deb

Anything else?

Docker image is build following way

# https://github.com/stereolabs/zed-docker/tree/master/4.X/l4t
FROM stereolabs/zed:4.0-runtime-l4t-r35.3

# mainly follow with ROS Noetic install from apt

As next step I am going to build from matching L4T + ZED-SDK version (between host and docker) and re-test but this will take hours.

Docker image was cross-built on x86_64

alassagne-sl commented 8 months ago

Hi,

It's a mismatch between the L4T version of your host (35.2) and your container (35.3). We'll make more clear in the documentation that a match is required.

bmegli commented 8 months ago

@alassagne-sl,

Thanks for a quick reply.

Partially expected as I am during image build from matching versions.

bmegli commented 8 months ago

Confirmed working when docker host and container L4T match.

From practical point of view it is a bit unfortunate:

as this means that host and L4T versions have to match
which defeats some (not all) benefits of using containers

It's a mismatch between the L4T version of your host (35.2) and your container (35.3). We'll make more clear in the documentation that a match is required.

I expect reason to be some hardware/driver/container/SDK interaction.

Nonetheless it would be great to understand reason and extent of potential (in)compatibility

in the context of long-term software maintenance
in the context of future coexistence with other software (and dependence on L4T versions)

obraun-sl commented 8 months ago

Hi,

The reason behind is that the driver and user-land SDKs are built with respect to NVIDIA Argus API structure on both side (kernel && user) Since it has changed between L4T 35.2 and 35.3 (on the timestamp structure especially) which is "only" binary changes, there is no errors triggered but the value gets invalid. Normally it should not change much starting from 35.3 but that's on NVIDIA hands.

bmegli commented 8 months ago

@obraun-sl

Thank you kindly for above clarification.

It has some impact as this means that:

matching L4T has guaranteed compatibility
non-matching L4T has potential compatibility which can be traced and tested

stereolabs / zed-ros-wrapper