[BUG] Camera field of view incorrect in foxglove

FabienDanieau commented 1 year ago

Describe the bug Hi, I am testing depthai-ros inside a docker for now, version 2.7.2. I'd like to use foxglove to monitor what's happening but the camera FoV is not properly displayed in the 3D view. I am not sure if this is a visual problem (foxglove) or if the camera_info is incorrect.

I have a OAK-D-Pro

Minimal Reproducible Example I just ran the basic example ros2 launch depthai_ros_driver camera.launch.py

and I logged everyting in a rosbag ros2 bag record -a -s sqlite3

When I open the *.db3 in foxglove, I see that the camera field of view is not properly rendered (see screenshot). The camera_info message in foxglove from the .db3 is the same that the one I observe in the docker with ros2 topic info /oak/rgb/camera_info . The logging is ok then.

Screenshot

The expected FoV should be a "rectangle" with the camera image inside.

Additional context FYI here the calibration info of the camera. I don't really see these values in the camera_info message.

$ python examples/calibration/calibration_reader.py 
RGB Camera Default intrinsics...
[[1583.7469482421875, 0.0, 958.818603515625], [0.0, 1581.7431640625, 537.4307250976562], [0.0, 0.0, 1.0]]
1920
1080
RGB Camera Default intrinsics...
[[1583.7469482421875, 0.0, 958.818603515625], [0.0, 1581.7431640625, 537.4307250976562], [0.0, 0.0, 1.0]]
1920
1080
RGB Camera resized intrinsics... 3840 x 2160 
[[3.16749390e+03 0.00000000e+00 1.91763721e+03]
 [0.00000000e+00 3.16348633e+03 1.07486145e+03]
 [0.00000000e+00 0.00000000e+00 1.00000000e+00]]
RGB Camera resized intrinsics... 4056 x 3040 
[[3.34566528e+03 0.00000000e+00 2.02550427e+03]
 [0.00000000e+00 3.34143237e+03 1.51457239e+03]
 [0.00000000e+00 0.00000000e+00 1.00000000e+00]]
LEFT Camera Default intrinsics...
[[787.0020751953125, 0.0, 680.6743774414062], [0.0, 793.16650390625, 400.1222839355469], [0.0, 0.0, 1.0]]
1280
800
LEFT Camera resized intrinsics...  1280 x 720
[[787.0020752    0.         680.67437744]
 [  0.         793.16650391 360.12228394]
 [  0.           0.           1.        ]]
RIGHT Camera resized intrinsics... 1280 x 720
[[795.56970215   0.         664.75164795]
 [  0.         801.89373779 369.28134155]
 [  0.           0.           1.        ]]
LEFT Distortion Coefficients...
k1: -4.595492839813232
k2: 69.36243438720703
p1: -0.00017447950085625052
p2: -0.002563486574217677
k3: -104.61964416503906
k4: -4.706033706665039
k5: 69.4855728149414
k6: -104.48624420166016
s1: 0.0
s2: 0.0
s3: 0.0
s4: 0.0
τx: 0.0
τy: 0.0
RIGHT Distortion Coefficients...
k1: -10.662757873535156
k2: 80.89090728759766
p1: -0.002335291588678956
p2: -6.226611731108278e-05
k3: -86.80615234375
k4: -10.712331771850586
k5: 80.81675720214844
k6: -86.43338775634766
s1: 0.0
s2: 0.0
s3: 0.0
s4: 0.0
τx: 0.0
τy: 0.0
RGB FOV 68.7938003540039, Mono FOV 71.86000061035156
LEFT Camera stereo rectification matrix...
[[ 1.02068902e+00 -3.86466868e-04 -3.92094130e+01]
 [ 5.46482739e-03  1.01077461e+00  1.92726729e+00]
 [ 1.48501431e-05 -6.18235423e-07  9.90046102e-01]]
RIGHT Camera stereo rectification matrix...
[[ 1.00969704e+00 -3.82260841e-04 -1.56483376e+01]
 [ 5.40597573e-03  9.99774063e-01 -3.14204365e+00]
 [ 1.46902193e-05 -6.11506994e-07  9.90392044e-01]]
Transformation matrix of where left Camera is W.R.T right Camera's optical center
[[ 9.99963284e-01  4.54613054e-03 -7.26302294e-03 -7.41231012e+00]
 [-4.53881081e-03  9.99989152e-01  1.02396740e-03  3.35490555e-02]
 [ 7.26759946e-03 -9.90964356e-04  9.99973118e-01  3.27607654e-02]
 [ 0.00000000e+00  0.00000000e+00  0.00000000e+00  1.00000000e+00]]
Transformation matrix of where left Camera is W.R.T RGB Camera's optical center
[[ 9.99932230e-01 -1.37393386e-03 -1.15534477e-02 -3.62703061e+00]
 [ 1.38474395e-03  9.99998629e-01  9.27659392e-04 -2.63648406e-02]
 [ 1.15521559e-02 -9.43595252e-04  9.99932885e-01  1.13737524e-01]
 [ 0.00000000e+00  0.00000000e+00  0.00000000e+00  1.00000000e+00]]

FabienDanieau commented 1 year ago

Here is the output from the docker. There are also errors related to color/images conversion

root@docker:/# ros2 launch depthai_ros_driver camera.launch.py
[INFO] [launch]: All log files can be found below /root/.ros/log/2023-05-25-09-27-48-375047-lotus-532
[INFO] [launch]: Default logging verbosity is set to INFO
[INFO] [robot_state_publisher-1]: process started with pid [546]
[INFO] [component_container-2]: process started with pid [548]
[robot_state_publisher-1] [INFO] [1685006868.516664340] [oak_state_publisher]: got segment oak
[robot_state_publisher-1] [INFO] [1685006868.516711874] [oak_state_publisher]: got segment oak-d-base-frame
[robot_state_publisher-1] [INFO] [1685006868.516717133] [oak_state_publisher]: got segment oak_imu_frame
[robot_state_publisher-1] [INFO] [1685006868.516720400] [oak_state_publisher]: got segment oak_left_camera_frame
[robot_state_publisher-1] [INFO] [1685006868.516723932] [oak_state_publisher]: got segment oak_left_camera_optical_frame
[robot_state_publisher-1] [INFO] [1685006868.516726962] [oak_state_publisher]: got segment oak_model_origin
[robot_state_publisher-1] [INFO] [1685006868.516729856] [oak_state_publisher]: got segment oak_rgb_camera_frame
[robot_state_publisher-1] [INFO] [1685006868.516732888] [oak_state_publisher]: got segment oak_rgb_camera_optical_frame
[robot_state_publisher-1] [INFO] [1685006868.516735917] [oak_state_publisher]: got segment oak_right_camera_frame
[robot_state_publisher-1] [INFO] [1685006868.516738840] [oak_state_publisher]: got segment oak_right_camera_optical_frame
[component_container-2] [INFO] [1685006868.751586720] [oak_container]: Load Library: /ws/install/lib/libdepthai_ros_driver.so
[component_container-2] [INFO] [1685006868.788958702] [oak_container]: Found class: rclcpp_components::NodeFactoryTemplate<depthai_ros_driver::Camera>
[component_container-2] [INFO] [1685006868.789009459] [oak_container]: Instantiate class: rclcpp_components::NodeFactoryTemplate<depthai_ros_driver::Camera>
[component_container-2] [INFO] [1685006868.792988488] [oak]: No ip/mxid specified, connecting to the next available device.
[component_container-2] [INFO] [1685006871.627235896] [oak]: Camera with MXID: 18443010E198000900 and Name: 3.4 connected!
[component_container-2] [INFO] [1685006871.628217407] [oak]: USB SPEED: SUPER
[component_container-2] [INFO] [1685006871.649019585] [oak]: Device type: OAK-D-PRO
[component_container-2] [INFO] [1685006871.651225204] [oak]: Pipeline type: RGBD
[component_container-2] [INFO] [1685006871.659113650] [oak]: NN Family: mobilenet
[component_container-2] [INFO] [1685006871.699100900] [oak]: NN input size: 300 x 300. Resizing input image in case of different dimensions.
[component_container-2] [INFO] [1685006871.953077332] [oak]: Finished setting up pipeline.
[component_container-2] [INFO] [1685006872.438657932] [oak]: Camera ready!
[INFO] [launch_ros.actions.load_composable_nodes]: Loaded node '/oak' in container '/oak_container'
[component_container-2] [ERROR] [1685006879.396425498] [compressed_depth_image_transport]: Compressed Depth Image Transport - Compression requires single-channel 32bit-floating point or 16bit raw depth images (input format is: bgr8).
[component_container-2] [WARN] [1685006879.401038046] [TheoraPublisher]: Couldn't set keyframe frequency to 0, actually set to 1
[component_container-2] [ERROR] [1685006879.460428225] [compressed_depth_image_transport]: Compressed Depth Image Transport - Compression requires single-channel 32bit-floating point or 16bit raw depth images (input format is: bgr8).
[component_container-2] [ERROR] [1685006879.489638166] [compressed_depth_image_transport]: Compressed Depth Image Transport - Compression requires single-channel 32bit-floating point or 16bit raw depth images (input format is: bgr8).
[component_container-2] [ERROR] [1685006879.518837971] [compressed_depth_image_transport]: Compressed Depth Image Transport - Compression requires single-channel 32bit-floating point or 16bit raw depth images (input format is: bgr8).
[component_container-2] [ERROR] [1685006879.539800358] [compressed_depth_image_transport]: Compressed Depth Image Transport - Compression requires single-channel 32bit-floating point or 16bit raw depth images (input format is: bgr8).
[component_container-2] [ERROR] [1685006879.560707274] [compressed_depth_image_transport]: Compressed Depth Image Transport - Compression requires single-channel 32bit-floating point or 16bit raw depth images (input format is: bgr8).
[component_container-2] [ERROR] [1685006879.659508245] [CompressedPublisher]: [16UC1] is not a color format. but [mono8] is. The conversion does not make sense
[component_container-2] [WARN] [1685006879.660800633] [TheoraPublisher]: Couldn't set keyframe frequency to 0, actually set to 1
[component_container-2] [ERROR] [1685006879.661175663] [TheoraPublisher]: cv_bridge exception: '[16UC1] is not a color format. but [bgr8] is. The conversion does not make sense'
[component_container-2] [ERROR] [1685006879.917993480] [CompressedPublisher]: [16UC1] is not a color format. but [mono8] is. The conversion does not make sense
[component_container-2] [ERROR] [1685006879.918584060] [TheoraPublisher]: cv_bridge exception: '[16UC1] is not a color format. but [bgr8] is. The conversion does not make sense'
[component_container-2] [ERROR] [1685006879.943404527] [compressed_depth_image_transport]: Compressed Depth Image Transport - Compression requires single-channel 32bit-floating point or 16bit raw depth images (input format is: bgr8).
[component_container-2] [ERROR] [1685006880.181582151] [CompressedPublisher]: [16UC1] is not a color format. but [mono8] is. The conversion does not make sense
[component_container-2] [ERROR] [1685006880.182243000] [TheoraPublisher]: cv_bridge exception: '[16UC1] is not a color format. but [bgr8] is. The conversion does not make sense'
[component_container-2] [ERROR] [1685006880.217680381] [compressed_depth_image_transport]: Compressed Depth Image Transport - Compression requires single-channel 32bit-floating point or 16bit raw depth images (input format is: bgr8).
[component_container-2] [ERROR] [1685006880.240037540] [compressed_depth_image_transport]: Compressed Depth Image Transport - Compression requires single-channel 32bit-floating point or 16bit raw depth images (input format is: bgr8).
[component_container-2] [ERROR] [1685006880.457151118] [CompressedPublisher]: [16UC1] is not a color format. but [mono8] is. The conversion does not make sense
[component_container-2] [ERROR] [1685006880.457973952] [TheoraPublisher]: cv_bridge exception: '[16UC1] is not a color format. but [bgr8] is. The conversion does not make sense'
[component_container-2] [ERROR] [1685006880.490084000] [compressed_depth_image_transport]: Compressed Depth Image Transport - Compression requires single-channel 32bit-floating point or 16bit raw depth images (input format is: bgr8).
[component_container-2] [ERROR] [1685006880.711212735] [CompressedPublisher]: [16UC1] is not a color format. but [mono8] is. The conversion does not make sense
[component_container-2] [ERROR] [1685006880.711786459] [TheoraPublisher]: cv_bridge exception: '[16UC1] is not a color format. but [bgr8] is. The conversion does not make sense'
[component_container-2] [ERROR] [1685006880.738354177] [compressed_depth_image_transport]: Compressed Depth Image Transport - Compression requires single-channel 32bit-floating point or 16bit raw depth images (input format is: bgr8).
[component_container-2] [ERROR] [1685006880.970190782] [CompressedPublisher]: [16UC1] is not a color format. but [mono8] is. The conversion does not make sense
[component_container-2] [ERROR] [1685006880.970777968] [TheoraPublisher]: cv_bridge exception: '[16UC1] is not a color format. but [bgr8] is. The conversion does not make sense'
[component_container-2] [ERROR] [1685006881.239124908] [CompressedPublisher]: [16UC1] is not a color format. but [mono8] is. The conversion does not make sense
[component_container-2] [ERROR] [1685006881.239739870] [TheoraPublisher]: cv_bridge exception: '[16UC1] is not a color format. but [bgr8] is. The conversion does not make sense'
[component_container-2] [ERROR] [1685006881.268619620] [compressed_depth_image_transport]: Compressed Depth Image Transport - Compression requires single-channel 32bit-floating point or 16bit raw depth images (input format is: bgr8).
[component_container-2] [ERROR] [1685006881.486592614] [CompressedPublisher]: [16UC1] is not a color format. but [mono8] is. The conversion does not make sense
[component_container-2] [ERROR] [1685006881.487244929] [TheoraPublisher]: cv_bridge exception: '[16UC1] is not a color format. but [bgr8] is. The conversion does not make sense'
[component_container-2] [ERROR] [1685006881.525559979] [compressed_depth_image_transport]: Compressed Depth Image Transport - Compression requires single-channel 32bit-floating point or 16bit raw depth images (input format is: bgr8).
[component_container-2] [ERROR] [1685006881.556077713] [compressed_depth_image_transport]: Compressed Depth Image Transport - Compression requires single-channel 32bit-floating point or 16bit raw depth images (input format is: bgr8).
[component_container-2] [ERROR] [1685006881.730482224] [CompressedPublisher]: [16UC1] is not a color format. but [mono8] is. The conversion does not make sense
[component_container-2] [ERROR] [1685006881.731155104] [TheoraPublisher]: cv_bridge exception: '[16UC1] is not a color format. but [bgr8] is. The conversion does not make sense'
[component_container-2] [ERROR] [1685006881.757655439] [compressed_depth_image_transport]: Compressed Depth Image Transport - Compression requires single-channel 32bit-floating point or 16bit raw depth images (input format is: bgr8).
[component_container-2] [ERROR] [1685006881.976827562] [CompressedPublisher]: [16UC1] is not a color format. but [mono8] is. The conversion does not make sense
[component_container-2] [ERROR] [1685006881.977574131] [TheoraPublisher]: cv_bridge exception: '[16UC1] is not a color format. but [bgr8] is. The conversion does not make sense'
[component_container-2] [ERROR] [1685006882.003955160] [compressed_depth_image_transport]: Compressed Depth Image Transport - Compression requires single-channel 32bit-floating point or 16bit raw depth images (input format is: bgr8).
[component_container-2] [ERROR] [1685006882.202520321] [CompressedPublisher]: [16UC1] is not a color format. but [mono8] is. The conversion does not make sense
[component_container-2] [ERROR] [1685006882.203128605] [TheoraPublisher]: cv_bridge exception: '[16UC1] is not a color format. but [bgr8] is. The conversion does not make sense'
[component_container-2] [ERROR] [1685006882.420382652] [CompressedPublisher]: [16UC1] is not a color format. but [mono8] is. The conversion does not make sense
[component_container-2] [ERROR] [1685006882.421069877] [TheoraPublisher]: cv_bridge exception: '[16UC1] is not a color format. but [bgr8] is. The conversion does not make sense'
[component_container-2] [ERROR] [1685006882.627316469] [CompressedPublisher]: [16UC1] is not a color format. but [mono8] is. The conversion does not make sense
[component_container-2] [ERROR] [1685006882.627942541] [TheoraPublisher]: cv_bridge exception: '[16UC1] is not a color format. but [bgr8] is. The conversion does not make sense'
[component_container-2] [ERROR] [1685006882.657103986] [compressed_depth_image_transport]: Compressed Depth Image Transport - Compression requires single-channel 32bit-floating point or 16bit raw depth images (input format is: bgr8).
[ERROR] [component_container-2]: process has died [pid 548, exit code -11, cmd '/opt/ros/humble/lib/rclcpp_components/component_container --ros-args --log-level info --ros-args -r __node:=oak_container -r __ns:=/'].
^C[WARNING] [launch]: user interrupted with ctrl-c (SIGINT)
[robot_state_publisher-1] [INFO] [1685006890.096533897] [rclcpp]: signal_handler(signum=2)
[INFO] [robot_state_publisher-1]: process has finished cleanly [pid 546]

Serafadam commented 1 year ago

Hi, thanks for the info, we're currently investigating this, though some initial thoughts:

If camera info was skewed by that much, there would be some problems with image rectification/pointcloud generation. Do you also experience those? If not, I guess this might be something on foxglove side.
Those warnings come from subscribing to compressed depth image topic, compressed info is not available with 16UC1 format (ROS image_transport publishes that automatically though)

FabienDanieau commented 1 year ago

Thanks. This is what I have with depthai-experiments/gen2-pointcloud/rgbd-pointcloud/main.py Point cloud looks ok.

Now, if I just the ROS command ros2 launch depthai_ros_driver rgbd_pcl.launch.py

$ ros2 topic list
/joint_states
/oak/imu/data
/oak/points
/oak/rgb/camera_info
/oak/rgb/image_raw
/oak/rgb/image_raw/compressed
/oak/rgb/image_raw/compressedDepth
/oak/rgb/image_raw/theora
/oak/stereo/camera_info
/oak/stereo/image_raw
/oak/stereo/image_raw/compressed
/oak/stereo/image_raw/compressedDepth
/oak/stereo/image_raw/theora
/parameter_events
/robot_description
/rosout
/tf
/tf_static
$ ros2 topic info /oak/points 
Type: sensor_msgs/msg/PointCloud2
Publisher count: 1
Subscription count: 0
$ ros2 topic echo /oak/points

I never see any pcl published. This happens in docker and with a regular ROS2 machine.

FabienDanieau commented 1 year ago

the example ros2 launch depthai_examples stereo_inertial_node.launch.py displays a point cloud but it doesn't look so good

saching13 commented 1 year ago

Will take a look at it and get back to you.

saching13 commented 1 year ago

foxglove visualization issue is because we are using rational polynomial calibration model which is not supported by foxglove yet. Most likely will come in next release.

And as of your rgbd_pcl try enabling the IR

FabienDanieau commented 1 year ago

Ok thanks I thought it was according to their doc

Serafadam commented 1 year ago

@FabienDanieau also, on the depth publishing/quality, main branch should be fixed, there was a minor typo with the topic names. Regarding stereo_inertial_node, it has dot projector/floodlight disabled by default, you can enable it following way: ros2 launch depthai_examples stereo_inertial_node.launch.py enableDotProjector:=true dotProjectormA:=800.0

saching13 commented 1 year ago

It is not. I discussed this with them and looked at the code too. I will send a PR to them in a week or so to include that.

saching13 commented 11 months ago

Closing this as it is fixed in foxglove here.

luxonis / depthai-ros

[BUG] Camera field of view incorrect in foxglove #319