Running three D430 on Coral Dev Board

brunovollmer commented 3 years ago

Note: I've created the same Issue on the librealsense repo: intelrealsense/librealsense#9443

Hey everybody,

I'm not sure if this is the right place but I thought maybe somebody had the same problem or some idea. I'm currently trying to connect three D430 + D4 vision boards to a coral dev board through one USB-C hub. Unfortunately I can't make it work as I either receive a segmentation fault (most of the times), a "Frame did not arrive within 5000" (sometimes) or it can't even find all cameras (disconnect solves that).

I've created a little test script to check if the cameras work:

import pyrealsense2 as rs
import numpy as np
import argparse
import cv2
import time

def parse_inputs():
    parser = argparse.ArgumentParser()
    parser.add_argument('--width',
                        default=848,
                        type=int,
                        help='width of stream')
    parser.add_argument('--height',
                        default=480,
                        type=int,
                        help='height of stream')
    parser.add_argument('--frame_rate',
                        default=30,
                        type=int,
                        help='frame rate of stream')
    parser.add_argument('--frames',
                        default=100,
                        type=int,
                        help='number of frames for the streaming test')

    return parser.parse_args()

def find_cameras(ctx):
    cameras = []
    devices = ctx.devices

    for dev in devices:
        name = dev.get_info(rs.camera_info.name)
        serial_number = dev.get_info(rs.camera_info.serial_number)

        cameras.append({'name': name, 'serial_number': serial_number})

    return cameras

def init_cameras(cameras, width, height, frame_rate):
    pipelines = {}

    for cam in cameras:
        serial_number = cam['serial_number']
        name = cam['name']

        pipe = rs.pipeline()
        cfg = rs.config()
        cfg.enable_device(serial_number)

        cfg.enable_stream(rs.stream.infrared,
                          width=width,
                          height=height,
                          format=rs.format.y8,
                          framerate=frame_rate)

        cfg.enable_stream(rs.stream.depth,
                          width=width,
                          height=height,
                          format=rs.format.z16,
                          framerate=frame_rate)

        profile = pipe.start(cfg)
        depth_scale = (
                profile.get_device().first_depth_sensor().get_depth_scale())

        cam['depth_scale'] = depth_scale

        pipelines[serial_number] = {'pipe': pipe, 'camera': cam}

    return pipelines

def test_stream(pipelines, nr_frames):

    colorizer = rs.colorizer()

    durations = []

    for i in range(nr_frames):            

        start_time = time.time()

        for serial_number, camera_data in pipelines.items():
            name = camera_data['camera']['name']
            pipe = camera_data['pipe']
            depth_scale = camera_data['camera']['depth_scale']

            frames = pipe.wait_for_frames()

            color_frame = frames.get_infrared_frame()

            depth_frame = frames.get_depth_frame()

            color = np.asanyarray(color_frame.get_data())
            color = cv2.cvtColor(color, cv2.COLOR_GRAY2RGB)

            depth_frame_color = np.asanyarray(
                colorizer.colorize(depth_frame).get_data())

            depth_frame = np.asanyarray(depth_frame.get_data())
            depth_frame = depth_frame * depth_scale

        end_time = time.time()

        durations.append(end_time - start_time)

    return 1/(sum(durations)/len(durations))

def main():
    args = parse_inputs()

    context = rs.context()

    cameras = find_cameras(context)
    print("Starting Intel Camera Test")
    print("-----------------------------------------------")

    print("Input Args")
    print(args)

    print("-----------------------------------------------")

    print(f"found {len(cameras)} camera(s)")
    print(f"camera details: {cameras}")

    pipelines = init_cameras(cameras, args.width, args.height, args.frame_rate)

    print(f"initialized {len(pipelines.keys())} camera(s)")

    print("-----------------------------------------------")

    print(f"Starting streaming test for {args.frames} frames")
    fps = test_stream(pipelines, args.frames)
    print("Finished streaming test")
    print(f"Average streaming FPS: {fps}")

if __name__ == "__main__":
    main()

The things that I've check so far:

Check USB-C hug: The USB-C Hub can handle the bandwidth and power as I can run the same script from my laptop without a problem.
Reduce resolution: I've tried to lower the resolution to 640x360 but that does not lead to any improvements unfortunately.
Lower frame-rate: Lowered it to 6 FPS but similar to reduced resolution.

I'm a bit lost now as I don't really know what to do to make it work. Unfortunately we have to run all cameras through one USB-C 3.0 port. Any ideas on what to try?

manoj7410 commented 3 years ago

Can you share the output of :

v4l2-ctl --list-devices

v4l2-ctl --list-formats-ext --device /dev/video1

brunovollmer commented 3 years ago

(env3.8) mendel@zippy-dog:~/soundmap$ v4l2-ctl --list-devices
i.MX6S_CSI (platform:30a90000.csi1_bridge):
    /dev/video0

Intel(R) RealSense(TM) Depth Ca (usb-xhci-hcd.0.auto-1.1):
    /dev/video1
    /dev/video2

Intel(R) RealSense(TM) Depth Ca (usb-xhci-hcd.0.auto-1.2):
    /dev/video3
    /dev/video4

Intel(R) RealSense(TM) Depth Ca (usb-xhci-hcd.0.auto-1.4):
    /dev/video5
    /dev/video6

(env3.8) mendel@zippy-dog:~/soundmap$ v4l2-ctl --list-formats-ext --device /dev/video1
ioctl: VIDIOC_ENUM_FMT
    Type: Video Capture

    [0]: 'Z16 ' (16-bit Depth)
        Size: Discrete 256x144
            Interval: Discrete 0.003s (300.000 fps)
            Interval: Discrete 0.011s (90.000 fps)
        Size: Discrete 424x240
            Interval: Discrete 0.011s (90.000 fps)
            Interval: Discrete 0.017s (60.000 fps)
            Interval: Discrete 0.033s (30.000 fps)
            Interval: Discrete 0.067s (15.000 fps)
            Interval: Discrete 0.167s (6.000 fps)
        Size: Discrete 480x270
            Interval: Discrete 0.011s (90.000 fps)
            Interval: Discrete 0.017s (60.000 fps)
            Interval: Discrete 0.033s (30.000 fps)
            Interval: Discrete 0.067s (15.000 fps)
            Interval: Discrete 0.167s (6.000 fps)
        Size: Discrete 640x360
            Interval: Discrete 0.011s (90.000 fps)
            Interval: Discrete 0.017s (60.000 fps)
            Interval: Discrete 0.033s (30.000 fps)
            Interval: Discrete 0.067s (15.000 fps)
            Interval: Discrete 0.167s (6.000 fps)
        Size: Discrete 640x480
            Interval: Discrete 0.011s (90.000 fps)
            Interval: Discrete 0.017s (60.000 fps)
            Interval: Discrete 0.033s (30.000 fps)
            Interval: Discrete 0.067s (15.000 fps)
            Interval: Discrete 0.167s (6.000 fps)
        Size: Discrete 848x100
            Interval: Discrete 0.003s (300.000 fps)
            Interval: Discrete 0.010s (100.000 fps)
        Size: Discrete 848x480
            Interval: Discrete 0.011s (90.000 fps)
            Interval: Discrete 0.017s (60.000 fps)
            Interval: Discrete 0.033s (30.000 fps)
            Interval: Discrete 0.067s (15.000 fps)
            Interval: Discrete 0.167s (6.000 fps)
        Size: Discrete 1280x720
            Interval: Discrete 0.033s (30.000 fps)
            Interval: Discrete 0.067s (15.000 fps)
            Interval: Discrete 0.167s (6.000 fps)
    [1]: '' (00000050-0000-0010-8000-00aa003)
(env3.8) mendel@zippy-dog:~/soundmap$

brunovollmer commented 3 years ago

Another thing I realized is that only connecting three but not using all of them leads to problem. So if I connect all three but only initialize two it won't work. I get more and more the feeling the problem is not related to the actual bandwidth but some USB connectivity issue of the coral dev board.

brunovollmer commented 3 years ago

@manoj7410 Do you think this problem could be related to the kernel and a newer version could fix it?

brunovollmer commented 3 years ago

Any ideas on this?

manoj7410 commented 3 years ago

I don't see 'YUYV' format supported in your camera. Can you please confirm if your camera supports 'YUYV' ?

brunovollmer commented 3 years ago

It seems to only support Y8 as a format for the infrared image and Z16 as format for the depth image. We are using D430s:

brunovollmer commented 3 years ago

If I compile librealsense without RUSB support it unfortunately does not work at all, as it only finds two of the three cameras and it can't initialize them. My question is if you have an idea of a possible patch to the native usb driver in the 4.14 kernel to make this work?

brunovollmer commented 3 years ago

I checked some more and it appears anyways even if the document says otherwise:

ioctl: VIDIOC_ENUM_FMT
    Type: Video Capture

    [0]: '' (00000032-0000-0010-8000-00aa003)
    [1]: 'UYVY' (UYVY 4:2:2)
        Size: Discrete 256x144
            Interval: Discrete 0.003s (300.000 fps)
            Interval: Discrete 0.011s (90.000 fps)
        Size: Discrete 424x240
            Interval: Discrete 0.011s (90.000 fps)
            Interval: Discrete 0.017s (60.000 fps)
            Interval: Discrete 0.033s (30.000 fps)
            Interval: Discrete 0.067s (15.000 fps)
            Interval: Discrete 0.167s (6.000 fps)
        Size: Discrete 480x270
            Interval: Discrete 0.011s (90.000 fps)
            Interval: Discrete 0.017s (60.000 fps)
            Interval: Discrete 0.033s (30.000 fps)
            Interval: Discrete 0.067s (15.000 fps)
            Interval: Discrete 0.167s (6.000 fps)
        Size: Discrete 640x360
            Interval: Discrete 0.011s (90.000 fps)
            Interval: Discrete 0.017s (60.000 fps)
            Interval: Discrete 0.033s (30.000 fps)
            Interval: Discrete 0.067s (15.000 fps)
            Interval: Discrete 0.167s (6.000 fps)
        Size: Discrete 640x480
            Interval: Discrete 0.011s (90.000 fps)
            Interval: Discrete 0.017s (60.000 fps)
            Interval: Discrete 0.033s (30.000 fps)
            Interval: Discrete 0.067s (15.000 fps)
            Interval: Discrete 0.167s (6.000 fps)
        Size: Discrete 848x100
            Interval: Discrete 0.003s (300.000 fps)
            Interval: Discrete 0.010s (100.000 fps)
        Size: Discrete 848x480
            Interval: Discrete 0.011s (90.000 fps)
            Interval: Discrete 0.017s (60.000 fps)
            Interval: Discrete 0.033s (30.000 fps)
            Interval: Discrete 0.067s (15.000 fps)
            Interval: Discrete 0.167s (6.000 fps)
        Size: Discrete 1280x720
            Interval: Discrete 0.033s (30.000 fps)
            Interval: Discrete 0.067s (15.000 fps)
            Interval: Discrete 0.167s (6.000 fps)
    [2]: 'GREY' (8-bit Greyscale)
        Size: Discrete 256x144
            Interval: Discrete 0.003s (300.000 fps)
            Interval: Discrete 0.011s (90.000 fps)
        Size: Discrete 424x240
            Interval: Discrete 0.011s (90.000 fps)
            Interval: Discrete 0.017s (60.000 fps)
            Interval: Discrete 0.033s (30.000 fps)
            Interval: Discrete 0.067s (15.000 fps)
            Interval: Discrete 0.167s (6.000 fps)
        Size: Discrete 480x270
            Interval: Discrete 0.011s (90.000 fps)
            Interval: Discrete 0.017s (60.000 fps)
            Interval: Discrete 0.033s (30.000 fps)
            Interval: Discrete 0.067s (15.000 fps)
            Interval: Discrete 0.167s (6.000 fps)
        Size: Discrete 640x360
            Interval: Discrete 0.011s (90.000 fps)
            Interval: Discrete 0.017s (60.000 fps)
            Interval: Discrete 0.033s (30.000 fps)
            Interval: Discrete 0.067s (15.000 fps)
            Interval: Discrete 0.167s (6.000 fps)
        Size: Discrete 640x480
            Interval: Discrete 0.011s (90.000 fps)
            Interval: Discrete 0.017s (60.000 fps)
            Interval: Discrete 0.033s (30.000 fps)
            Interval: Discrete 0.067s (15.000 fps)
            Interval: Discrete 0.167s (6.000 fps)
        Size: Discrete 848x100
            Interval: Discrete 0.003s (300.000 fps)
            Interval: Discrete 0.010s (100.000 fps)
        Size: Discrete 848x480
            Interval: Discrete 0.011s (90.000 fps)
            Interval: Discrete 0.017s (60.000 fps)
            Interval: Discrete 0.033s (30.000 fps)
            Interval: Discrete 0.067s (15.000 fps)
            Interval: Discrete 0.167s (6.000 fps)
        Size: Discrete 1280x720
            Interval: Discrete 0.033s (30.000 fps)
            Interval: Discrete 0.067s (15.000 fps)
            Interval: Discrete 0.167s (6.000 fps)
    [3]: 'Y8I ' (Interleaved 8-bit Greyscale)
        Size: Discrete 256x144
            Interval: Discrete 0.003s (300.000 fps)
            Interval: Discrete 0.011s (90.000 fps)
        Size: Discrete 424x240
            Interval: Discrete 0.011s (90.000 fps)
            Interval: Discrete 0.017s (60.000 fps)
            Interval: Discrete 0.033s (30.000 fps)
            Interval: Discrete 0.067s (15.000 fps)
            Interval: Discrete 0.167s (6.000 fps)
        Size: Discrete 480x270
            Interval: Discrete 0.011s (90.000 fps)
            Interval: Discrete 0.017s (60.000 fps)
            Interval: Discrete 0.033s (30.000 fps)
            Interval: Discrete 0.067s (15.000 fps)
            Interval: Discrete 0.167s (6.000 fps)
        Size: Discrete 640x360
            Interval: Discrete 0.011s (90.000 fps)
            Interval: Discrete 0.017s (60.000 fps)
            Interval: Discrete 0.033s (30.000 fps)
            Interval: Discrete 0.067s (15.000 fps)
            Interval: Discrete 0.167s (6.000 fps)
        Size: Discrete 640x480
            Interval: Discrete 0.011s (90.000 fps)
            Interval: Discrete 0.017s (60.000 fps)
            Interval: Discrete 0.033s (30.000 fps)
            Interval: Discrete 0.067s (15.000 fps)
            Interval: Discrete 0.167s (6.000 fps)
        Size: Discrete 848x100
            Interval: Discrete 0.003s (300.000 fps)
            Interval: Discrete 0.010s (100.000 fps)
        Size: Discrete 848x480
            Interval: Discrete 0.011s (90.000 fps)
            Interval: Discrete 0.017s (60.000 fps)
            Interval: Discrete 0.033s (30.000 fps)
            Interval: Discrete 0.067s (15.000 fps)
            Interval: Discrete 0.167s (6.000 fps)
        Size: Discrete 1280x720
            Interval: Discrete 0.033s (30.000 fps)
            Interval: Discrete 0.067s (15.000 fps)
            Interval: Discrete 0.167s (6.000 fps)
        Size: Discrete 1280x800
            Interval: Discrete 0.033s (30.000 fps)
            Interval: Discrete 0.067s (15.000 fps)
    [4]: 'Y12I' (Interleaved 12-bit Greyscale)
        Size: Discrete 640x400
            Interval: Discrete 0.040s (25.000 fps)
            Interval: Discrete 0.067s (15.000 fps)
        Size: Discrete 1280x800
            Interval: Discrete 0.040s (25.000 fps)
            Interval: Discrete 0.067s (15.000 fps)

manoj7410 commented 3 years ago

I see YUY2 which is similar to YUYV. See this note at https://coral.ai/docs/dev-board/camera/#connect-a-usb-camera : Be sure that your list includes Pixel Format: 'YUYV'. Currently, YUYV is the only format supported. But the commands below refer to this format with the name YUY2, which is just a different name for the same thing.

Did you try to connect a single camera instead of multiple cameras ?

brunovollmer commented 3 years ago

Single camera does not work if I don't enable RUSB support when I built the library.

manoj7410 commented 3 years ago

Based on the conversation at https://github.com/IntelRealSense/librealsense/issues/9443 marking this thread as closed for now. Feel free to reopen this issue whenever you need to.

google-coral / edgetpu

Running three D430 on Coral Dev Board #422