RuntimeError: Frame did not arrive within 5000

[vinceclifford]

Required Info
Camera Model	D455
Firmware Version	5.14.0
Operating System & Version	Linux (Ubuntu 22.04)
Kernel Version (Linux Only)	5.15.0-112-generic
Platform	Emulating Ubuntu 22.04 through VM 'UTM' on MacBook Pro
SDK Version	2.55.1
Language	python
Segment	Robot

Issue Description

My issue is streaming live video from the described camera to my MacBook Pro (Chip: M3 Pro). I want to run a python script that gets the camera data and performs some computer vision algorithms.

Compilation on Mac OS Sonoma

Initially, I tried downloading the SDK and compiling the SDK directly on my Mac with MacOS Sonoma. However, this does work straight out of the box. I followed the instructions here: https://dev.intelrealsense.com/docs/macos-installation-for-intel-realsense-sdk.The CMake fails to download nlohmann/json. Running the CMake results in the following output

CMake Error at CMake/external_json.cmake:40 (message):
  Failed to download nlohmann/json
Call Stack (most recent call first):
  CMake/external_json.cmake:56 (get_nlohmann_json)
  third-party/CMakeLists.txt:3 (include)
  CMakeLists.txt:61 (include)

Trying to further understand what caused this download error, I added some debugging information in the CMake to figure out why the download does not work anymore. The error code given is 65 and the more elaborate error is

error: accessing build database "[path_to_SDK]/RealSense_SDK/librealsense/build/external-projects/json-download/build/XCBuildData/build.db": disk I/O error

Since the SDK is not supported officially on MacOS, I thought this was not a big problem; I'll just start a VM running Linux

Running on Ubuntu 22.04 using VM UTM

Next, I downloaded a VM, installed Linux and Ubuntu, and compiled the IntelRealsense SDK. It worked.

I can use the realsence-viewer, and it works. I get the camera and the depth stream. However, here's where I run into major issues that I have been trying to resolve for the past two days:

If I try to receive the color frames from my Python script, it does not work. I do not receive any input. I'm running the following code:

import pyrealsense2 as rs
import numpy as np
import cv2

pipe = rs.pipeline()
config = rs.config()

config.enable_stream(rs.stream.color, 640, 480, rs.format.bgr8, 30)
config.enable_stream(rs.stream.depth, 640, 480, rs.format.z16, 30)

pipe.start(config)

while True:
    frame = pipe.wait_for_frames()
    depth_frame = frame.get_depth_frame()
    color_frame = frame.get_color_frame()

    depth_image = np.asanyarray(depth_frame.get_data())
    color_image = np.asanyarray(color_frame.get_data())

    cv2.imshow('Color Image', color_image)
    cv2.imshow('Depth Image', depth_image)

    if cv2.waitKey(1) & 0xFF == ord('q'):
        break

pipe.stop()

Running this code results in an error in line 14, frame = pipe.wait_for_frames(). The error is a RuntimeError: Frame did not arrive within 5000.

The interesting part is that if I try not to receive the camera stream and only the depth stream, and run the following code:

import pyrealsense2 as rs
import numpy as np
import cv2

pipe = rs.pipeline()
config = rs.config()

config.enable_stream(rs.stream.depth, 640, 480, rs.format.z16, 30)

pipe.start(config)

while True:
    frame = pipe.wait_for_frames()
    depth_frame = frame.get_depth_frame()

    depth_image = np.asanyarray(depth_frame.get_data())

    cv2.imshow('Depth Image', depth_image)

    if cv2.waitKey(1) & 0xFF == ord('q'):
        break

pipe.stop()

I get the depth stream of the camera! After playing around with the code, I realized I could get all other streams, e.g., the infrared stream. I only receive this issue if I try to collect the camera stream.

Already taken measures

I have already tried multiple approaches to resolve this issue.

First, I tried to update the camera's firmware to the newest version 5.16.0.1. This does not work. If I try to update the camera version from within Intel Realsense Viewer, I get a segmentation error. Additionally, I tried to update the firmware explicitly according to this documentation: https://dev.intelrealsense.com/docs/firmware-update-tool. I get an error:

Warning: the camera is connected via USB 2 port, in case the proccess fails, connect the camera to USB 3 Port and try again 
std::bad_alloc

I tried to connect to my USB 3 ports, but this does not work. For some reason, the camera is always registered via a USB 2.1 port. I tried 3 different cables, with and without hubs, and it simply makes no difference.

At this point, I have no clue why I get a timeout and can't get any color frames. What is the difference between the color-stream and the depth-stream/infrared-stream. Am I limited by the bandwidth of the cable if I connect it via USB 2.1? It shouldn't be, right? Else the Intel RealSense Viewer would have trouble as well displaying the color image.

I would be very thankful if you could provide any leads or tips. Thanks!

[MartyG-RealSense]

Hi @vinceclifford The Mac installation instructions that you linked to are for old Intel-based Macs and will usually not work on a modern Apple Silicon 'M' model of Mac.

The last known Apple Silicon guide that has been able to work is one for Aple Silicon and MacOS Monterey at the link below.

https://lightbuzz.com/realsense-macos/

This guide has since had problems with more recent MacOS versions such as Ventura and Sonoma though. A RealSense team member has been looking at installation problems on the latest MacOS versions to see whether a new guide can be developed but there is not information to report about this effort at the time of writing this.

Installation with brew is also supported, though RealSense Mac users have experienced problems with this approach too.

https://formulae.brew.sh/formula/librealsense

Regarding the camera being detected as USB 2.1 in a USB3 port, please test whether reversing the direction that the micro-sized connector in the base of the D455 camera is inserted in makes a difference (USB-C cables are two-way insertion at the micr-sized end).

A USB2 connection's speed is 480 Mbps (megabits per second) and USB3 is 5000 Mbps. So there is a significant difference in terms of available bandwidth. A USB2 cable should be able to handle depth, infrared and RGB simultaneously though if the resolution is only 640x480.

If you are unable to receive the RGB color stream but can receive infrared, you could try streaming RGB from the infrared sensor instead, a mode which the D455 camera model supports. In the amended script below, the config line for the color stream is changed to the infrared stream and the parameter 'color_frame' is set to retrieve the infrared frame (which is streaming in RGB color format instead of Y8 infrared).

import pyrealsense2 as rs
import numpy as np
import cv2

pipe = rs.pipeline()
config = rs.config()

config.enable_stream(rs.stream.infrared, 640, 480, rs.format.bgr8, 30)
config.enable_stream(rs.stream.depth, 640, 480, rs.format.z16, 30)

pipe.start(config)

while True:
    frame = pipe.wait_for_frames()
    depth_frame = frame.get_depth_frame()
    color_frame = frame.get_infrared_frame()

    depth_image = np.asanyarray(depth_frame.get_data())
    color_image = np.asanyarray(color_frame.get_data())

    cv2.imshow('Color Image', color_image)
    cv2.imshow('Depth Image', depth_image)

    if cv2.waitKey(1) & 0xFF == ord('q'):
        break

pipe.stop()

[vinceclifford]

Thank you very much for your help. I was already aware of the guide for installing RealSense Viewer on Silicon Chips. However, I had no success with it or with brew.

Unfortunately, reversing the microsized connector does not make a difference.

I just want to stream the camera feed. Infrared and depth streaming are not crucial at the moment. So, USB 2.0 should be fine. So, this problem can be neglected right now.

Are there any differences between streaming from the actual RBG stream and the RGB stream from the infrared sensor? When I run your script, the infrared RGB stream looks different from the "normal" one in the RealSense Viewer. Here are the streams compared. The proposed script:

RealSense Viewer:

I've got a feeling that the depth camera is not recognized correctly. If I run the command usb-devices in the terminal of the VM, only the depth camera is listed. Neither the infrared nor the RGB camera. Is this intended? Are there any known issues?

[MartyG-RealSense]

Yes, RGB from the infrared sensor is colored differently from the RGB sensor image. The infrared sensor version of RGB can be a useful substitute though in situations where there is a problem with delivering frames from the RGB sensor.

It is difficult to conclude whether your problems with accessing the camera's RGB on an Ubuntu VM are due to the particular brand of VM that you are using or another factor. In your script you could try setting the color stream's format to RGB8 instead of BGR8 in its config line to see whether it makes a difference.

A RealSense Mac user who also has a MacBook Pro found that using Asahi Linux instead of Ubuntu worked well for them. According to that user, it is so well designed that it does not require a VM and runs on the Mac's bare hardware. They created an installation guide for it.

https://github.com/HasanTheSyrian/librealsense-hasan/blob/master/doc/installation_asahi.md

[vinceclifford]

Unfortunately, changing the color stream's format does not change the behavior.

Thanks for pointing me to Asahi Linux. Since my MacBook is relatively new, my Chip (M3 Pro) is incompatible. For that option to work, I would have to wait until Asahi Linux is released for M3 Pro.

I'll try some other workarounds over the weekend.

[MartyG-RealSense]

Thank you for the information. I look forward to your next report after trying workarounds. Good luck!

[vinceclifford]

Short update: I acquired a different cable, and the camera now uses a USB 3.2. connection. Which is an initial success. However, I'm still not able to stream the camera-fee. Because I'm now communicating via a 3.2, connection, I could update the camera's firmware to 5.16.0.1. However, when running the command

rs-fw-update -s 214623065773 -f  Signed_Image_UVC_5_16_0_1

I get the following error message:

Search for device with serial number: 214623065773
std::bad_alloc

The serial number is correct; I double-checked and triple-checked. Is this a connecting issue to the RuntimeError: Frame did not arrive within 5000? Trying a different VM would be my last resort.

[MartyG-RealSense]

How long is the USB cable that you acquired, please? It is recommendable to use the official 1 meter cable supplied with the camera for firmware updates if possible.

Are you able to use the 5.16.0.1 bin file to update to that version in the RealSense Viewer? You can do this by going to the 'More' option near the top of the Viewer's options side-panel, selecting 'Update Firmware' from its drop-down menu and then using the file navigation box that pops up to select the bin file on your computer and start the firmware update.