High-speed capture mode of Intel® RealSense™ Depth Camera D435

BertieBcs commented 1 month ago

Before opening a new issue, we wanted to provide you with some useful suggestions (Click "Preview" above for a better view):
- Consider checking out SDK examples.
- Have you looked in our documentations?
- Is you question a frequently asked one?
- Try searching our GitHub Issues (open and closed) for a similar issue.
All users are welcomed to report bugs, ask questions, suggest or request enhancements and generally feel free to open new issue, even if they haven't followed any of the suggestions above :)

Required Info	set high-speed capture mode for IR camera
Camera Model	{ D435 }
Firmware Version	5.16.0.1
Operating System & Version	Win 10
Platform	PC
SDK Version	{ v2.56.1 }
Language	Python

Issue Description

Hello, I would like to use high-speed capture mode by using the 300fps monochrome image stream via the IR cameras; as described in the 2020 white-paper: https://dev.intelrealsense.com/docs/high-speed-capture-mode-of-intel-realsense-depth-camera-d435

the following way: pipeline_profile = cfg.resolve(pipe) device = pipeline_profile.get_device()

for sensor in device.sensors: if sensor.name == 'Stereo Module': for profile in sensor.profiles: if profile.stream_type() == rs.stream.infrared: cfg.enable_stream(profile.stream_type(), 1, 848, 100, rs.format.y8, 300)

pipe.start(cfg)

I somehow cannot get above 60 FPS. I am using Python 3.11 and OpenCV

Thanks for any suggestions!

best regards,

Bertie

MartyG-RealSense commented 1 month ago

Hi @BertieBcs 848x100 High Speed Capture mode is supported on D435 for the depth and infrared streams, so you should be able to access the infrared stream at 300 FPS with Python code.

Do you get a Couldn't resolve requests error if you try to use an FPS higher than 60, or does the stream just seem much slower than it should be

Is 300 FPS provided if the simple test script below is used?

import pyrealsense2 as rs

pipeline = rs.pipeline()
cfg = rs.config()
cfg.enable_stream(rs.stream.infrared, 848, 100, rs.format.y8, 300)

pipe.start(cfg)

BertieBcs commented 1 month ago

Dear MartyG,

no, I don't get a Couldn't resolve requests error.

The test code you provided, as well as this one below

import pyrealsense2 as rs import numpy as np import cv2

pipe = rs.pipeline() cfg = rs.config() cfg.enable_stream(rs.stream.infrared, 848, 100, rs.format.y8, 300) pipe.start(cfg)

while True: frame = pipe.wait_for_frames() ir_frame = frame.get_infrared_frame() ir_image = np.asanyarray(ir_frame.get_data()) cv2.imshow('ir',ir_image)

if cv2.waitKey(1) == ord('q'):
    break

pipe.stop() cv2.destroyAllWindows()

works properly and displays a 848*100 stream (just not with 300 FPS).

Best regards,

B

On Thu, 3 Oct 2024 at 12:24, MartyG-RealSense @.***> wrote:

Hi @BertieBcs https://github.com/BertieBcs 848x100 High Speed Capture mode is supported on D435 for the depth and infrared streams, so you should be able to access the infrared stream at 300 FPS with Python code.

Do you get a Couldn't resolve requests error if you try to use an FPS higher than 60?

Is 300 FPS provided if the simple test script below is used?

import pyrealsense2 as rs

pipeline = rs.pipeline() cfg = rs.config() cfg.enable_stream(rs.stream.infrared, 848, 100, rs.format.y8, 300)

pipe.start(cfg)

— Reply to this email directly, view it on GitHub https://github.com/IntelRealSense/librealsense/issues/13395#issuecomment-2391053852, or unsubscribe https://github.com/notifications/unsubscribe-auth/BLZR5WLR6F3I3XBFH6PG62LZZULMTAVCNFSM6AAAAABPJOECGOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGOJRGA2TGOBVGI . You are receiving this because you were mentioned.Message ID: @.***>

MartyG-RealSense commented 1 month ago

How do you confirm what the FPS of the infrared stream provided by the Python script is, please?

Do you have access to the RealSense Viewer tool? If you do, what is the displayed FPS for an 848x100 infrared stream set at 300 FPS?

This is from my own D435:

BertieBcs commented 1 month ago

The hardware FPS is indeed 300 and the viewer FPS is around 60 - but, because of my display specs, which don't allow me to set higher refresh rate then 60 Hz, I guess. (Driver is uptodate)

Best regards,

B

On Sat, 5 Oct 2024 at 11:47, MartyG-RealSense @.***> wrote:

How do you confirm what the FPS of the infrared stream provided by the Python script is, please?

Do you have access to the RealSense Viewer tool? If you do, what is the displayed FPS for an 848x100 infrared stream set at 300 FPS?

This is from my own D435:

image.png (view on web) https://github.com/user-attachments/assets/f47189b6-fb46-4708-8a5c-2fd8f6188dfb

image.png (view on web) https://github.com/user-attachments/assets/6a89ae07-1b18-4c05-a49c-66ef18ec57b6

— Reply to this email directly, view it on GitHub https://github.com/IntelRealSense/librealsense/issues/13395#issuecomment-2394999950, or unsubscribe https://github.com/notifications/unsubscribe-auth/BLZR5WPXJOTFLV5TYB5QGN3ZZ6YSBAVCNFSM6AAAAABPJOECGOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGOJUHE4TSOJVGA . You are receiving this because you were mentioned.Message ID: @.***>

MartyG-RealSense commented 1 month ago

The 'Viewer FPS' value can be disregarded. The FPS value that matters is the one displayed as an overlay at the top of the stream panel when the 'i' icon on the stream panel's toolbar is enabled, like in my image above.

BertieBcs commented 1 month ago

It appears that the camera stream is fed in with that low frame frate, at least if I open it with Python:

https://github.com/BertieBcs/RealSense-D435.git

MartyG-RealSense commented 1 month ago

There was a past Python case at https://github.com/IntelRealSense/librealsense/issues/6578#issuecomment-643811438 where the user was using 848x100 at 300 FPS and was not getting a Couldn't resolve requests error but felt that they were not receiving 300 FPS consistently.

You could check the real FPS with Python code by retrieving the value of a camera metadata parameter called Actual FPS. The Python code for retrieving and printing the value would be something like this:

fps = rs.frame.get_frame_metadata(frame, rs.frame_metadata_value.actual_fps)  
print("FPS : ", fps)

BertieBcs commented 1 month ago

I am getting this error message:

RuntimeError: Infrared frame does not support metadata "Actual Fps"

if only the IR camera is on.

If the color and/or the depth stream is enabled, than it prints the actual FPS.

Regards,

B

On Sun, 6 Oct 2024 at 10:24, MartyG-RealSense @.***> wrote:

There was a past Python case at #6578 (comment) https://github.com/IntelRealSense/librealsense/issues/6578#issuecomment-643811438 where the user was using 848x100 at 300 FPS and was not getting a Couldn't resolve requests error but felt that they were not receiving 300 FPS consistently.

You could check the real FPS with Python code by retrieving the value of a camera metadata parameter called Actual FPS. The Python code for retrieving and printing the value would be something like this:

fps = rs.frame.get_frame_metadata(frame, rs.frame_metadata_value.actual_fps) print("FPS : ", fps)

— Reply to this email directly, view it on GitHub https://github.com/IntelRealSense/librealsense/issues/13395#issuecomment-2395346124, or unsubscribe https://github.com/notifications/unsubscribe-auth/BLZR5WN3V7JB3WPXT5IJGKTZ2DXURAVCNFSM6AAAAABPJOECGOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGOJVGM2DMMJSGQ . You are receiving this because you were mentioned.Message ID: @.***>

MartyG-RealSense commented 1 month ago

Assuming that if there was a problem with infrared FPS then it would also affect the depth FPS in the same way, what Actual FPS value is provided if depth is enabled?

BertieBcs commented 1 month ago

Hi, it appears that Actual FPS is calculated only when two cameras are on, in any combination. If depth+ir are enabled, it prints roughly 300 000; as well as if color + IR are on. As the color camera does not support high frame rates, it means that my IR can indeed capture 300 fps (pls. correct me if I am wrong). Thanks for your support, maybe if you have time, can you please tell me why two cameras are needed to get the actual FPS?

Best regards,

Bertalan

On Mon, 7 Oct 2024 at 10:50, MartyG-RealSense @.***> wrote:

Assuming that if there was a problem with infrared FPS then it would also affect the depth FPS in the same way, what Actual FPS value is provided if depth is enabled?

— Reply to this email directly, view it on GitHub https://github.com/IntelRealSense/librealsense/issues/13395#issuecomment-2396308360, or unsubscribe https://github.com/notifications/unsubscribe-auth/BLZR5WOOMGGXGTZAHMLF2NLZ2JDMLAVCNFSM6AAAAABPJOECGOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGOJWGMYDQMZWGA . You are receiving this because you were mentioned.Message ID: @.***>

MartyG-RealSense commented 1 month ago

I generated the 'frame metadata' overlay in the RealSense Viewer with only Infrared enabled and the Actual FPS was updating in real-time and running at around 300.

Because 848x100 mode in the Viewer produces a stream panel of a short height, a scroll-bar at the side of the stream panel has to be used to scroll the metadata details overlay down to find the Actual FPS value.

BertieBcs commented 1 month ago

Now it seems to have another issue: there is no IR stream at all. Python throws an error: pipe.start(cfg) RuntimeError: Couldn't resolve requests and in the viewer the option infrared stream does not even appear. Is the camera gone or just some settings problem?

Regards,

B

On Tue, 8 Oct 2024 at 15:31, MartyG-RealSense @.***> wrote:

I generated the 'frame metadata' overlay in the RealSense Viewer with only Infrared enabled and the Actual FPS was updating in real-time and running at around 300.

Because 848x100 mode in the Viewer produces a stream panel of a short height, a scroll-bar at the side of the stream panel has to be used to scroll the metadata details overlay down to find the Actual FPS value.

image.png (view on web) https://github.com/user-attachments/assets/931cbbba-904e-4be3-8417-7357253bb30e

— Reply to this email directly, view it on GitHub https://github.com/IntelRealSense/librealsense/issues/13395#issuecomment-2399858277, or unsubscribe https://github.com/notifications/unsubscribe-auth/BLZR5WNKOQMMP4I7MPAT44LZ2PNDBAVCNFSM6AAAAABPJOECGOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGOJZHA2TQMRXG4 . You are receiving this because you were mentioned.Message ID: @.***>

MartyG-RealSense commented 1 month ago

If you are on Windows then there is a rare problem where the infrared streams can disappear, and in the Stereo Module of the RealSense Viewer only the depth stream is listed.

The problem is usually related to an issue with the RealSense depth and RGB drivers in the Windows Device Manager. It can be resolved by using a procedure at the link below to uninstall these drivers completely, leaving no trace of them on the computer, and then reinstalling them.

https://support.intelrealsense.com/hc/en-us/community/posts/4419989666323/comments/4431239847443

BertieBcs commented 1 month ago

Hello and thanks for the support. I can confirm that my camera works fine now.

Best regards,

B

On Wed, 9 Oct 2024 at 14:29, MartyG-RealSense @.***> wrote:

If you are on Windows then there is a rare problem where the infrared streams can disappear, and in the Stereo Module of the RealSense Viewer only the depth stream is listed.

The problem is usually related to an issue with the RealSense depth and RGB drivers in the Windows Device Manager. It can be resolved by using a procedure at the link below to uninstall these drivers completely, leaving no trace of them on the computer, and then reinstalling them.

https://support.intelrealsense.com/hc/en-us/community/posts/4419989666323/comments/4431239847443

— Reply to this email directly, view it on GitHub https://github.com/IntelRealSense/librealsense/issues/13395#issuecomment-2402187159, or unsubscribe https://github.com/notifications/unsubscribe-auth/BLZR5WLD22SXUVHGFZQEXFDZ2UOQ7AVCNFSM6AAAAABPJOECGOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMBSGE4DOMJVHE . You are receiving this because you were mentioned.Message ID: @.***>

MartyG-RealSense commented 1 month ago

You are very welcome. I'm pleased to hear that your camera is working now. Thanks very much for the update!

MartyG-RealSense commented 1 month ago

Hi @BertieBcs Do you require further assistance with this case, please? Thanks!

BertieBcs commented 1 month ago

No, I am fine. Thank you again!

BB

On Sun, 20 Oct 2024 at 12:58, MartyG-RealSense @.***> wrote:

Hi @BertieBcs https://github.com/BertieBcs Do you require further assistance with this case, please? Thanks!

— Reply to this email directly, view it on GitHub https://github.com/IntelRealSense/librealsense/issues/13395#issuecomment-2424841815, or unsubscribe https://github.com/notifications/unsubscribe-auth/BLZR5WKBIGLI322B6B6NE7TZ4OEF3AVCNFSM6AAAAABPJOECGOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMRUHA2DCOBRGU . You are receiving this because you were mentioned.Message ID: @.***>

MartyG-RealSense commented 1 month ago

You are very welcome. As you do not require further assistance, I will close this case. Thanks again!

IntelRealSense / librealsense

High-speed capture mode of Intel® RealSense™ Depth Camera D435 #13395

Issue Description