IntelRealSense / librealsense

Intel® RealSense™ SDK
https://www.intelrealsense.com/
Apache License 2.0
7.6k stars 4.83k forks source link

High variance in observed FPS in multi-camera setup #11321

Closed michaelcukier closed 1 year ago

michaelcukier commented 1 year ago
Required Info
Camera Model D405
Firmware Version latest
Operating System & Version Ubuntu (for Jetson)
Kernel Version (Linux Only) 5.10
Platform Jetson AGX Orin
SDK Version latest
Language python
Segment robotics

Issue Description

I'm running three D405 using pyrealsense2. I'm running a simple python code that turns on the three cameras and count how many frames were received, and I sometimes get 4 FPS on average for all three cameras, and some other times 29 FPS, even though the setup is exactly the same. I don't understand why.

I tried running my code for a very long time (1 hour) twice, compared the results, and still got very different observed FPS. I am using the exact same three USB cables. I've also noticed that, very often, one camera has a significantly superior FPS compared to the two others.

What could be the reason(s) for this massive variance in FPS between each tests? Why is the observed FPS not stable at all? I could understand a difference of 1 or 2 FPS during each test, but a difference of 25 is really strange.

Note: when trying two or one D405, this problem seems to go away and the FPS is always at nearly 30FPS.

MartyG-RealSense commented 1 year ago

Hi @michaelcukier As the number of RealSense cameras that are simultaneously active on the same computer is increased, the processing burden on the computer of handling those multiple cameras will increase, necessitating a higher-specification processor. For example, the Up Squared single-board computer has a 1.8 Ghz processor and can handle two RealSense 400 Series cameras simultaneously. The AGX Orin has a 2.2 Ghz processor.

For four simultaneously active cameras on the same computer, a processor equivalent to an Intel Core i7 or better is recommendable.

It is possible that whilst your board can handle two cameras simultaneously at almost 30 FPS, three cameras is a processing tipping-point where performance begins to degrade. You could test whether performance improves if the FPS for all cameras is reduced from 30 to 15.

michaelcukier commented 1 year ago

Thanks but I am monitoring CPU usage and when running three cameras at the highest resolution and 30FPS, it never goes above 40%.

On 13 Jan 2023, at 15:19, MartyG-RealSense @.***> wrote:



Hi @michaelcukierhttps://github.com/michaelcukier As the number of RealSense cameras that are simultaneously active on the same computer is increased, the processing burden on the computer of handling those multiple cameras will increase. For example, the Up Squared single-board computer has a 1.8 Ghz processors and can handle two RealSense 400 Series cameras simultaneously. The AGX Orin has a 2.2 Ghz processor.

It is possible that whilst your board can handle two cameras simultaneously at almost 30 FPS, three cameras is a processor load tipping-point where the performance of the cameras begins to degrade. You could test whether performance improves if the FPS for all cameras is reduced from 30 to 15.

— Reply to this email directly, view it on GitHubhttps://github.com/IntelRealSense/librealsense/issues/11321#issuecomment-1382007068, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AH6VJ7ID7PUMUSHMLYELZTLWSFW6FANCNFSM6AAAAAAT2PODFA. You are receiving this because you were mentioned.Message ID: @.***>

MartyG-RealSense commented 1 year ago

Are you capturing only depth frames, or both depth and color frames please?

michaelcukier commented 1 year ago

Both depth and color.

On 13 Jan 2023, at 17:25, MartyG-RealSense @.***> wrote:



Are you capturing only depth frames, or both depth and color frames please?

— Reply to this email directly, view it on GitHubhttps://github.com/IntelRealSense/librealsense/issues/11321#issuecomment-1382165778, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AH6VJ7MHVWIOVXQKCQ43IFLWSGF2DANCNFSM6AAAAAAT2PODFA. You are receiving this because you were mentioned.Message ID: @.***>

MartyG-RealSense commented 1 year ago

If both depth and color are enabled then there may sometimes be a drop in FPS. A fix for this can be to have auto-exposure enabled and have an RGB option called Auto-Exposure Priority disabled. This causes the RealSense SDK to try to enforce a constant FPS for both depth and color streams. A simple Python code snippet for disabling Auto-Exposure Priority can be found at https://github.com/IntelRealSense/librealsense/issues/5885#issuecomment-590502775

Edit: The D405 model does not have the Auto Exposure Priority setting, so that excludes that particular solution in this case.

If you are using manual exposure values then there can be a drop in FPS if exposure goes into a certain range, as described at https://github.com/IntelRealSense/librealsense/issues/1957#issuecomment-400715579

michaelcukier commented 1 year ago

I'm not using manual exposure values. I still tried adding the code snippet you sent and I'm getting "RuntimeError: object doesn't support option #30".

On 13 Jan 2023, at 17:36, MartyG-RealSense @.***> wrote:



If both depth and color are enabled then there may sometimes be a drop in FPS. A fix for this can be to have auto-exposure enabled and have an RGB option called Auto-Exposure Priority disabled. This causes the RealSense SDK to try to enforce a constant FPS for both depth and color streams. A simple Python code snippet for disabling Auto-Exposure Priority can be found at #5885 (comment)https://github.com/IntelRealSense/librealsense/issues/5885#issuecomment-590502775

If you are using manual exposure values then there can be a drop in FPS if exposure goes into a certain range, as described at #1957 (comment)https://github.com/IntelRealSense/librealsense/issues/1957#issuecomment-400715579

— Reply to this email directly, view it on GitHubhttps://github.com/IntelRealSense/librealsense/issues/11321#issuecomment-1382177723, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AH6VJ7P6IFI6KPVFVIBOQ33WSGHDFANCNFSM6AAAAAAT2PODFA. You are receiving this because you were mentioned.Message ID: @.***>

MartyG-RealSense commented 1 year ago

I added an update that the D405 model does not have the Auto Exposure Priority setting, so that excludes that particular solution in this case. As message edits do not go to your email, I will add additional thoughts to a new comment. :)

MartyG-RealSense commented 1 year ago

A way to confirm whether the FPS is slowing due to both depth and color being enabled would be to test with only depth frames and see if the third camera still has the FPS drop.

michaelcukier commented 1 year ago

This is correct. I have just tried now running all 3 cameras with only depth enabled, and I'm getting exactly 30FPS for all three cameras, so no drop in FPS at all.

On 13 Jan 2023, at 17:45, MartyG-RealSense @.***> wrote:



A way to confirm whether the FPS is slowing due to both depth and color being enabled would be to test with only depth frames and see if the third camera still has the FPS drop.

— Reply to this email directly, view it on GitHubhttps://github.com/IntelRealSense/librealsense/issues/11321#issuecomment-1382185489, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AH6VJ7IR6VHZCJO6F3YTYSLWSGIDRANCNFSM6AAAAAAT2PODFA. You are receiving this because you were mentioned.Message ID: @.***>

MartyG-RealSense commented 1 year ago

Thanks very much for the confirmation. If you need to have both depth and color enabled, please next test whether performance of the third camera improves if the FPS for all cameras is reduced from 30 to 15.

michaelcukier commented 1 year ago

At 15 FPS performance seems better. I ran 3 tests:

1st test: 15.0/15.0/15.0 2nd test: 13.5/13.9/15.0 3rd test: 15.0/15.0/15.0

However we do need 30 FPS for this application.

On 13 Jan 2023, at 18:08, MartyG-RealSense @.***> wrote:



Thanks very much for the confirmation. If you need to have both depth and color enabled, please next test whether performance of the third camera improves if the FPS for all cameras is reduced from 30 to 15.

— Reply to this email directly, view it on GitHubhttps://github.com/IntelRealSense/librealsense/issues/11321#issuecomment-1382208031, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AH6VJ7NLGIPIM2HGYCB3M4LWSGKZPANCNFSM6AAAAAAT2PODFA. You are receiving this because you were mentioned.Message ID: @.***>

MartyG-RealSense commented 1 year ago

Are you able to reduce the resolution instead of the FPS in order to reduce the volume of data that needs to be processed?

michaelcukier commented 1 year ago

Reducing the resolution also improves things a lot, and we I get 30 FPS in my tests. But we need the highest resolution possible, 1280*720.

On 13 Jan 2023, at 18:43, MartyG-RealSense @.***> wrote:



Are you able to reduce the resolution instead of the FPS in order to reduce the volume of data that needs to be processed?

— Reply to this email directly, view it on GitHubhttps://github.com/IntelRealSense/librealsense/issues/11321#issuecomment-1382240970, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AH6VJ7M7STWJWVBUEFCT3KDWSGO4RANCNFSM6AAAAAAT2PODFA. You are receiving this because you were mentioned.Message ID: @.***>

MartyG-RealSense commented 1 year ago

As you are using the color stream, potentially you could get some extra performance by building the librealsense SDK on your Jetson with CUDA support enabled if you have not done so already, as the SDK's CUDA support automatically offloads the processing of YUY to RGB color conversion from the Jetson's CPU to its Nvidia graphics GPU so the CPU does not have to do that color processing.

MartyG-RealSense commented 1 year ago

Hi @michaelcukier Do you require further assistance with this case, please? Thanks!

michaelcukier commented 1 year ago

Hi @MartyG-RealSense Thank you. I compiled using CUDA support and the problem still persist. GPU usage however seems rather low (around 30%) when running three cameras.

I just tried buying a PCIe USB 3.2 gen2 expansion card for my Jetson Orin and I connected a single D405 to it. It behaves strangely: the camera captures a couple frames and then freezes completely. Note that if I connect a USB webcam to the PCI it works fine. Any idea?

MartyG-RealSense commented 1 year ago

Which Ubuntu version and JetPack version are you using with Orin, please? D405 has been able to be successfully used by RealSense users with Jetson Orin, Ubuntu 20.02 and JetPack 5.0.2, as described at https://github.com/IntelRealSense/librealsense/issues/10416

michaelcukier commented 1 year ago

I have the ones you have just mentioned.

MartyG-RealSense commented 1 year ago

Thank you. Have you been able to test D405 successfully with other RealSense programs such as realsense-viewer?

Also, official D405 support was introduced in librealsense version 2.51.1, so 2.51.1 or newer should be used with D405.

michaelcukier commented 1 year ago

Yes, with the realsense-viewer it seems to work fine. The librealsense I'm using is the latest one

MartyG-RealSense commented 1 year ago

Does the camera still freeze if used with the basic Python multiple camera script at https://github.com/IntelRealSense/librealsense/issues/1735#issuecomment-1264583048

michaelcukier commented 1 year ago

The camera freezes when used in single-camera and/or multi-camera setup. Again this is only when connected through a PCIe USB expansion card.

MartyG-RealSense commented 1 year ago

It is difficult to advise about PCIe USB expansion cards as I have not used one for years, and it was on a desktop PC rather than a Jetson. On the USB expansion cards at that time though, they had an optional connector for linking the expansion board to the computer's power supply so that it could handle devices with high power draw instead of just low-draw devices such as keyboard and mouse. It would be worth checking whether your own card has some form of external power connector in case your expansion card is providing insufficient power to the RealSense camera for it to be able to operate.

michaelcukier commented 1 year ago

Ive just check and it seems the max supply for my PCIe card on the USB-A port is 5W. Is this enough for a single d405 ?

MartyG-RealSense commented 1 year ago

You should budget 2W for each RealSense camera. So if the card supplies 5W shared between all devices attached to it then it should be sufficient for a single D405 so long as there are not additional devices attached to the card that are drawing power.

michaelcukier commented 1 year ago

Ok then we discard this problem, as in my tests the only device connected to the PCIe is a single D405.

MartyG-RealSense commented 1 year ago

What happens if you use realsense-viewer with the camera attached to the PCIe card?

michaelcukier commented 1 year ago

It shows me the first frame and then freezes. I get a "no frame received" in red error after a bit.

MartyG-RealSense commented 1 year ago

In a Jetson discussion about PCIe USB expansion cards at the link below, one commenter suggests adding pcie_aspm=off to the kernel command line in case the card has a problem with low power states.

https://forums.developer.nvidia.com/t/third-party-pci-e-usb-3-0-expansion-card-for-xavier-board/

MartyG-RealSense commented 1 year ago

Hi @michaelcukier Did the advice in the above comment make a difference, please? Thanks!

MartyG-RealSense commented 1 year ago

Hi @michaelcukier Do you require further assistance with this case, please? Thanks!

MartyG-RealSense commented 1 year ago

Case closed due to no further comments received.