SpectacularAI WARN: Dropping frames with Realsense D435i on Intel Pentium N4200

[patrickpoirier51]

Hello again,

I am working with the latest non-commercial plugin for realsense (1.22.0) On the development PC with an I7 there is no issue but I am planning to build a demonstrator Drone flying with an UP Squared based on an Intel N4200 Pentium.

Running basic demonstration python3 vio_visu.py show a lot of dropped frames. The probable cause is overload of the 4 cores as we can see here:

Can we pass parameter through vio_config.yaml in order to reduce cpu loading or this is only for playback ?

[oseiskar]

Hello Patrick,

I am not sure if this is going to fly [pun intended]. Unlike the Luxonis devices, RealSense cameras have no computer vision accleration (VPU), which make them badly suited for running the Spectacular AI SDK on lower-end processors.

While you can modify the YAML-configuration also through the internalParameters key-value pair map in the spectacularAI::rsPlugin::Configuration object (or reduce the input frame rate to reduce CPU consumption), the accuracy may suffer too heavily in changes large enough to succificently reduce the CPU usage.

The recommended option for lower-end CPUs would is a well-calibrated OAK-D (Wide) device https://shop.luxonis.com/products/oak-d-w .

BR, Otto

[patrickpoirier51]

Thanks Otto for the information.

Question: I might be interested to order the OAK-D Pro W instead and benefit of IR illumination and active stereo features. Will these features will enhance the SpectacularAI pipeline on low light and low textures ?

In the meantime, I will push experimentation and fly this I7 based 550 quad , I wrote the python script that extract for Json and convert to Mavlink vision_position_estimate_message. The process on an 11th gen I7 is insignifiant (less than 1%).

[patrickpoirier51]

Ok I see the use of https://shop.luxonis.com/products/oak-d-pro-w-dev on this video https://www.youtube.com/watch?v=zFXW7EDh_eE

So I will order the OAK-D Pro W as it is stock at Mouser in Canada

I will update here of my experimentshere and ultimately write a Blog on Ardupilot.org

[oseiskar]

I might be interested to order the OAK-D Pro W instead and benefit of IR illumination and active stereo features. Will these features will enhance the SpectacularAI pipeline on low light and low textures ?

If you want to try this, you can order a Pro W since it is as least as good as the "W" (practically the same if you turn off the IR).

However, I would not expect greatly improved results because the RGBD has its own challenges. The tricky thing is that if we turn on the IR projector, we must use the RGB camera instead of the gray cameras for feature tracking, because the IR pattern breaks that in the gray cameras. For this purpose, it is imporant that the RGB camera is global shutter and synchronized with the gray cameras. This is not the case with all Pro W variants (you can order them with different types of RGB sensors). This feature can be enabled with the useColor flag in the SDK, which is currently experimental.

If you record data on a live drone, also feel free to send that to us and we can check if there are some parameters that we would recommend tuning to improve performance in that setup. For example, the useVioAutoExposure may or may not help, depending on the case.

[patrickpoirier51]

Hello

I just received the OAK-D-W and tested on the Up Board, and the load is good now. Just need to minimize drift as discussed in issue 84