Long-Range Depth Model - OAK-D-LR

[Luxonis-Brandon]

Pre-order on shop here

Start with the why:

While the OAK-D Series 1 and 2 (here) cover medium depth ranges (from ~20cm to 16 meters), and the coming OAK-D-SR (short-range) series (USB https://github.com/luxonis/depthai-hardware/issues/241, PoE https://github.com/luxonis/depthai-hardware/issues/244) specializes in close-in depth (from 0 to 1 or 2 meters), OAK/DepthAI technology is actually capable of much longer depth sensing - and we've tested up to 350 meters (as below) when doing a 30cm stereo baseline:

And the only current way to accomplish this is to use OAK-FFC (e.g. OAK-FFC-3P or OAK-FFC-4P with modular cameras such as the OV9282; 1MP Global Shutter Grayscale, OV9782; 1MP Global Shutter Color, or AR0234; 2.3 MP Global Shutter Color.

This works, but is not "production ready" and dealing with FFC cables is just annoying. So although it is possible to make a wide-stereo-baseline + narrow-FOV stereo pair using the options above (as shown below) such a setup is larger and harder to integrated than is desirable in many cases.

For longer ranges, using the setups above, we've found that a stereo baseline of 15cm (2x the 7.5cm baseline of the OAK-D Series 2), coupled with variable optics, can cover quite a wide range of depth sensing needs. And also from this testing, we've found the AR0234 to be quite beneficial for long-range sensing, given its large pixel size (matching the OV9282) while having a higher resolution of 2.3MP, which effectively doubling the maximum depth sensing range compared to the OV9282.

The AR0234 also provides the benefit of supporting both global shutter grayscale (for maximizing low-light performance) and also global shutter color (for native pixel-aligned RGBD).

The desired maximum depth range various quite a bit per application - with some situations requiring 100 meters, others 200 meters, and some 300+ meters. (The furthers the Luxonis team has tested with the AR0234 is 350 meters.) Supporting M12-mount lenses in the design enables choosing optics with FOVs (fields of view) corresponding to the required sensing distance.

Move to the how:

Leverage existing DepthAI ecosystem support for AR0234 to implement a dual-AR0234 M12-mount OAK-D-LR.

Move to the what: [NEW; based on feedback below]

• triple AR0234 [NEW]
• Camera Spacing {right} <-5cm-> {middle} <-10cm-> {left} [NEW]
  • Gives 3 stereo baselines: 5cm, 10cm, and 15cm for short, medium and long-range depth
• M12-mount lenses
• M12 lock rings (to keep focus/alignment)
• Front Gorilla Glass [NEW; Replaceable front for various filter options by unscrewing front]
• USB3.1
• 802.3at PoE [NEW]
• IP67 Sealed [NEW]
• Aluminum Enclosure
• OAK-SoM-Pro based so can support Myriad X and KeemBay equally [NEW]

[Erol444]

@ynjiun please send an email to support@luxonis.com.

[Luxonis-David]

! Update on OAK-D-LR compatibility with RVC3 based OAK-SoM-Pro-S3 (DM2399) !

We will be fulfilling first batch of samples assembled with RVC2 OAK-SoM-Pro only. The reason for being, DepthAI is not yet fully supported on RVC3 and we are also working on missing working drivers for the PCIe --> ETH bridge used in the product.

All preorders will be subject to the above if not specifically noted that order is placed for RVC3 based product and someone would be willing to wait on RVC3 support which is at the moment TBD.

[SzabolcsGergely]

5. is that possible to provide a call-back API to allow user to define a function to fuse 3 dumpdispcost says (3,height,width,96) into 1 dumpdispcost (1,height,width,96) and pipe it back to compute the final depth and disparity? If this is possible, then the above API achitecture can be simplified to a single depth and disparity output instead of 3X each.

@ynjiun Feeding precomputed cost values to stereo HW is possible (however not exposed as API). However dumping out 3x100 MB buffers puts a lot of stress on memory which slows down stereo/whole system. Other than that, fusing them takes a lot of time too, due to the sheer amount of data that needs processing. Not sure how cross-related-partial-dump helps, which you brought up in the mail, but feel free to expand the idea.

Cost values are refined using 3-way SGBM. Dynamic programming of SGBM is the forward path of Viterbi Algorithm. It computes the minimum accumulated cost of the path for all the disparity values of each pixel. Because we don't know which disparity is finally taken, we need the cost for all possible disparity values. Path for taking the minimum acuumulated cost can be totally different.

Also note that cost dump debug output is not available on RVC3. None of the debug outputs are available on RVC3. In addition confidence map output is not available either, since the HW doesn't support outputting it. A simple way of fusing multiple disparity maps on RVC2 would be using the confidence map and taking the one that has the higher confidence (lower confidence score in our case).

[ynjiun]

Cost values are refined using 3-way SGBM.

Hi @szabi-luxonis

Thank you for the feedback.

Regarding "3-way SGBM", which 3-way? left-to-right (A1), top-left-to-bottom-right (A2), top-to-bottom (A3)? Thanks.

[SzabolcsGergely]

Cost values are refined using 3-way SGBM.

Hi @szabi-luxonis

Thank you for the feedback.

Regarding "3-way SGBM", which 3-way? left-to-right (A1), top-left-to-bottom-right (A2), top-to-bottom (A3)? Thanks.

The SGBM algorithm paths implemented:

• left of the current pixel: →
• right of the current pixel: ← and
• vertically above the current pixel: ↓

[ynjiun]

Also note that cost dump debug output is not available on RVC3. None of the debug outputs are available on RVC3. In addition confidence map output is not available either, since the HW doesn't support outputting it.

Hi @szabi-luxonis

hmmm.... "confidence map output is not available either": is this also a h/w limitation? So what's supported in RVC3 then? could we have a list to compare RVC2 vs. RVC3 support outputs? Thank you.

[ynjiun]

RVC3. In addition confidence map output is not available either, since the HW doesn't support outputting it.

Hi @szabi-luxonis

For RVC3, would it be still possible to set the confidence threshold to remove those low confidence disparity?

Thanks.

[SzabolcsGergely]

RVC3. In addition confidence map output is not available either, since the HW doesn't support outputting it.

Hi @szabi-luxonis

For RVC3, would it be still possible to set the confidence threshold to remove those low confidence disparity?

Thanks.

yes

[stephansturges]

Quick check-in: does this model come equipped with an IMU? I suspect I'm not the only one who is really interested in using this sensor for SLAM...

[cafemoloko]

@stephansturges OAK-D-LR comes with BMI270.

[Erol444]

Note: for production we plan to have BNO085/6. The background is that when we were designing the LR we didn't have any stock of BNO, but now that we have them we will redesign the device. Thoughts?

[stephansturges]

Thanks for confirming the early units come with BMI270. Is this the same unit as in the CM4-PoE? BNV086 would be ideal from the specs that I see online... ultimately I think high-quality IMU will be very valuable for this sensor because the range makes it really useful for outdoor VIO / SLAM.

[stephansturges]

For some context: here is a great video from SpectacularAI on the importance of IMU quality for VSLAM https://www.youtube.com/watch?v=HpeVrCPTlF4

Those guys use the OAK cameras, and I would recommend asking them for recommendations on this topic!

[Luxonis-David]

Update: We shipped the first samples of OAK-D-LR yesterday so preorders will be fulfilled in these days just before the New Year. Note: These units may not have the best depth due to ongoing development, so recalibration later on might be needed to achieve best depth results. Since we do promote this product being in early access we figured out that it is better shipping the units sooner than later.

[ynjiun]

Hi @Luxonis-David

Excited! Just received the unit.

Try to connect to USB and run a simple stereo depth. But encounter:

[1844301081575E1200] [1.11] [1.358] [ColorCamera(1)] [error] Camera not detected on socket: 2
[1844301081575E1200] [1.11] [1.359] [ColorCamera(0)] [error] Camera not detected on socket: 1

Could we post a simple demo python code to show how to construct a pipeline of the stereo depth for (Left/Right color camera pair) and other pairs for example (Left/Middle pair) or (Middle/Right pair)?

Thanks a lot!

[themarpe]

@ynjiun

Do you mind running on develop branch? There is also a depth_preview_lr.py example on depthai-python repository, which will do just that :)

[ynjiun]

Sure. Just "git pull origin develop" with origin https://github.com/luxonis/depthai-python.git And run the depth_preview_lr.py then got the following message:

(oak) (base) paul@uai2:~/oak/depthai-python/examples/StereoDepth$ python depth_preview_lr.py 
[1844301081575E1200] [1.11] [1.343] [ColorCamera(1)] [error] Camera not detected on socket: 2
[1844301081575E1200] [1.11] [1.343] [ColorCamera(0)] [error] Camera not detected on socket: 1
[1844301081575E1200] [1.11] [1.385] [StereoDepth(2)] [error] Maximum disparity value '1520' exceeds the maximum supported '1024' by median filter. Disabling median filter!

Do I need to update other SDK? such as depthai, or others? Please advise. Thanks.

[themarpe]

@ynjiun Steps so far spot on, just run python examples/install_requirements.py and you should be set.

(This updates the depthai python library to the latest develop one, which contains support for LR)

[ynjiun]

Hi @themarpe

Cool. It works now! By the way, do I need to recalibrate the stereo cameras (rectified parameters)? if yes, please provide me the link (reference process) for re-calibration. Thanks a lot for your help!

[ynjiun]

so recalibration later on might be needed to achieve best depth results.

Hi @Luxonis-David

I got my unit running thru @themarpe's help. Now I tried to calibrate.py the unit and got the following error message:

Error: Couldn't retrieve left, rigth and color frames for more than 5 seconds. Exiting...

What shall I update to proceed?

I did "git pull origin develop" where origin is https://github.com/luxonis/depthai.git

Please advise. Thanks.

[Luxonis-David]

@ynjiun we are still working on the depth quality so I wouldn't say that recalibrating the device at the moment would give any huge difference on what you already have in hand WRT depth quality. If you still want to recalibrate the device you can follow this steps:

1. checkout modular-calibration-lr branch
2. Install requirements in requirements.txt
3. Run calibration script setting the parameters according to your target, example: ./calibrate.py --board resources/boards/OAK-D-LR.json -s 8.05 --defaultBoard -cd 0 -c 1 --fps 60

Please note that you should use a bit bigger target for OAK-D-LR because of the larger baseline. A bigger TV would also do it for a start.

[themarpe]

@ynjiun

You may calibrate OAK-D LR on modular-calibration-lr branch of depthai. We'll merge this in soon Example of command line to do so:

./calibrate.py --board resources/boards/OAK-D-LR.json -s 8.05 --defaultBoard -cd 0 -c 1

Note -s 8.05 parameter should be changed according to your boards' square size

[ynjiun]

Hi @Luxonis-David and @themarpe

Thanks for the calibration code and it runs. I use a 60" big screen TV for the chart with -s 8.25. After calibration, the rectification errors are all in range 0.1~0.2 and seems reasonably good (what's the typical expectation here? smaller than 0.1 to be considered as good calibration?). However, if I visually check the pixels, it might still off by about 2 pixels in y dimension. Plus when running the depth preview lr and I add the depth output, it seems the depth is off by a scale factor of 2/3(?).

Few questions:

1. in OAK-D-LR.json: hfov: 90; but according to spec the DFOV:96 => HFOV would be around 75 not 90 right?
2. OAK-D-LR native resolution is 1920x1200: what's the resolution used by calibration.py for calibration? resize to 1280x800 or crop to 1280x800?
3. when depth computation: depth = baseline*focal/disparity => what's the focal value in pixels? Is 1920x1200 and HFOV=90 used to compute focal? or 1280x800 and HFOV=?? used to compute focal?
4. Is there a way to calibrate using native resolution 1920x1200? if so the intrinsic/extrinsic matrix need to be changed for stereoDepth reification for 1280x800 resolution?

Please advise. Thank you very much for your help.

[Luxonis-David]

@ynjiun sorry about the delay, we used M25360H06S lens on these prototypes

1. According to my calculations the exact HFOV for the AR0234 sensor should be 91.7°. The calculations could be wrong so I will get the FOV measure on the samples today when I get to the office. For the rest I would need @themarpe to answer.

[themarpe]

@ynjiun

2. OAK-D-LR gets calibrated at full 1920x1200 resolution
3. Focal is taken from intrinsics matrix first element if i'm not mistaken
4. Calibration happens at full while Stereo happens at downscaled 2/3 resolution. This change also modifies matrices accordingly.

And thanks for the headsup on the Y displacement issue. Do you perhaps still have the dataset and any relevant logs of that? If so, do you mind sending those over for us to check

[chad-green]

According to the AR0234 datasheet the active area of the sensor is 5.784 x 3.624 mm: https://www.onsemi.com/pub/collateral/570ck.pdf

[Luxonis-David]

@ynjiun the above calculations are not correct as I suspected in first place, I have for some reason selected a Stereographic lens, but M25360H06S is rectilinear lens. The FOV is as follows and was also confirmed practically. Horiz. FoV --> 77.3 ° Vertical FoV --> 53.1 ° Diag. FoV --> 86.7 °

@chad-green the exact die size is 5805um x 3.645um thus the pixel count is 1935 x 1215.

[stephansturges]

I’ve been working with the LR and for my use it’s already excellent, I’m excited to see how it improves with RVC3 and the final software.

quick question: is there any chance we will see full IP sealing on the final version? If not: can you develop a custom housing alternative that I can purchase separately or should I design my own? (this is crucial for my use case)

[Luxonis-David]

@stephansturges we are working on improving the IP rating of the device, especially the rain cover will be improved in order that we get better waterproofness than just from water splash. Second batch of the devices will already have this and you can expect those in March/April. X-coded M12 with waterproof USB connector is also on the roadmap but that would come later in the year. Thoughts?

[stephansturges]

@Luxonis-David are the first batch supposed to come with a rain cover already? I didn't receive any with my delivery at least, I'd be curious to test this first and see if that will solve the use case for me. To be honest I'm 90% certain that the rain cover won't cut it for a "final product" spec for my use case however, what I really need is the full waterproof spec of the CM4-PoE.

Worst case I can design the casing myself as in the prototype I'm still seing overheating in continuous operation as will the CM4-PoE, even inside in 15 degrees C... so I might have to design my own case to solve the thermal issues anyway.

[Luxonis-David]

@stephansturges we were not shipping the rain-covers because the design needs to be corrected first. When we manage to get new covers we can ship them. Will keep you updated.

About CM4, we did manufacture the updated design with USB HUB that does not overheat, you can request a replacement which should solve the issues with overheating. We are requesting shipping back the old units though.

[stephansturges]

@stephansturges we were not shipping the rain-covers because the design needs to be corrected first. When we manage to get new covers we can ship them. Will keep you updated.

-> gotcha... I misinterpreted your previous post as meaning there was a "gen1 raincover" that would be replaced with a "gen2 raincover"

About CM4, we did manufacture the updated design with USB HUB that does not overheat, you can request a replacement which should solve the issues with overheating. We are requesting shipping back the old units though.

That's very interesting... I have a number of those units that will need replacing, thanks for the tip.

[stephansturges]

FYI with the current design of the LR model I lose connection once the SHAVE cores get over 42 degrees celsius, which is about 1h30 of operation in the example below (in a small-ish room with ambient temperature at 18 degrees C).

I'm going to test more and try to get more details, but I've observed this a dozen times already.

[themarpe]

@stephansturges thanks for the data point. This is indeed an odd number, as it doesn't relate to anything (in terms of different processing, etc...). I would reckon that this is more of a time sensitive issue than temperature.

If you have a MRE for this issue, we'll take a look and try to replicate on our end

[stephansturges]

@stephansturges thanks for the data point. This is indeed an odd number, as it doesn't relate to anything (in terms of different processing, etc...). I would reckon that this is more of a time sensitive issue than temperature.

If you have a MRE for this issue, we'll take a look and try to replicate on our end

I think you are right that it's not a temperature problem, testing specifically for this issue I'm seeing it run stable up to 50 degC, and with the CM4 I only had systematic crashes when the SHAVE cores reached 60 degC...

I suspect this is something to do with the stereo algorithm because that's the only part of the process where I'm not catching errors, I'll experiment with that and let you know what I find.

[ynjiun]

@ynjiun the above calculations are not correct as I suspected in first place, I have for some reason selected a Stereographic lens, but M25360H06S is rectilinear lens. The FOV is as follows and was also confirmed practically. Horiz. FoV --> 77.3 ° Vertical FoV --> 53.1 ° Diag. FoV --> 86.7 °

@chad-green the exact die size is 5805um x 3.645um thus the pixel count is 1935 x 1215.

@Luxonis-David

Thank you for the information. It helps a lot.

Question: I understand the current stereoDepth resolution limit is 1280x800. Is there a way to get a rectifiedLeft/Right with 1920x1080 resolution using existing depthai pipeline? Please refer to an example code to show how. Thanks for your help again.

[SzabolcsGergely]

@stephansturges what is "SHAVE block temp?

[Luxonis-David]

@szabi-luxonis would you know the answer to the question that @ynjiun did post above about the depth resolution?

[SzabolcsGergely]

@ynjiun the limit is 1280x65535 for stereo depth. You can use ImageManip or Warp node and provide the rectification mesh to obtain rectified outputs at full resolution.

[stephansturges]

@stephansturges what is "SHAVE block temp?

t.upa as reported by: https://docs.luxonis.com/projects/api/en/latest/samples/SystemLogger/system_information/

On the first design of the CM4-PoE sensors this was an excellent indicator of when the system is overheating: when the SHAVE block temp reached 60degC you loose USB connection every time.

[stephansturges]

I think I've caught the error that is causing my pipeline to fail randomly. I don't have an MRE you can run yet but sharing the error in case it helps.

[ynjiun]

You can use ImageManip or Warp node and provide the rectification mesh to obtain rectified outputs at full resolution.

@szabi-luxonis thank you for the information. Could you provide an example code or link to show how to retrieve the retification mesh from the firmware to setWarpMesh and activate ImageManip? Thanks a lot for your help.

[SzabolcsGergely]

You can use ImageManip or Warp node and provide the rectification mesh to obtain rectified outputs at full resolution.

@szabi-luxonis thank you for the information. Could you provide an example code or link to show how to retrieve the retification mesh from the firmware to setWarpMesh and activate ImageManip? Thanks a lot for your help.

https://github.com/luxonis/depthai-python/blob/main/examples/ColorCamera/rgb_undistort.py

[chengguizi]

Is the hardware schematics available for OAK-D-LR? I could not find on https://github.com/luxonis/depthai-hardware (though I can find SR version already)

One thing I would like to check from the schematics is that:

• How is the three camera being synchronised now? Is it through FSIN? Or the old way that is using FW-based micro timing adjustment?

[Luxonis-David]

@chengguizi sensors can be synchronized through FSIN but that needs support that in FW, which will be done when the official release is made for OAK-D-LR if not earlier. I have attached the schematic of first design revision (R0M0E0), but please note that we are refactoring the schematic and also the layout will be updated with smaller corrections. Schematic: Production.PDF

@Luxonis-Blaz could you please upload the project files to the hardware repo when you get a chance? Best would be to add files when the latest schematic with all corrections will be available and modified PCB layout so that it would serve as a good reference.

[chengguizi]

Thanks for the quick reply! @Luxonis-David

I have just checked, it is great that all three cameras' FSIN are connected together.

And yes, currently the FW in develop seems to have problem with hardware sync support. I tried putting camA as master (dai::CameraControl::FrameSyncMode::OUTPUT), and camB and camC as slave (dai::CameraControl::FrameSyncMode::INPUT). It didn't work. (exact behaviour, camb is streaming, but not cama and camc)

If I turn off the hardware sync. I can see the camB and camC are effectively synchronised, but camA is like free running, not synced.

Is this expected, for now?

[chengguizi]

I also notice that the LED at the bottom changes color when i start streaming, pretty neat!

It seems to be achieved by IO38,39,43. Is this feature available for all OAK-SoM-Pro based product? Where do I find some documentation on the status of the LEDs. (I suppose it is a new feature:) )

[chengguizi]

I have a question on what's happening when we feed color image stream to the StereoDepth Node.

https://github.com/luxonis/depthai-python/blob/develop/examples/StereoDepth/depth_preview_lr.py

From the example script, I can see the ColorCamera isp is directly fed into StereoDepth node, without converting to RAW8 first. Did the FW automatically add the color conversion itself? Or perhaps the StereoDepth conveniently just take the first channel, which i suppose is R channel?

[SzabolcsGergely]

@chengguizi StereoDepth takes the first image plane. First image plane of NV12(video output)/YUV420(isp output) is the luminance (grey).

[chengguizi]

@szabi-luxonis Got it! That makes sense. So actually for all practical purpose, we can directly feed in color image to StereoDepth. I vaguely remember this wasn't allow in the pass, on FW.