Trying to limit the maximum exposure time in auto-exposition (better IQ with low blur for SLAM)

[Gomzinator]

Start with Why?

I am developping SLAM algorithm for my laboratory at CEA (french R&D institue) and I bought an OAK-D W for benchmark purposes. To me it is crucial that even in low light environment I have an upper bound on my camera exposure time to avoid blurred images. From my understanding, it is not currently possible to have such upper bound on exposure time while having automatic exposure from the 3A algorithm. Only manual exposure can achieve what I need, but in that case I would need to implement "automatic exposure" myself to set a new value with the manual exposure command.

How can this be solved?

There are a few ways I could achieve what I want : 1/ Setup a high framerate to force the 3A algorithm to use a lower maximum time of exposure and switch to gain 2/ Setup a maximum value for the exposure value and let the 3A algorithm works as usual 3/ Skip the automatic exposure/gain part and code a small exposure/gain computation that set manual values for each "next frame" in a Script Block based on "preview" RGB camera output (scaled to something small like 300x300, and converted to YUV)

For 1/, I tried to set Mono and RGB camera to 200fps and add a Script block to skip some images (subsampling of 10 to achieve 20fps). I thought that it could work but the SDK/API does not let me go higher than 120fps for Mono and 60fps for RGB. It might not be feasable to work around long automatic exposure time that way ...

For 2/, there is a mention about tuning blob for the 3A algrithm here : https://github.com/luxonis/depthai-python/blob/main/docs/source/tutorials/image_quality.rst#camera-tuning. I understound that it's not a trivial tasks based on https://github.com/luxonis/depthai-core/issues/433 I did not find information about how this tuning blob is generated (name of the Intel tool) nor an "open source library" with available blob that might fit my use case. Can you tell us where did the blobs linked in the camera-tuning section come from ? If there is a blob that limits the exposure to 5ms instead of letting the exposure go to 200ms it would be perfect !

For 3/, I don't want to start something time consuming if you think that it's not reasonnable/achievable. I don't need this to have a high refresh rate, but I might have a hard time implementing this one if I believe this issue : https://github.com/luxonis/depthai-core/issues/688

Do you have a recommendation regarding this problem ? I am pretty excited about Luxonis products and all the things that I could achieve with all the hardware available (especially the OAK-FFC series) but this issue would be a deal breaker in my case.

Olivier GOMEZ

[Gomzinator]

Hello,

Is it possible to have at least more information about the tuning blob for the 3A algrithm ? (explained in the camera-tuning section)

If I know which Intel's tool to use, I could try acquiring a licence and partialy solve my problem (by hardcodeding an upper-bound on the exposure and maybe open sourcing the resulting blob if that's possible)

[themarpe]

@Gomzinator for that you'll have to step in contact with Intel, unless you reverse it, I guess:)

[drdas]

This is a deal breaker for people doing SLAM and looking to transition from the Intel D400 series to the OAK-D Pro. I am not sure why it is not important for almost all dynamic, real-time computer vision algorithms to set the auto exposure value limit (to avoid motion blur) from the API.

The need for a separate tuning blob is interesting, not sure I understand why the limit cannot be sent from the API to the camera firmware.

Either way it would be good to understand this and mention this at the top of the camera control wiki/docs so people don't have false hopes of using this camera to replace the Intel D400 series in computer vision pipelines.

[themarpe]

Update - we've added the capabilities, which took a bit but required reworking/hacking around the underlying 3A impl, to put it into such state: #926 - but its ready now and can be used to lock max exposure as one sees fit, specifically for such needs as described in the OP :tada: