Noise in IR image and point cloud quality

[BartvanMarrewijk]

I actually have two issues. I am creating a point cloud from my lettuce with a top-down image. However, the accuracy of my point cloud seems to be influenced by some noise in IR image.

Device OAKD PRO AF

This noise is shown in the image below Left rectified IR image Right rectified IR image Point cloud images

I am not sure what is causing this. At first I thought a gain setting, but since it only occured at the lower part of the image i guess it is something else. Does anyone experienced this before?

Next to that even for the sharp parts, within the point clouds the leaves appears to be really curvy with "impossible" fluctuations. I am already using the high confidence presetting and appeared the post processing filters:

config.postProcessing.speckleFilter.enable = True ##default=False
config.postProcessing.speckleFilter.speckleRange = 50##default=50
config.postProcessing.temporalFilter.enable = True ##default=False ## really improves the image for statis scence
config.postProcessing.spatialFilter.enable = True

Still this example below shows a gap of 2.4 cm in z-direction, whereas this gap is actually a smooth leaf.

Although similar as this issue #945 regarding point cloud accuracy, I believe that given the IR images above this is another issue related to pointcloud quality

[themarpe]

Thanks for the report @studentWUR - we'll take a look at this and get back to you as soon as possible

[Erol444]

Hi @studentWUR , could you please send an email to support@luxonis.com, so we can replace the device asap? It seems there's an issue with the sensors themselves (the lower part is quite noisy). We apologize for the inconvenience.

[BartvanMarrewijk]

Dear @Erol444 & @themarpe . Thank you for your fast response. So I am not sure, if it is actually the sensor. For example: At 10:00 (left IR camera) (right IR camera)

At 11:00 (left IR camera) (right IR camera)

If it is really a sensor issue, I am afraid I have to send multiple cameras. I have another camera that is exactly facing the same problem. See example below: (Left IR camera)

Right IR camera

[Erol444]

@studentWUR are you using the latest depthai version? Readout should be decreased a bit, which might help.

[BartvanMarrewijk]

@Erol444 The camera with the issue above had the newest bootloader version 0.0.24, DepthAI version 2.21.2.0

[BartvanMarrewijk]

@Erol444 I found out that the issue might be related to AutoExposure setting. In the two imaes below I used the mono camera controller and added a print statement for the current exposure and ISO. With autoexposure:

With manual exposure (just pressing "k"), but ISO won't be lower due to minimum:

As visibile somehow both the print statements are the same. As evidence that the printed exposure setting is correct hereby another example with the manual iso set to 200.

@Erol444 Could you please have a look

[Erol444]

Hi @studentWUR , I am not sure what exactly is the problem, could you elaborate? Also note that you are likely averaging exposure times incorrectly - it can't be 0.

[BartvanMarrewijk]

So the problem is that the auto exposure resuts in image with a lot of noise. In the first image I send you can see al those spots , causing the point cloud to be incomplete. In the second image with the manual exposure the image looks much better without any noise. So somehow at high light conditions, my image ends up noise with the autoexposure setting.

mm then this averaging must be done somewhere in the msg, because the only thing what I do is this: print("exposure [ms] %s %d"%(cam,msg.getExposureTime().total_seconds()*1000)) print("ISO %s %d"%(cam,msg.getSensitivity()))

[Erol444]

Hi @studentWUR , Are you sure the 1st image is in fact AE, and not fixed? Regarding exposure time, you can print it like this:

imgFrame = qRgb.get()
print(imgFrame.getExposureTime())

[BartvanMarrewijk]

@Erol444

Thank you for your quick reply. Yes, because if I shift go to manual control and back to AE then it happens again. And yes, I use the same function as you to print the exposure. I can make a video if you want

[BartvanMarrewijk]

And to show the influence of this noise on the point cloud: This is with AE:

This is with manual setting exp to 1 and iso to 100

[Erol444]

Hi @studentWUR , I see, it might be that AE isn't capable of such high light levels. Could you maybe measure lux value on that area? One option would be to use ND filters to lower light intensity. Regarding the API, the implementation I used returns correctly:

0:00:00.019994

[BartvanMarrewijk]

Hi @Erol444,

sorry I used total.seconds() after getting the exposure. This explaines why both were showing the same exposure time, but different output. See updated results below: AE Manual

The manual result is better, because it is going to 10us. It seems that either the autoexposure cannot deal high light levels (outside) or it cannot select the minimum exposure time.

[Erol444]

Hi @studentWUR , I see - I just checked internally, and with the current camera tuning, AE can only lower exposure to 20us on OV9x82, as your screenshots above also confirm. I'd suggest using filters, if possible, to lower light intensity in such environments. Thoughts?

[BartvanMarrewijk]

Hi @Erol444. I am working with plants, so lowering the light intensity is not an option ;). Preferably I would not use any filters, since it is an additional manual step and to me this seems to be a software issue. As the manual setting shows it is possible to have a nice IR image in high light conditions. I might could make something if AE==20us, switch to manual hard coded exposure. The question is then when to switch back to AE control. Is there a possibility to do the calculation of the AE controller by hand? Is this code open source?

[Erol444]

Hi @studentWUR , one option would be to switch to AE every so often, and check whether AE sets exposure higher than 20us. One option would also be to switch to AE and check depth quality (maybe confidence, or fill-rate) and comparing that to manual exposure. Unfortunately, AE controller is quite complex and is part of the firmware (close source).

[BartvanMarrewijk]

Thanks for the suggestions, maybe the confidence would work, on the otherhand already from the Infrared image it can be seen that there is to much noise. Maybe an "sharpness" indicator is already enough. It might be interesting to build in a warn print statement in case the AE selects the minimum value of 20us? I think that more people that use this camera outdoors have the same issue.

At least I am happy we found the cause of this noise. Thanks for your help