Unexpected artifact in depth stream when capturing through transparent surfaces

[tchapi]

Required Info
Camera Model	SR300
Firmware Version	multiple
Operating System & Version	Win 8.1 / Linux
Kernel Version (Linux Only)	4.4.0-101
Platform	PC
SDK Version	2.9.1

Description

We have been using different setups (PC, embedded platforms, etc) and various OS (Linux, Win) to test this behaviour and it is quite consistent (we tested on various SR300 firmwares and scanners, too).

We are trying to scan through transparent material, and we see a artifact, always at the same spot, when doing so.

We have tested on 3 different types of PMMA (obviously, glass is a no go since it's opaque to IR), including one that is opaque to visible light. For each test, we capture the scene that is after the material quite well (proportions, dimensions are respected and the model is as clean as if captured without the transparent material) BUT for a little spot in a specific place in the stream. The PMMAs are clean and there is no default on the surface that could explain the artifact.

This "spot" is linked to the orientation of the transparent material : at orientations roughly > 45°, it gets out of the frustum and is not detectable anymore. But when the material is somewhat parallel to the lens, it's here.

We tested in a dark room too, to eliminate any visible light incidence.

I have attached pictures for 3 tests that we have done (1) Transparent PMMA

(2) Opaque-to-Visible-Light PMMA

(3) Different Transparent PMMA

(1bis) Transparent PMMA at an angle (spot is no more visible)

My question is:

• Do you guys have any idea why this happens ? It looks kind of a reflection of IR, but why would it only be a single spot ? The index of refraction differs between the two surfaces obviously but I don't see how the law of refraction would give such a result
• Could this be a limitation of the library in itself ?
• Can it be detected / removed by the lib ? We have tried to detect it and remove it via reconstruction algorithms (poisson, marching cubes etc) with smoothing parameters but it's not up to what we expect

Any suggestion welcome ;) Thanks a lot Best

[dorodnic]

Hi @tchapi Could you please share the infrared images using the Viewer? Also, I see you are using 2.9.1, please try with the latest release (2.11.0)

[RealSense-Customer-Engineering]

[Realsense Customer Engineering Team Comment] hi @tchapi,

sorry, I want to confirm when you test, do you remove the lens cover? just want make sure it's removed so we can continue.

[JasonChang13]

somehow can't attach image from official account

[tchapi]

Hi all Sorry had not had the time to test with 2.11 yet, will do in the coming week. But as for the lens cover, yes, we remove it, it doesn't change anything to the results. We have tried also on a "bare" SR300 Camera module and it's exactly the same behaviour.

[tchapi]

Hi — Here it is with v2.11. The artefact is still very visible. I've attached the RAW file and the frame and intrinsic info (.raw.txt to rename →.raw) :

Frame	Info
Type	Depth
Format	Z16
Frame Number	41537
Timestamp (ms)	1394219.09
Resolution x	640
Resolution y	480
Bytes per pixel	2

Intrinsic	Info
Fx	476.389709
Fy	76.389709
PPx	315.013855
PPy	245.617447
Distorsion	Inverse Brown Conrady

Snapshot v2_Depth.raw.txt

[AntoineWefit]

Hi @tchapi,

Even though I'm not an expert, here is a small contribution.

I think your main problem is as you said : "It looks kind of a reflection of IR [...]" If this "blind spot" only appears when you try to scan special/reflective surfaces such as your PMMA, then I don't think you need to look any further. I'm pretty sure that the IR is somehow reflected by those surfaces and the camera is "blinding itself".

It's actually really easy to check, as @dorodnic said : "Could you please share the infrared images using the Viewer?" I would also suggest that you provide the RGB images, because you might simply see the reflection of the camera itself. If so, please check if the "blind spot" is actually superposed to the IR emitter location on the RGB image. I might be wrong, but that's my best suggestion.

If the problem is not corrected, you could also try to adjust (if possible) this parameters : Exposure Gain Laser Power

I would also point out : "obviously, glass is a no go since it's opaque to IR" I think this statement can be discussed because the SR300 laser wavelength is actually 860 nm, which is categorized as Near Infrared. The almighty Google says that this type of IR is not (or nearly not) affected by glass. Although, for greater wavelengths (such as those used for thermal imaging), you are right. You might then be OK to do further tests with glass :) Can somebody else confirm that ?

[RealSense-Customer-Engineering]

[Realsense Customer Engineering Team Comment] Hi @tchapi,

1. It's a good check to IR/RGB images to understand the actual scene, not just review the depth image as @AntoineWefit and @dorodnic mentioned. 2. Those parameters @AntoineWefit said also affect the depth image.
2. Cover material can be PMMA or glass with AR coating to insure the high transmission rate. Higher than 92%@860nm recommended for better use range as you can review SR300 datasheet chapter 8.7.2.

[tchapi]

Thanks for your valuable inputs.

Yes it's indeed the IR projector as we can see here and as I imagined :

In fact we had tested thick glass in the past (because we need a thick panel for our use case), and above 6mm it's literally opaque for the scanner, so I considered it a lost cause. Anti-glare / AR coating didn't help, and neither did polarised glass ... (we tested today)

Adjusting the laser power did not yield any result. As for gain and exposure, I haven't tested yet but I doubt this will improve things significantly since it will also have an effet on the quality of the scan itself (the model we want to scan behind the transparent surface).

So I guess I'm stuck with trying to remove this artefact programmatically. Any hint on how to do it ? I'm thinking about removing the dots "where the reflection is" (if ever I can find this easily) and then let Poisson do its magic to fill the hole, but any suggestion is welcome.

Thanks again for your time

[AntoineWefit]

I don't own an SR300 so I don't really know which parameters you can tune, but I think you should try 2 simple things that might solve your problem (under conditions), in order :

• If your object-to-be-scanned is located at a fixed distance from the camera. -> Measure that distance precisely and set the value in the Intel RealSense Viewer software under Advanced Controls/Depth Table/Depth Clamp Min. What it's going to do is restricting the depth of the scanned area by ignoring anything below this distance. I understood that you want to see something behind a glass, and not something in front of it, correct ? By setting the correct distance, you could ignore anything, including the glass, "too close" to the camera, which is quite nice. The downside is that you are stuck with that distance and can't move your model/camera closer along the Z axis anymore. You'll get better results with the camera and the glass being parallel. Be also aware that the units of the distance input is directly correlated with the value set under Controls/Depth Units (ie Depth Units : 0.001 -> Depth Clamp : [mm])

• As I see, the artifact is pretty small. It's not that bad, so it should be easy to correct with some corrections, as you said. -> Go in the Intel RealSense Viewer software, and tick the Post-Processing box. In here, some options will impact the filling of the holes : 1) If you don't really care about the amount of vertices of your output, increase the Decimation Filter. From what I understood, the Filter Magnitude will divide the number of vertices by 2^(input value) (ie Filter Magnitude : 3 -> Output number of points = Max number of vertices/2^3). So basically, """less points = less artifact""" :P 2) Under Post-Processing/Spacial Filter/Holes filling, switch to 2, 4 or 8 pixels radius depending of the size of your holes. I wouldn't recommend to increase this value too much because it'll deteriorate your output, specially the border of your object. 3) Finally, under the Post-Processing menu again, there is the Hole Filling Filter. I find this one to be quite powerfull (a bit too much imho), but really difficult to use, so tune it with care.

Keep us updated and good luck !

[tchapi]

Hello Antoine and thanks for your ideas.

I'll definitely test the depth clamp since the distance to the glass is mechanically fixed, so this could help a lot.

As for the post-processing options, it's hard to tune since we need a high number of points (and a good detail) for our use case, so I cannot decimate too much. I'll give it a go anyway (we already do heavy post-processing and fill holes to a certain extent).

Thanks again Best

[tchapi]

BTW, what is the easiest method to set depth clamp programmatically ? It seems it is doable in advanced mode but I only see this available for the D400 (not the SR300)?

[AntoineWefit]

As I don't own an SR300, I can't answer this question, sorry.

[RealSense-Customer-Engineering]

[Realsense Customer Engineering Team Comment] Hi @tchapi,

"depthClampMin" parameter is not supported by SR300.

1. You can try the visual preset in realsense-viewer, like short range, mid range, etc to see how it affects. The major controls are accuracy, confidence and filter. You can have a try.
2. Probably, you can refer to the concept of rs-align code and have a kind of foreground removal based on the depth value you know to remove the near-range objects.

[tchapi]

Thanks for this confirmation.

We have tried to fiddle with the presets but to no avail.

Just so anybody runs into the same problem, the easiest and most efficient way we found was to simply remove points near the "reflected IR center" before our reconstruction algorithm (Poisson). Fortunately this area was not a feature area for us.

YMMV but for us , this reflected IR center was located at {x: -0.0205f, y: 0.0100f } (in meters) and it had the shape of a rectangle (sort of) with dimensions {w: 0.0190f, h: 0.0160f}.

[RealSense-Customer-Engineering]

[Realsense Customer Engineering Team Comment] Hi @tchapi,

Nice to hear that you resolved the issue. If no other question, will close this.