missing depth data for small object when no background is present in frame.

[omerbrandis]

hello realsense support.

I'm trying to get depth data for a black/dark grey metal wire with 2mm diameter at a distance of 30cm to 130cm. using D435, rs version 2.16.1. , firmware version 05.10.06.00.

when i use the high density preset , and place a flat background (like a wall) behind the wire by a few dozen cm, the camera is able to calculate depth data at a distance of about 90cm. ( uploaded exmaple).

when I move the flat background farther away , more then 1 meater from the wire, ( like when i'm outdoors or in a hallway ) , the camera is unable to determine depth data for the same object at 90cm and closer. ( uploaded a few examples) .

I'm wondering/asking if this behavior is due to stereo vision limitations, d400 limitations , or if anything can be done/configured to allow me to receive depth data for the wire when there's no background in close proximity to the wire.

wire attached to chair with a wall behind it.

wite attacked to chair, depth data is available for part of the wire that has a background near it , but not for the secion where the background is at a distance of 4m

similar example in a hallway.

thanks, Omer.

[RealSenseCustomerSupport]

---

Hi @omerbrandis, Did you use the default projector power? You may try higher power. Did you try different preset, like high accuracy preset? Could you please attach the left IR image along with the depth image, so that we can understand the scenes better?

Thanks,

[omerbrandis]

Hello RS customer support.

1. i've now tried using the value 360 for laser power. ( default is 150 ), it did not have any visible effect .

2. i've tried the high accuracy preset , it returns significantly less data.

3. below is the image returned by rs-viewer for infra1

please advise, Omer.

[RealSenseCustomerSupport]

---

Hi @omerbrandis,

OK. Please keep using the high density preset. The IR image below does not exactly match the depth image. Did you take the two images at the same scene? Please note in the left image below, there are some areas around the door and on the floor are over exposed. The camera will not generate depth for these areas. please avoid the over expose in the scene. Another suggestion is using the ROI feature to select the metal wire area. You may also play with different post processing to see if they can make any difference.

Thanks,

[omerbrandis]

Hello RS customer support.

1. the uploaded images are not of the exact same scene at the exact same time, but its the same hallway , chair and wire.

2. regarding post processing, there are 4 types , each with its own set of parameters , some of which are "Continuous" in nature, are you suggesting I check all the possible combinations of them ? that's sounds somewhat unfeasible , besides the purpose of this issue/ticket was to avoid this approach to this problem. I'm asking someone at with a true understanding of the internal's/"physics" of the d435 can confirm that the "missing data" is due to the cameras "built in" limitations, or suggest a specific solution.

thanks, Omer.

[RealSenseCustomerSupport]

---

Hi Hi @omerbrandis,

OK. Please try the ROI feature to see if it can improve.

Detecting 2mm diameter wire in 130cm with different backgrounds is pretty challenging for the all depth cameras.

Another suggestion is using D415, since D415 uses 2M depth sensor and D435 use 1M depth sensor. D415 should provide a better performance for this usage.

Thanks,

[omerbrandis]

Hello RS customer support. i've tried:

1. setting roi ( using high density)
2. hand and medium density
3. each of the post processing functions ( using default values for parameters ) none of the above made any difference.

I should also be able to retest using D415 in a few days time.

would it be possible to forward my question to development ?

thanks, Omer.

[RealSenseCustomerSupport]

---

Omer,

A few general comments on your question and situation and a few suggestions:

I believe the primary factors in seeing such a feature are contrast and resolution. Depth settings may be adjusted slightly to improve things but are unlikely to overcome the fundamental resolution and contrast limitations.

Resolution is fairly obvious - a small feature like a 2mm wire at ~1m from the D435 camera corresponds to ~1 pixel, which is close to the limit of what can be detected and I would expect marginal capability at best. However, depth values for such size objects may be obtainable depending on other factors like scene content around the object (e.g., wire) and to some extent depth parameters. Some specific types of post processing may also help as discussed below.

I suspect the reason you see better results with a wall (presumably a uniform surface with little or no natural texture) behind the wire is the improved contrast near the wire. The wire basically stands out more relative to the wall background. If that wall or surface were well textured, you would likely not see the wire as well or at all. You should also find that if the wall is too close to the wire, it will start to blend in with the wall and not be distinguishable.

Also, though it may be counter-intuitive, you might try reducing the projector power or possibly turning it off. This would further reduce the background texture and the wire will probably be more easily detected. Of course, this will also degrade the overall depth quality in most cases so may not be a feasible solution. But, you could try it to see the effect.

You are already using the High density preset which has the lowest preset thresholds and will provide the most valid depth along the wire, as you have seen. Lowering the first few of these (2nd peak, neighbor, median) further may provide some additional benefit, but I think more improvement will be seen by reducing the "remove threshold" value in the RSM section of advanced parameters, or possibly turning it off completely (select RSM Bypass). This will come at the expense of more noisy depth over the entire image but should improve the ability to see fine/small objects. If the overall depth quality is not degraded too much for your application, you could try using this setting.

There are 2 types of post processing that also may help but their feasibility will also depend on the requirements of your usage.

1. Under spatial filter, there is a section called "hole filling mode". Selecting an N-pixel radius will fill in depth values around a valid pixel over the specified radius. You can try enabling this and setting to a radius that provides sufficient depth around the wire.
2. Under temporal filter, there is a "persistency mode" that does something similar in the time domain - if a valid pixel is detected, it will persist based on the condition chosen. You can try some of these settings to see how much persistency is needed.

For either of these types of filter, they will only help if there is some depth at all to "fill out". If there is never any valid depth in the wire region, then they will not do much, if anything. Also, as with any filter, it's best to use as little as possible to meet your needs.

Overall, I think the largest improvement in seeing the wire under general conditions will be obtained by adjusting the RSM, as described above, but with the potential side effects mentioned.

The D415 does have higher resolution and is worth trying but it will still be close to its resolution limit and so any improvement may be marginal. All of the other suggestions and considerations above would apply to the D415 as well.

Let us know what you find or if you have any additional questions.

[omerbrandis]

Hello RS customer support.

I have tested your suggestions:

1. changing laser power did not help.
2. decreasing ""remove threshold" produced significant improvement.
3. the suggested temporal and hole filling filters produced marginal improvement.
4. d415 outperforms the d435 for this use case.

our use case is not indoors. ( I discovered the missing data when no background is present in frame when i tested outdoors, then found that same error can be reproduced in doors...).

our real requirement is for use outdoors in apple orchards, I have tested the last suggestions ( rsm, l- power, post processing) outdoors ( not in an actual orchard , thus the scene is not 100% the same ), and can confirm the same results for indoors, but the range was still too short. i.e. in one scene I teststed , using remove threshold = 0, i was able to detect the wire using d435 - good depth data at a range of 50cm , some at 60 , after 70cm very little data d415 - good results for upto 90cm.

considering the fact that we can't use the d415 instead of the d435, would be able to suggest other parameters that may improve our ability to detect the depth of the wire outdoors ? ( you mentioned contrast, should changing the wire's color to red ( against a mostly green and brown background) help ? )

my outdoor scene today

some apple orchard examples

thanks, Omer.

[RealSenseCustomerSupport]

---

Omer,

Thanks for additional info. See comments in line:

_I have tested your suggestions:

1. changing laser power did not help.
2. decreasing ""remove threshold" produced significant improvement.
3. the suggested temporal and hole filling filters produced marginal improvement.
4. d415 outperforms the d435 for this use case._

These observations make sense. In bright lighting, especially outdoors, the projector power will not be a large factor. D415 should be better but the degree of improvement really depends on other factors.

our use case is not indoors. ( I discovered the missing data when no background is present in frame when i tested outdoors, then found that same error can be reproduced in doors...). our real requirement is for use outdoors in apple orchards,

If the usage is outdoors, then the projector aspect is mostly irrelevant, the exception being for very close range and/or low ambient lighting. But, generally you can ignore the projector contribution (in fact, you may be better off with keeping it off).

I have tested the last suggestions ( rsm, l- power, post processing) outdoors ( not in an actual orchard , thus the scene is not 100% the same ), and can confirm the same results for indoors, but the range was still too short. i.e. in one scene I teststed , using remove threshold = 0, i was able to detect the wire using d435 - good depth data at a range of 50cm , some at 60 , after 70cm very little data d415 - good results for upto 90cm.

Ok, this is a reasonable result - D415 a bit better than D435 (by almost 2x), but will still be limited in range due to small feature size.

considering the fact that we can't use the d415 instead of the d435,

Is this due to FOV or some other feature?

would be able to suggest other parameters that may improve our ability to detect the depth of the wire outdoors ? ( you mentioned contrast, should changing the wire's color to red ( against a mostly green and brown background) help ? )

Yes, I believe increasing the contrast relative to background should generally help. But, it looks like you have a large amount of variation in background based on your sample scenes (green leaves, brown limbs, dark shadow regions, light sky,...), so it may be difficult to find a color that will work in all cases. Also, note that the D435 camera is monochrome and so will not distinguish based on color, but only if the color affects the intensity of the image (relative to background). The D415 does use color imagers though.

Do you actually have control over the color or visible properties of the wire (maybe it can be painted for example)? If so, then a few other suggestions:

The ideal object is one with good contrast and texture. If the wire could be coated with a spatially varying reflectivity, such as bright and dark regions, ideally non-periodic (e.g., polka dots), then it would stand out very well against most backgrounds and probably detected over the largest range. Basically a higher resolution version of the projector pattern applied to the wire should work very well. It may not need to be extremely dense in order to work. You could test with applying some dots (either darker or lighter than the main wire) along a section. Or perhaps wire with a striped coating or any varying surface should help.

This may be infeasible in practice, but maybe worth a try to see what improvement could be obtained.

Another option would be to use a specifically colored wire (e.g., red) and place red filters over the cameras. This would increase the visibility of the wire, but of course will block out much of the surroundings. I don't know if you only need to see the wire or the entire scene.

Finally, though it's obvious and probably impractical, increasing the diameter of the wire will clearly help.