ARE INVALID DEPTH VALUES SET TO 0.0?

[ghost]

Hello,

Using Ubuntu 14.04 and OpenGLDepthPacketProcessor to get depth images from kinect2 sensor.

My question is about the obtained depth image and the applied filters inside the drivers:

• I see that depths less than or greater than [mindepth, maxdepth] parameters are set 0.0 -> OK
• We can enable/disable as well bilateral and edge filtering ->OK (by the way, where exactly in the code can I check what is exactly done?)
• What happens with invalid (because of measuring problems or whatever) raw depth values? Are those set to 0.0 as well?

Many thanks.

My conceptual piece of code: listenIrDepth = new libfreenect2::SyncMultiFrameListener(libfreenect2::Frame::Ir | libfreenect2::Frame::Depth); libfreenect2::FrameMap frames; listenIrDepth->waitForNewFrame(frames); libfreenect2::Frame *depth = frames[libfreenect2::Frame::Depth]; cv::Mat(depth->height, depth->width, CV_32FC1, depth->data).copyTo(rawDepth);

[xlz]

https://openkinect.github.io/libfreenect2/classlibfreenect2_1_1Frame.html#a1d1cfd8ffb84e947f82999c682b666a7

unit: millimeter. Non-positive, NaN, and infinity are invalid or missing data.

Their respective sources are quite miscellaneous to explain.

Check filter1.fs and filter2.fs in https://github.com/OpenKinect/libfreenect2/tree/master/src/shader for the implementation of the two filters.

[xlz]

I guess this answers your question.

[ghost]

Hello, thanks for your reply, although it only answers my question partially.

After looking the code you suggested, I suspect that all invalid, NaN or infinity are just set to 0 value. I checked out code for cpu_depth_packet_processor.cpp file (assuming it does the same than in the GL case, since I can not understand the GL code). I understood the main functions are:

• For each pixel of image call "ProcessPixelStage1" function (line 459 in cpu_depth_packet_processor.cpp)
• If bilateral filter is enabled, call "FilterPixelStage1" (line 486)
• If edge_filter is enabled call "ProcessPixelStage2" (line 578) and "FilterPixelStage2" (line 725), otherwise just call "ProcessPixelStage2" with different pre-processing

Is in "ProcessPixelStage2" where the depth value seems to be computed. And precisely there, there is a comment on the code telling (line 586) "if m9 is positive or pixel is invalid (zmultiplier) we set it to 0 otherwise to its absolute value O.o" In fact when depth value is set (in lines 691 to 710) seems to be so, that everything that went bad is set to 0.

I performed a check with my camera and code, obtaining images with and without filters (I mean, bilateral and edge filters) and never obtained an invalid, NaN or infinity, just 0 values where there are troubles.

My main interest would be to separate those values that have been just cut to "max_depth" from those that are invalid for a posterior processing.

Many thanks

[xlz]

You could set max_depth to infinity and threshold it yourself.

[ghost]

Yes, this is what I'm trying just now, set MaxDepth to a value bigger (8m) than the distance in my current room (max 6m). In this way I assume the obtained "0" values are those non-valid; but still the driver does not return negative or NaN values, just "0" values for invalid data.

[xlz]

Nan, negative, and zero are values that you should not use - the meaning of "invalid". The driver does not return negative or NaN values, so what's the problem? I don't see any problem with this.

[ghost]

The problem is in the documentation itself may be.

If I read: https://openkinect.github.io/libfreenect2/classlibfreenect2_1_1Frame.html#a1d1cfd8ffb84e947f82999c682b666a7

I assume I'll have invalid values labelled as Nan, Infinite or negative in my depth image.

But the driver, as you say, just return 0 values, with the added problem that any good value > maxdepth is also set to 0. So the only way to separate invalid values from good ones is setting maxdepth to infinity. Note tat the concept "invalid measure" is not the same as "measured_depth > maxdepth"

After our conversation here is now clear to me, but I think this is not clear at all in the documentation, and clearing it out could help as well others.

Many thanks for your fast answers and help

[xlz]

I tried but I don't understand your problem. Why do you want to "separate invalid values from good ones"? Invalid just means "do not use". It has no physical meaning. You can't derive anything from the fact that some data is invalid.

The default, data > 4.5 meter is marked as invalid for a reason: the accuracy is reduced under the nominal level beyond 4.5 meter making it effectively unusable.

The filters mark some data as invalid for good reason as some papers show those data greatly reduces accuracy in some algorithms.

And for whatever reason division by zero or infinity can happen in the GPU code without any errors.

There are many sources that can make the depth value outside of the normal range, thus invalid.

[ghost]

I agree, I can not use invalid values, but this is the point to me: for me it is different "invalid" because whatever reason, than just simply because measured depth is bigger than maxdepth. And the filter labels them the same. And in a posterior use of depth images, the knowledge you get with two concepts is bigger than just using one.

I read out the work by Felix Järemo, his MsThesis, and he explains the process of measurement in the kinect hardware pretty well; if I understoodwell, the limit of 4.5m is not hardware but just software, depending on the way the phase is used to get a depth value. He even proposes some changes to libfreenect2, I do not know if the developing team are aware of this work. Here is the link to it: liu.diva-portal.org/smash/get/diva2:854680/FULLTEXT01.pdf

Anyway, my conversation with you was fruitful; at least to me, since I cleared out my doubts. We can leave it at this point, I do not pretend to take you time away. So, again thanks for your help.

[thinever]

Hello,i want to ask a fool quesion,why enable both bilateral and edge filtering ,i can not get the depth data?The window shows blank, And i can only enable bilateral filter. Wait for your reply!