search

diablodale / dp.kinect2

Kinect v2 (XBox One) plugin for Cycling '74 Max
21 stars 4 forks source link

Align color to depth causes a shadowed color image #3

Closed gavspav closed 9 years ago

gavspav commented 9 years ago

I have a Desktop PC running 8.1 64 bit. GTX 860, i5 4690K, 8GB Ram, Z87X D3H Max 6.1.8 32bit running the month demo

Just using the dpkinect2 demo/help patch, if I select align 1.color to depth, I get a weird ghosting of the color image. This is something to do with the inaccuracy of the alignment close to the camera but I don't really get what is happening here.

diablodale commented 9 years ago

I don't know. I can't see what you see. Perhaps a screenshot can better describe your issue.

diablodale commented 9 years ago

I suspect you are seeing expected behavior. The below images where you see two hands is expected behavior of both Kinect v1 and v2 sensors. Lots of debate over the years with Microsoft. It has to do with the fact that the rgb and ir cameras are not in the same physical location. So there is an offset. Which can lead to ghosting or unseeable areas which then affects visuals when you use alignment algorithms. This artifact is more visible when closer to the camera than further away. A few quotes from Microsoft:

gavspav commented 9 years ago

This is what I am seeing. If you look at the hand on the right you see there is a doubling of the finger and arm.

So when align to depth is not selected, the colormap matrix is just a feed from the rgb camera. When it is selected there is some processing going on - resizing and some kind of overlay of the black areas. The black areas confused me - I thought that the rgb image was mixed with the depth map but it was only the black areas that showed through. It would be good to know what is going on here - are you using the sdk/api to map the image or your own algorithm?

.[image: Inline image 1]

Gavin

On Fri, Oct 24, 2014 at 8:04 PM, Dale Phurrough notifications@github.com wrote:

I suspect you are seeing expected behavior. The below images where you see two hands is expected behavior of both Kinect v1 and v2 sensors. Lots of debate over the years with Microsoft. It has to do with the fact that the rgb and ir cameras are not in the same physical location. So there is an offset. Which can lead to ghosting or unseeable areas which then affects visuals when you use alignment algorithms. This artifact is more visible when closer to the camera than further away. A few quotes from Microsoft:

  • That is to be expected. See the coordinate mapping basics sample. When mapping depth to color, the depth point will not be a 1:1 mapping as the edge can fall between different depth planes around edges. With people it is easier to filter out with the body index frame.
  • that is the correct behavior for mapping depth to color. It is not a 1:1 mapping because of the varying levels of depth and field of view. You may be able to visualize it if you generate the point cloud and align the view with a ray cast in depth space. [image: doublehand] https://cloud.githubusercontent.com/assets/679350/4774657/6c9004a4-5baf-11e4-9d88-d728808ff5cc.png

— Reply to this email directly or view it on GitHub https://github.com/diablodale/dp.kinect2/issues/3#issuecomment-60434251.

Gavin www.digitalfunfair.co.uk

diablodale commented 9 years ago

I can't see your image; only text. Yes, I use the SDK API for the mapping. The colormap outlet always outputs RGB or YUV data and it is optionally processed depending on attributes (e.g. @align). There is no overlaying, resizing, etc. of the colormap. However, if you want to align two frames (depth and color) then they must have the same output res. Then, one pixel (e.g. 124, 87) is the same "data" in both frames. Reference what I wrong above about different viewpoints leading to obscuring or unknown matching depth/color. This is a topic to read, digest, and think through yourself. Eventually, you will grok it. This is normal usual learning topic for sensors like the Kinect.

I recommend referencing google, max forums, and the Microsoft Kinect documentation. Examples include: http://msdn.microsoft.com/en-us/library/dn785530.aspx https://github.com/diablodale/dp.kinect/wiki/Attributes http://stackoverflow.com/questions/10391719/kinect-depth-and-image-frames-alignment https://www.google.de/search?q=kinect+depth+color+align+double

gavspav commented 9 years ago

Thanks for that information. I do understand that rgb and depth will never match totally because they are in different physical locations. I was just expecting the colormap image (when aligned) to simply be a remap of the image from the rgb camera whose matrix would line up with the depth map image as best as possible. The overlay of the shadows and the doubling which you can hopefully see in the image linked below confused me. https://www.dropbox.com/s/bp1t39n3ih4ukkt/Screenshot%20%283%29.png?dl=0

On Mon, Oct 27, 2014 at 11:43 AM, Dale Phurrough notifications@github.com wrote:

I can't see your image; only text. Yes, I use the SDK API for the mapping. The colormap outlet always outputs RGB or YUV data and it is optionally processed depending on attributes (e.g. @align https://github.com/align ). There is no overlaying, resizing, etc. of the colormap. However, if you want to align two frames (depth and color) then they must have the same output res. Then, one pixel (e.g. 124, 87) is the same "data" in both frames. Reference what I wrong above about different viewpoints leading to obscuring or unknown matching depth/color. This is a topic to read, digest, and think through yourself. Eventually, you will grok it. This is normal usual learning topic for sensors like the Kinect.

I recommend referencing google, max forums, and the Microsoft Kinect documentation. Examples include: http://msdn.microsoft.com/en-us/library/dn785530.aspx https://github.com/diablodale/dp.kinect/wiki/Attributes

http://stackoverflow.com/questions/10391719/kinect-depth-and-image-frames-alignment https://www.google.de/search?q=kinect+depth+color+align+double

— Reply to this email directly or view it on GitHub https://github.com/diablodale/dp.kinect2/issues/3#issuecomment-60580572.

Gavin www.digitalfunfair.co.uk

diablodale commented 9 years ago

Its not perfect due to technology/methodology and physics. The end use/project will guide you on what additional processing of the output you might want. Perhaps you want to subtract background > 4m, isolate bodies using the playermap/bodyindex, greenscreen, etc. Techniques like these and more might be what you want.

lwoodbury commented 9 years ago

I was going to report this, but it looks like you are saying this is expected functionality. I just wanted to ask why this isn't an issue with dp.kinect? I understand that the resolutions match as one is half the other, but they are still out of line aren't they? How come the aligned colour image does not have these black edge and double image artefacts?

with align

diablodale commented 9 years ago

It is a behavior also with the Kinect v1 (all apps). In the color picture above you attach I do see the same problems. There are unmappable (usually rendered as black) pixels in bulk on the left and top and bottom. There are unmappable black pixels along the edges of objects like the file cabinet, vents/ducts above, etc. If you were to put your hadn extremely close to the Kinect v1 (and off center for most affect) you will see a duplicate of the topmost Kinectv2 image. The color and depth cameras can not occupy the same physical space, therefore they will not have the same viewpoint and therefore can not always map data equally. There isn't anything wrong. Its expected and part of the process/method inherit in current depth/color camera technologies.

lwoodbury commented 9 years ago

Apologies, you are right, it does happen to some extent with dp.kinect1 as well (the previous screenshot was v2, this one is v1). It doesn't seem to be very apparent around the user like it is with v2 and the double image thing doesn't happen. In the v2 one you can see the doubling on my back and the back of my head whereas I can't pick out anything like that with the v1 image, but there are still some black lines around background equipment which doesn't show up too well in this picture.

dp kinect1

diablodale commented 9 years ago

Attached here is a screenshot of dp.kinect using the original Kinect. On the left you can see the two outlines of my hand in the depthmap and on the right the duplicated rgb image of my hand. This is expected behavior and matches much of what you see in the Kinect v2. k1

lwoodbury commented 9 years ago

OK, I guess in my current use (green screening) v1 has been a lot more forgiving and I didn't even notice it as an issue, but with v2 it has become very obvious and there are thick black lines all around the user and some obvious doubling. However, I understand that you are saying this is not an issue with your implementation and something inherent in the way the sensor works. Thanks for the help.

mrmaarten commented 9 years ago

Hi, I would to chime in: with dp.kinect1 and the beta of dp.kinect2 I have had no issue aligning the depthmap (filtered on distance and with 100% contrast) to the colormap and with a jit.pack I could create a good key.

example: https://vimeo.com/119710704

I never used @align 1 because this would always create the black in the colormap, as described above.

With the beta (I am not sure I ever used dp.kinect2 1.0) of dp.kinectv2 I would scale the filtered depthmap up to 1304x1080 and I would crop the colormap by 308pixels on either side. jit.pack-ing this together would create a fine key. Unfortunately now not anymore!

Now the depthmap and colormap dont align anymore and when I offset it depends a lot on the distance. I get the explanation, but to my idea it was working pretty well before.

Could you maybe verify that is was working in the beta before release? Thanks!

diablodale commented 9 years ago

It is rarely reliable to do aligning with your method (scale, crop, etc.) that you write. Every Kinect hardware has a different lens and sensor alignment. In addition, the specific pixels which should be aligned change based on distance and angle to the Kinect.

Only the use of @align will create consistent reliable alignment across all Kinects and all scenes. A combination of @align and the playermap (or @depthonlyplayers) can create something similar to a greenscreen affect. A version of dp.kinect2 currently in testing allows for aligning the depth->color image which enables you to retain the higher resolution of the color image.

mrmaarten commented 9 years ago

Yeah I get what you are saying. With dp.kinectv1 I didn't apply scaling or cropping, and it works really well. It isn't perfect for my use good enough (As seen in the video the offset is minimal in a wide variety of distances. I applied a blur to the mask, for aesthetic reasons and that creates even the most offset because of a slight delay in processing.)

And I understand that the Kinectv2 hardware is different from the Kinectv2, but in the last beta it also worked fine (with cropping).But I cannot test it anymore because the beta's don't work anymore.

I just mention it for reference/ data points. But it sounds that your version of aligned depthmap will work well and I am looking very much forward to it.