Kinect v2: shows only blank screen

[Maghoumi]

I've been having this problem for a while. InfiniTAM compiles and executes, however when using Kinect v2, the screen is blank. The framerate keeps updating, but nothing is shown or constructed... This is what I get.

I've tried two different machines, but both exhibit the same behavior. Interesting thing is, Kinect v1 works just fine (reconstruction is shown, etc.)

Any help is greatly appreciated :)

[Maghoumi]

Also, I've compiled the library with Kinect v2 support (undefined COMPILE_WITHOUT_Kinect2).

I noticed that in Kinect2Engine.cpp, all the important stuff are commented out:

void Kinect2Engine::getImages(ITMUChar4Image *rgbImage, ITMShortImage *rawDepthImage)
{
    //Vector4u *rgb = rgbImage->GetData(MEMORYDEVICE_CPU);
    //if (colorAvailable)
    //{
    //}
    //else memset(rgb, 0, rgbImage->dataSize * sizeof(Vector4u));

    //float *depth = out->depth->GetData(MEMORYDEVICE_CPU);
    //if (depthAvailable)
    //{
    //  IDepthFrame* pDepthFrame = NULL;
    //  UINT16 *pBuffer = NULL;
    //  UINT nBufferSize = 0;

    //  HRESULT hr = data->depthFrameReader->AcquireLatestFrame(&pDepthFrame);

    //  if (SUCCEEDED(hr))
    //  {
    //      if (SUCCEEDED(hr))
    //          hr = pDepthFrame->AccessUnderlyingBuffer(&nBufferSize, &pBuffer);

    //      if (SUCCEEDED(hr))
    //      {
    //          for (int i = 0; i < imageSize_d.x * imageSize_d.y; i++)
    //          {
    //              ushort depthPix = pBuffer[i];
    //              depth[i] = depthPix == 0 ? -1.0f : (float)depthPix / 1000.0f;
    //          }
    //      }
    //  }

    //  SafeRelease(pDepthFrame);
    //}
    //else memset(depth, 0, out->depth->dataSize * sizeof(short));

    //out->inputImageType = ITMView::InfiniTAM_FLOAT_DEPTH_IMAGE;

    return /*true*/;
}

What's going on here?

[Maghoumi]

OK, I managed to get something out of it by modifying the function getImages a bit. Now I do get reconstruction with Kinect v2. However, it seems that tracking is not very robust. I imagine that's due to the fact that in the function (in its current form) there is no color information assigned to rgbImage.

Here's what I have right now:

void Kinect2Engine::getImages(ITMUChar4Image *rgbImage, ITMShortImage *rawDepthImage)
{
    // Set all color stuff to 0;
    Vector4u *rgb = rgbImage->GetData(MEMORYDEVICE_CPU);
    memset(rgb, 0, rgbImage->dataSize * sizeof(Vector4u));

    short *depth = rawDepthImage->GetData(MEMORYDEVICE_CPU);
    if (depthAvailable)
    {
        IDepthFrame* pDepthFrame = NULL;
        UINT16 *pBuffer = NULL;
        UINT nBufferSize = 0;

        HRESULT hr = data->depthFrameReader->AcquireLatestFrame(&pDepthFrame);

        if (SUCCEEDED(hr))
        {
            hr = pDepthFrame->AccessUnderlyingBuffer(&nBufferSize, &pBuffer);

            for (int i = 0; i < imageSize_d.x * imageSize_d.y; i++)
            {
                ushort depthPix = pBuffer[i];
                depth[i] = depthPix == 0 ? -1 : depthPix;
            }
        }

        SafeRelease(pDepthFrame);
    }
    else memset(depth, 0, rawDepthImage->dataSize * sizeof(short));

    return;
}

Any help from the silent population is greatly welcome :D

[Maghoumi]

Nope... I fixed the above code, it now writes pixel colors to rgbImage as well but still tracking is terrible. Even small movements confuses the system. I'm thinking maybe this could be a calibration problem... Maybe wrong calibration data is loaded (if any data loaded for Kinect v2 at all).

This might also provide more background information: I tried the newly compiled InfiniTAM library with SemanticPaint. With Kinect v1, everything works the way it is supposed to. With the fixed Kinect v2 code, SemanticPaint works, but I'm seeing an "offset" between the mouse cursor location and the one rendered on the OpenGL window (look at the image below). I initially thought this could be due to wrong calibration. But I could be wrong (sorry, super new to InfiniTAM and SemanticPaint).

By the way, will submit a pull request with the modifications I made so that others could also use Kinect v2 with InfiniTAM. Just have to clean the code up a bit :)

[sgolodetz]

The offset in SemanticPaint is likely to be a calibration issue, as you suggest. If you take a look in

https://github.com/torrvision/spaint/blob/master/apps/spaintgui/main.cpp

you'll observe that SemanticPaint is currently hard-coded to use the internal calibration of the camera. You could try turning off internal calibration and loading a proper calibration file for the Kinect v2. I haven't tried SemanticPaint with a Kinect v2 (I don't have one), so I don't have a calibration file for it, but @victorprad or @olafkaehler might. I'll ask them tomorrow.

[Maghoumi]

Thank you for your response. Does that possibly explain the tracking problem? I'm not sure whether TAM tracks using depth or RGB data. The thing is, moving the sensor in any direction by anything more than 10 inches will mess tracking and the reconstruction. No matter how small the movement is.

[olafkaehler]

• The default tracking in InfiniTAM on its own is set up to use the depth image only. Look at ITMLibSettings to change this, but by default, RGB images are completely ignored, as is the calibration of RGB to depth images. This is not the case for Semantic Paint, where colour is accumulated by default, but it's still not used for tracking.
• The Kinect v2 has a different resolution for its depth images, so it would almost certainly also require different intrinsic parameters for the depth camera. I once calibrated some values like fx 358.435, fy 357.912, cx 256.526, cy 205.474, but they may be arbitrarily wrong for your Kinect v2.
• Apart from that, the Kinect v2 has a time of flight camera rather than a structured light sensor. Due to imperfections in the modulation of the active illumination, it is known that most of these time of flight sensors tend to have biased depth values, e.g. at a distance of 2m, everything is about 5cm closer than measured, at a distance of 3m everything is about 5cm further away than measured. I've never investigated how good the Kinect v2 actually is and the given numbers are just random examples. In practice, we have noticed a few 3D geometry artefacts due to this effect, though, and since this is an inherent property of time of flight sensors, we have largely ignored the Kinect v2 so far. Ideally you'd calibrate the biases for your specific camera and such a calibration should then be used by InfiniTAM, but most of the time we just stick with ASUS, PrimeSense, Occipital StructureSensor or other structured light devices that do not have this problem.

[Maghoumi]

@olafkaehler Thank you for your response. I tried your calibration values as well, but the reconstruction still works terribly (I think tracking is lost too quickly). Yes the depth camera's resolution is 512 x 424. Also, I can't be certain the issue you mentioned with ToF camera is the evil here. Say there is a cup on the table. If the sensor is stationary, nothing happens and the model looks fine. As soon as the sensor is rotated a bit so that the cup is translated in the image a bit, the system thinks it's seeing a new cup and will start "desolving" the previous one and will create a new one.

[victorprad]

It really was not so bad with the kinect 2 when I tested (version 1 on InfiniTAM). Reconstruction worked overall fine. The only problem was that the reflective areas were not considered noise, but rather developed very weird artefacts. Maybe have a look at the depth calibration. Make sure the appropriate conversion is done -- I think the kinect 2 just needs a division by 1000, whereas for the primesense we used calibration. I've got a very bad flu right now, but I will recheck the kinect 2 version by the end of the week, when I get back to work.

On Tuesday, 8 December 2015, Mehran Maghoumi notifications@github.com wrote:

@olafkaehler https://github.com/olafkaehler Thank you for your response. I tried your calibration values as well, but the reconstruction still works terribly (I think tracking is lost too quickly). Yes the depth camera's resolution is 512 x 424. Also, I can't be certain the issue you mentioned with ToF camera is the evil here. Say there is a cup on the table. If the sensor is stationary, nothing happens and the model looks fine. As soon as the sensor is rotated a bit so that the cup is translated in the image a bit, the system thinks it's seeing a new cup and will start "desolving" the previous one and will create a new one.

— Reply to this email directly or view it on GitHub https://github.com/victorprad/InfiniTAM/issues/33#issuecomment-162725537 .

[This email was sent from a wireless mobile device]

[Maghoumi]

@victorprad That division by 1000 was there (in the commented out code) but they whole thing only worked after I got rid of it. I inspected the values, and they look very similar to the ones obtained by OpenNI from Kinect 1. Would be cool if we could get it to work. Hope the flu goes away soon and you feel better.

[vestri]

Hello, I am trying to use Infinitam with KinectV2 on window 8.1. What is the right way to compile it? There is no flag concerning Kinect in Cmake while there is one for REal sense. Do I have to undefine COMPILE_WITHOUT_Kinect2 and add path manually? Thanks

[Masterxilo]

@vestri Yes, undefine COMPILE_WITHOUT_Kinect2 and make sure the kinect 2 api is in the path. Also add the adjustments the OP mentioned.