Playback function

[fberna]

Hi, I am trying to use properly the playback function in order to have a "capture variable". From this variable, I will extract color and vertice coordinates. My idea was just to reverse the method of recording, but it seems not to work. Do you have any suggestions? Thank you FB

 private async Task KinectLoop()
    {
        while (true)
        {
            using (playback = await Task.Run(() => new Playback(mkvPath)).ConfigureAwait(true))
            {
                {
                    //Initialize Kinect Configuration
                    playback.GetRecordConfiguration(out var recordConfig);
                    playback.GetCalibration(out calibration);

                    config = new DeviceConfiguration
                    {
                        CameraFps = recordConfig.CameraFps,
                        ColorFormat = recordConfig.ColorFormat,
                        ColorResolution = recordConfig.ColorResolution,
                        DepthMode = recordConfig.DepthMode,
                    };

                    playback.TryGetNextCapture(out Capture capture);
                   }
            }
       }
    }

[baSSiLL]

Try this code

using (var playback = new Playback(mkvPath))
{
    playback.GetCalibration(out var calibration);
    playback.GetRecordConfiguration(out var config);
    while (playback.TryGetNextCapture(out var capture))
    {
        using (capture)
        {
            // use data from the capture
        }
    }
}

[fberna]

I tried with:

 private void Test()
    {
        using (var playback = new Playback(mkvPath))
        {
            Debug.Log(mkvPath);
            playback.GetCalibration(out var calibration);
            playback.GetRecordConfiguration(out var config);
            while (playback.TryGetNextCapture(out var capture))
            {
                using (capture)
                {
                    Debug.Log(capture.DepthImage.WidthPixels);
                 }
            }
         }
    }

It gives me: "NullReferenceException: Object reference not set to an instance of an object." The mkv path seems to be correct. There is a way to check if the capture is really open?

[baSSiLL]

There is a way to check if the capture is really open?

When TryGetNextCapture returns true then the output argument should contain a valid Capture object. What exact object is null in your case? Probably MKV was recorded without depth and capture.DepthImage is null? What is the value of config.DepthMode property?

[fberna]

You are right, TryGetNextCapture is correctly creating a valid Capture object. I checked into MKV recorded are present 3 tracks (depth, IR, color).

Regarding the config.DepthMode: NarrowViewUnbinned.

I am debugging that's why I inserted Debug.Log(capture.DepthImage.WidthPixels); In reality, it looks more like this:

while (playback.TryGetNextCapture(out var capture))
            {
                using (capture)
                {
                    Debug.Log(capture.ColorImage.WidthPixels);
                    //Getting color information
                    byte[] colorArrayBuffer = new byte[capture.DepthImage.WidthPixels * capture.DepthImage.HeightPixels * 4];
                    using (var transformation = calibration.CreateTransformation())
                    {
                        using (var colorArrayImage = Image.CreateFromArray(colorArrayBuffer, ImageFormat.ColorBgra32, capture.DepthImage.WidthPixels, capture.DepthImage.HeightPixels))
                        {
                            transformation.ColorImageToDepthCamera(capture.DepthImage, capture.ColorImage, colorArrayImage);
                        }
                    }

[baSSiLL]

So on which line the NullReferenceException is thrown and what's in its call stack?

[fberna]

For exemple: Debug.Log(capture.ColorImage.WidthPixels);

also if I try: Debug.Log(capture.ColorImage);

it gives me Null

[baSSiLL]

Here's an excerpt from the description of TryGetNextCapture method

Capture objects returned by the playback API will always contain at least one image, but may have images missing if frames were dropped in the original recording. When calling K4AdotNet.Sensor.Capture.ColorImage, K4AdotNet.Sensor.Capture.DepthImage, or K4AdotNet.Sensor.Capture.IRImage, the image should be checked for null.

Thus you just need to add a check that all images needed by your code are present in a Capture object. If not, you can either skip that capture altogether or take missing images from the previous capture.

[fberna]

I tried to check as you suggest but it seems like it is always checking the same frame. After disposing of capture the increment should be automatic, right?

while (playback.TryGetNextCapture(out Capture capture))
            {
                using (capture)
                {
                    Debug.Log(capture.IsDisposed);

                    if (capture.DepthImage == null && capture.ColorImage == null && capture.IRImage == null)
                    {
                        Debug.Log("frame missing");

                    }
                    else
                    {

                        Debug.Log(capture.ColorImage.Format);
                    }

                    capture.Dispose();
                  }
             }

[baSSiLL]

TryGetNextCapture advances to the next frame in a video file each time you call it. I think you've got your check wrong. You check if ALL images are missing at once, otherwise access a color image. But your check will also pass (to the else clause) if only a color image is missing. Thus you'll get an exception in this case. So when you use a color image check if a color image is present.

[fberna]

Yes, I agree with you. It was more to check if any of the images wasn't missing.

private void Test()
    {
        using (Playback playback = new Playback(mkvPath))
        {
            // Get information from video file
            playback.GetCalibration(out Calibration calibration);
            playback.GetRecordConfiguration(out RecordConfiguration config);

            // Display some information on console
            Debug.Log(playback.FilePath);
            Debug.Log(config.CameraFps);
            Debug.Log(config.ColorResolution);
            Debug.Log("ColorCameraFrames=" + config.ColorTrackEnabled);

            // Call function to extract frame by frame
            while (playback.TryGetNextCapture(out capture) == true)
            { 
                Debug.Log(capture.IsDisposed);

                if (capture.ColorImage != null)
                    {
                    Debug.Log(capture.ColorImage.Format);   
                    }
                    else
                    {
                    Debug.Log("color frame missing");
                    }

                if (capture.DepthImage != null)
                {
                    Debug.Log(capture.DepthImage.Format);
                }
                else
                {
                    Debug.Log("depth frame missing");
                }
                capture.Dispose();
                Debug.Log(capture.IsDisposed);
           }
      }
}

The result doesn't change. I will have all the frames that are missing. I tried also different videos. Probably, I do something wrong with the sequence...

[fberna]

In editor log I receive these error for every frames:

[error] [t=6076] D:\a\1\s\extern\Azure-Kinect-Sensor-SDK\src\allocator\allocator.c (118): k4a_capture_t_get_context(). Invalid k4a_capture_t 0000019DBDB52140
[error] [t=6076] D:\a\1\s\extern\Azure-Kinect-Sensor-SDK\src\allocator\allocator.c (295): Invalid argument to capture_dec_ref(). capture_handle (0000019DBDB52140) is not a valid handle of type k4a_capture_t

[fberna]

@baSSiLL do you have any suggestions?