Recording a video with the Azure Kinect DK failed

[LeebHelmut]

Hello, with psi and the Azure Kinect DK we are recording videos with a Full HD Resolution and a framerate of 30fps.To get the best result for our task we use the depth mode NFOV_2x2Binned. For videos with less than five minutes everything working well. If we are recording videos that are longer than five minutes, the following exception is thrown:

Array dimensions exceeded supported range.

When a video is recorded with 5fps it is possible to record videos over one hour. The Azure Kinect SDK v1.4.1 is used. In the output window of Visual Studio, we often get the following messages:

[2021-03-19 07:57:44.989] [error] [t=59332] D:\a\1\s\extern\Azure-Kinect-Sensor-SDK\src\capturesync\capturesync.c (142): replace_sample(). capturesync_drop, releasing capture early due to full queue TS: 194917733 type:Depth

D pid(43060) tid(50888) 08:05:18 EdgeSnapFeature::PostExitSizeMove: WM_TWINVIEW_SHOW_GRIDLINES -> off

Our use case is to record videos with about two hours.

Snippet of the source code:

public void Record()
        {
            pipeline = CreatePipeline();
            var audio = new AudioCapture(pipeline,
                new AudioCaptureConfiguration
                {
                    OptimizeForSpeech = true,
                    Format = WaveFormat.CreatePcm(settings.Audio.Frequency,
                                                  settings.Audio.Bits,
                                                  settings.Audio.ChannelCount),
                    DeviceName = settings.Audio.MicName,
                });

            var azureKinect = new AzureKinectSensor(pipeline, new AzureKinectSensorConfiguration()
            {
                ColorResolution = settings.Frame.GetKinectResolution(),
                CameraFPS = settings.Frame.GetKinectFPS(),
                BodyTrackerConfiguration = new AzureKinectBodyTrackerConfiguration()
                {
                    TemporalSmoothing = settings.Frame.TemporalSmoothing,
                    CpuOnlyMode = settings.Frame.CpuOnly
                },
                DepthMode = DepthMode.NFOV_2x2Binned
            });
            var ts = new TimeSpan(0, 0, 0, 1, 0);
                var store = PsiStore.Create(pipeline, settings.Name, settings.StoreDirectory);
                var depth = azureKinect.DepthImage;
                var bodies = azureKinect.Bodies;
                var calibrationDeviceInfo = azureKinect.DepthDeviceCalibrationInfo;
                var skeletons = bodies.GetSkeletons(depth, calibrationDeviceInfo);
                azureKinect.ColorImage.Sample(ts).Write("ColorImage", store);
                skeletons.Sample(ts).Write("Skeletons", store);
                audio.Write("Audio", store);
            }
            pipeline.RunAsync(TimeInterval.Infinite, false);
        }

Our use case is to record videos with about two hours.

Any help would be great.

Best regards

Helmut

[chitsaw]

From the exception message, it appears that an internal queue is filling up. I notice that in the call to pipeline.RunAsync, you have specified false for the enforceReplayClock parameter (the second argument). Normally this should be set to true (the default value) if running in live capture mode. What happens if you set this to true?

[LeebHelmut]

Thank you for your advice. Now I have tried records where the enforceReplayClock is true. The same error didn’t occur anymore, but I got the following errors:

The program '[33652] SHCTT.Recorder.exe' has exited with code -1073741819 (0xc0000005) 'Access violation'.
The program '[33652] SHCTT.Recorder.exe: Program Trace' has exited with code 0 (0x0).

I have done four records with the same error. The application was processing about 25 minutes before it crashed.

Recording the video without processing and saving the application was working without an exception.

public void Record()
        {
            pipeline = CreatePipeline();
            var audio = new AudioCapture(pipeline,
                new AudioCaptureConfiguration
                {
                    OptimizeForSpeech = true,
                    Format = WaveFormat.CreatePcm(settings.Audio.Frequency,
                                                  settings.Audio.Bits,
                                                  settings.Audio.ChannelCount),
                    DeviceName = settings.Audio.MicName,
                });

            var azureKinect = new AzureKinectSensor(pipeline, new AzureKinectSensorConfiguration()
            {
                ColorResolution = settings.Frame.GetKinectResolution(),
                CameraFPS = settings.Frame.GetKinectFPS(),
                BodyTrackerConfiguration = new AzureKinectBodyTrackerConfiguration()
                {
                    TemporalSmoothing = settings.Frame.TemporalSmoothing,
                    CpuOnlyMode = settings.Frame.CpuOnly
                },
                DepthMode = DepthMode.NFOV_2x2Binned
            });
            var ts = new TimeSpan(0, 0, 0, 1, 0);
                var store = PsiStore.Create(pipeline, settings.Name, settings.StoreDirectory);
                //var depth = azureKinect.DepthImage;
                //var bodies = azureKinect.Bodies;
                //var calibrationDeviceInfo = azureKinect.DepthDeviceCalibrationInfo;
                //var skeletons = bodies.GetSkeletons(depth, calibrationDeviceInfo);
                //azureKinect.ColorImage.Sample(ts).Write("ColorImage", store);
                //skeletons.Sample(ts).Write("Skeletons", store);
                //audio.Write("Audio", store);
            }
            pipeline.RunAsync(TimeInterval.Infinite, false);
        }

Sometimes the following message was shown in the output window during the recording:

[2021-03-20 09:39:00.446] [error] [t=33704] D:\a\1\s\extern\Azure-Kinect-Sensor-SDK\src\capturesync\capturesync.c (142): replace_sample(). capturesync_drop, releasing capture early due to full queue TS:5481551044 type:Color

Do you know the reason for the errors?

Many thanks in advance

[chitsaw]

That error is from the Azure Kinect sensor SDK, and it might indicate that processing is not keeping up with the color stream. As to the access violation leading to the crash, it is not clear from the information provided what could be the cause. Certainly it could be something that is happening in the processing since you are able to run it without crashing when commented out. What does GetSkeletons() do? It could be something in the handling of the underlying Azure Kinect types that is causing the crash. Perhaps running in debug with more exception tracing might provide more clues.

[LeebHelmut]

We are mostly running in debug, but there is no further information. An exception handling has already been implemented in all places in the code, but none has been thrown. The Application crashes without a caught exception. The GetSkeleton method, which is calculation some needed points, we already eliminated as the cause of the error. We tried a dummy GetSkeleton method and saving the bodies without any processing. However, this has not changed the behavior.

Do you have any other advice? Many thanks in advance

[xiangzhi]

Just some ideas to help you debug/solve this. Have you tried logging the output from Azure? Here's the code that I use when I was debugging it.

        static void Main(string[] args)
        {
            // Start AzureKinect Loggers
            Logger.Initialize();
            Logger.LogMessage += Program.k4aErrorMsg;

            using (var p = Pipeline.Create())
            {
                var store = PsiStore.Create(p, "body", @"C:\Data\body-test");
                var k4a = new AzureKinectSensor(p, new AzureKinectSensorConfiguration()
                {
                    OutputDepth = true,
                    OutputColor = true,
                    BodyTrackerConfiguration = new AzureKinectBodyTrackerConfiguration()
                });

                k4a.Bodies.Write("body", store);
                k4a.ColorImage.EncodeJpeg(50).Write("color", store);
                p.RunAsync();
                Console.ReadLine();
            }
        }

        private static void k4aErrorMsg(LogMessage obj)
        {
            Console.WriteLine(obj);
        }

This should print out all the Azure internal message and help you debug it.

[LeebHelmut]

Hello, thank you for your very helpful advice with the logger. I don’t know why, but since we are using the logger, the problem doesn’t occur anymore 🤔. We recorded videos with about two hours several times without any problem.

Do you know why this behavior is like it is?

[xiangzhi]

@LeebHelmut I'm glad it works!

If the code works again even without the logging code, it is possible that the build system just didn't clean the build files correctly in a previous build. Unfortunately, the body tracking part is close source so it is pretty hard to track down what might have happened internally.

[HaipingLe]

@xiangzhi ， I have enabled the logger for kinect and the same issue can be reproduced.

//Logs **** Device Info **** K4A SDK version: 1.4.1 usb_cmd_io(). XFR: Cmd=00000115, PayloadSize=255 Serial Number: 000798411912 RGB Sensor Version: 1.6.110 Depth Sensor Version:1.6.79 Mic Array Version: 1.6.14 Sensor Config: 6109.7 Build type: Release Signature type: MSFT

---

k4a_image_t_destroy(). Destroyed k4a_image_t 00000281C2254000 k4a_image_t_create(). Created k4a_image_t 0000028261701C00 k4a_image_t_create(). Created k4a_image_t 00000282B058A490 k4a_image_t_create(). Created k4a_image_t 00000282B0589C90 Exception thrown: 'System.FieldAccessException' in Microsoft.Azure.Kinect.BodyTracking.dll

In my program, i use GPU to run the body tracking.

            // Create the Azure Kinect sensor component, and configure it to enable body tracking.
            var azureKinectSensor = new AzureKinectSensor(pipeline, new AzureKinectSensorConfiguration
            {
                ColorResolution = ColorResolution.R720p,
                CameraFPS = FPS.FPS5,
                BodyTrackerConfiguration = new AzureKinectBodyTrackerConfiguration
                {
                    TemporalSmoothing = 0.1f,
                    LiteNetwork = true,
                    //If use GPU, should always set mode to Cuda, not GPU. Otherwise will occur error.
                    ProcessingMode = _configuration.UseGpu ? TrackerProcessingMode.Cuda : TrackerProcessingMode.Cpu,
                },
            });