Barracuda PoseNet Tutorial Pt. 4 | Christian Mills

[utterances-bot]

Barracuda PoseNet Tutorial Pt. 4 | Christian Mills

This post covers how to process the output of the PoseNet model.

https://christianjmills.com/Barracuda-PoseNet-Tutorial-4/

[JoeProgram]

Hello! In this section of the tutorial, you mention that reading values from the GPU forces the rest of the model to wait.

After some digging, I found Unity Barracuda documentation for allowing the model to process over several frames.

I was able to get this working with your project by:

1. Switching the code in the Update() function to a Coroutine that's wrapped in an infinite loop
2. Adding a bool hasMoreWork; at the top of the class
3. Swapping out engine.Execute(input) for this function:

void ExecuteInParts(IWorker worker, Tensor I)
    {
        var executor = worker.ExecuteAsync(I);

        do
        {
            hasMoreWork = executor.MoveNext();
        } while (hasMoreWork);
    }

4. Yielding the new Update-Coroutine until the work is done with

   yield return new WaitUntil(() => hasMoreWork == false);

This creates an interesting tradeoff.

The downside is that the skeleton is now behind the image by a frame or two, and isn't updated every single frame, since it isn't being run every single frame.

The upside is that I'm getting +10 FPS on both the "Compute Precompiled" setting, as well as the "C Sharp Burst" setting.

I'll have to get further into my project to know for sure, but my suspicion is the user experience will feel better if the video feed is smoother, even if the skeleton is a little bit late.

Hope this is helpful to others!

[cj-mills]

Hi again! Glad you got the scheduled execution approach working and thanks for sharing your solution here.

I had planned to do another post covering how to use scheduled execution where I would explore that very tradeoff. I had also started working on a workaround where the output of the model was assigned to a RenderTexture.

The pure GPU approach I used in the tutorial was mainly due to me being spoiled with a powerful desktop GPU. The halting operation is actually a significant bottleneck for me. However, scheduled execution is required when the computer isn't powerfull enough to execute the model in a single frame. In those cases, the tradeoff can still be worth it if you are not doing something like controlling a character in real-time.

[Lliu666]

Hello! I followed your instructions to complete the recognition of the video, but the recognition result is not very satisfactory. Your project on github is also running poorly, is it because of the unity version, or is it because of the barracuda version?

[cj-mills]

Hi @Lliu666! First, this project version is outdated, and I recommend following version 2 instead. The second version offers some improvements in both speed and accuracy. Even version 2 is slightly out-of-date, so I recommend checking the project README for more recent updates.

It is difficult for me to diagnose any issues with pose estimation accuracy without more information. However, the pretrained model used in the tutorial is certainly not perfect. For speed improvements, I recommend using the MobileNet version of the model available in the second version of the project.