Open utterances-bot opened 3 years ago
Git clone Barracuda-PoseNet-Tutorial & I followed the tutorial, but keypoints didn't show There are no errors on the console When played, it appears in the hierarchy window but not in the game view Please tell me the solution.
https://gyazo.com/91dc4bf28bc64114af596c41ee71ec74 https://gyazo.com/fe9e6b364f5c0c26ed1ad890027240b3
This is my repository. https://github.com/k0hh2/Posenet-Barracuda
Hi @k0hh2,
From the screen captures you shared, it appears that you are on Mac and are using the Compute Precompiled
GPU backend. Since the key point GameObjects are not toggling on and off in the Hierarchy tab while the video plays, that would suggest that the model is either not receiving proper input or the inference engine is not working properly.
If you check the Stats
in the Game View, does the framerate suggest that the model is actually running?
Even though Unity indicates that GPU inference is supported on Metal, could you try using the C Sharp Burst
CPU backend?
I do not know which Mac you have and do not have any Macs to test the project myself. However, I have heard there are some issues with using the video player on AMD GPUs. Have you tried using the webcam as input?
It is also possible that there are bugs in the preview version of Barracuda used in this tutorial that affects performance on Macs. You could try downgrading the Barracuda package to version 1.3.0
and see if that helps.
Hi! I followed the tutorial, and everything was working perfectly, but when I attempted to build the project (with useGPU set to False) for WebGL (development build) I got this error: Kernel TextureToTensor and TextureToTensor_NHWC are both missing I also attempted to build using CSharp and CSharpBurst as CPU backend, but I still got the same error. In the Barracuda official manual, they say WebGL is a supported platform, using CPU inference. Please, can you tell me if you have any idea of what may be causing this error?
Hi @LuanaPorciuncula!
I just tried building the github project for WebGL using useGPU=False
and CSharpBurst
. The project built without issue, but I am only getting a blank screen once the game starts in the browser. I will be leaving for the weekend shortly (family stuff), but I can take a closer look when I get back next week.
Hi @LuanaPorciuncula! So, it appears that converting from a texture (e.g. a RenderTexture) to a tensor is not supported on WebGL.
It should be possible to get around this by reading the pixel data to an array. You can do this by reading the pixel data from the RenderTexture to a Texture2D (as shown here) and then calling GetRawTextureData() from the Texture2D. Or you could use a for loop.
A tensor can then be initialized from the array of pixel data. Unfortunately, this will impact performance to some degree.
There may also be additional changes needed for playing a video or webcam feed in WebGL.
Hi! I tried the tutorial multiple times but I do not get the skeleton color shown on the videos when I play it. What can be the possible errors I made? Can it be version errors or something?
Hi @Roshen98! As a sanity check, does using the project in the GitHub repository have the same issue for you? Also, are you on a Mac or Windows? I have had reports of Unity related issues on Macs (I only have a Windows machine).
I did not try the Github repository but downloaded Unity on my MacBook Pro. I am sure I did everything right but the skeleton is not showing. Maybe it depends on the computer or system versions.
Hi @Roshen98, It is possible that there is a Mac-specific issue. Could you confirm whether the issue you are encountering looks the same as these screen captures from another Mac user? https://gyazo.com/91dc4bf28bc64114af596c41ee71ec74 https://gyazo.com/fe9e6b364f5c0c26ed1ad890027240b3
If, so try switching the Worker Type to C Sharp Burst
. The other Mac user said the keypoints were visible when using the CPU, but not the GPU. There were also issues with using the webcam on their Mac.
Unfortunately, I don't have a Mac to test the project on, so I cannot verify potential solutions for resolving Mac-specific issues.
Yeah I have the same issue. I will try C Sharp Burst. Thanks for the immediate reply! Appreciate it.
It works! Yeah, the GPU does not work and the worker type needs to be set to C Sharp Burst. Mac IOS version sure has a lot different from Windows version
Yeah, and that is not even considering the new the M1 Macs. I might need to get one at some point for testing purposes, if Unity/Apple don't resolve whatever the root cause is.
I was considering trying out one of the newer M1 cloud instances just to check things out. I was recently sent a new Intel laptop for a different project. Otherwise, the new MacBook Pros would be extremely tempting.
@cj-mills I have a question on how the program determines the number of jumps I did in the videos. What scripts can I write to make that count appear on the screen?
Hi @Roshen98,
The project as it is does not implement any methods for keeping track of physical movement like jumping, so I am guessing you made some modifications for that. As for displaying the count value to the screen, you can create a UI Text or Text - TextMeshPro object in the Hierarchy tab and update the text value in a C# script.
To use a TextMeshPro object in a script, add using TMPro;
at the top of the script. The variable will be of type TextMeshProUGUI
and the text can be updated through the variableName.text
property (e.g. textContent.text = $"Jump Count: {count}";
). example
If you have not yet implemented a method for keeping track of jumps, one simple approach could be to set a starting threshold value for the coords.y
value under which would be considered not jumping and update the counter whenever the value goes above that threshold. The threshold value could either be set manually or initialized based on the first predictions by the model.
This approach might be problematic if the user gets significantly closer or further away from the camera as the model does not track distance from the camera.
A potential workaround for that could be to dynamically adjust the height threshold based on something like the distance between keypoints on screen. The idea there being that the keypoints for say the left and right shoulders will be further apart the closer to the camera the user is.
It is just past 2 AM here, so I'll check if you have any follow up questions after I have gotten some sleep.
How do I just put the box in the video I created? I tried following the example but I feel I do not need that much detector but simply a box with the number of jumps I did.
Hi @Roshen98, Sorry, I might not have been clear. The linked "example" is from a different project that happens to display and update text on screen, so I thought that might be helpful to link. Maybe checkout this tutorial from Unity from Unity on using TextMeshPro or one from YouTube to get an idea of how to update text in the UI for Unity.
Hi @cj-mills! Thanks very much for the tutorial, this has been really helpful. I'm running into the same problem as @LuanaPorciuncula above. I think I've nearly reached a solution, but there appears to be an offset in the skeleton that I haven't been able to solve.
I've added the following to the PoseEstimator.ProcessImage method after the else statement:
// Convert render texture to pixel array
image = RenderTexture.active;
if (tex2D == null) tex2D = new Texture2D(image.width, image.height);
tex2D.ReadPixels(new Rect(0, 0, image.width, image.height), 0, 0);
tex2D.Apply();
// Stores the raw color and pixel data for tex2D
Color[] pixels = tex2D.GetPixels();
float[] floatValues = new float[pixels.Length * 3];
for (int i = 0; i < pixels.Length; i++) {
floatValues[i * 3 + 0] = pixels[i].r;
floatValues[i * 3 + 1] = pixels[i].g;
floatValues[i * 3 + 2] = pixels[i].b;
}
// Initialize the Tensor from the pixel data array
input = new Tensor(1, image.height, image.width, 3, floatValues);
// Download the tensor data to an array
float[] tensor_array = input.data.Download(input.shape);
// Apply preprocessing steps
preProcessFunction(tensor_array);
// Update input tensor with new color data
input = new Tensor(input.shape.batch,
input.shape.height,
input.shape.width,
input.shape.channels,
tensor_array);
and an #endif after the original code. Any ideas on what I might be able to do to address the offset I'm seeing? Thanks!
Hi @aradicaldreamer! I was just about to head to bed, so I will take a closer look tomorrow. However, I would recommend checking out the recent branch I added to the GitHub repository. It uses the latet version of the Barracuda library which added support for creating tensors without compute shaders as well as a Pixel Shader worker type that allows models to run on the GPU without compute shaders.
I made a quick WebGL build with these settings and everything seemed to run fine in the browser (tested on Firefox). Although, you would need to add a delay for the webcam initialization to make sure the resolution is properly set.
Wonderful! Thank you very much for getting back to me so quickly. I'll give this a try
Hey guys, I'm still struggling to get my WebGL build to work, it builds using the settings above, but I still get a blank screen. @aradicaldreamer! would you be so kind as to post your script changes again in full. I think I've missed in your implementation. For info I'm trying to get this working as a project for Alderhey hospital here in the UK to try and combat childhood obesity. Any help greatly appreciated.
Hey @Aspeers1,
I made a new branch with a quick-and-dirty WebGL version. I only included the MobileNet version of the model for now. I still need to tweak some settings to get the best results. I will clean it up and make a dedicated post when I have some free time.
Note: You might get some complaints from Unity the first time you try to build the project. Just click build and run again and it should finish building the second time.
You can check out a live demo at this link. You will need to allow webcam access to use it. If you don't see a prompt asking for webcam access, try refreshing the page.
Hey @cj-mills, Thank you so much for taking to time to help. I will lookout for further updates with interest. But this is brilliant! And the best tutorial I've seen for implementing this kind of functionality in Unity. All the best Adam
Hi, I followed the tutorial correctly, but I'm having some problems:
1. Shows this warning _Unexpected timestamp values detected. This can occur in H.264 videos not encoded with the baseline profile. Timestamps will be skewed to correct the playback for C:/../../../.../.../Unity-Barracuda/Assets/Videos/pexelsboardslides.mp4
2. When I play the code the PoseSkeleton isn't working, but the gameObject body parts' positions appear in the Scene, I'm suspecting being functions hierarchy, could be it?
3. The videos aren't appearing on screen too
I'm using Windows 10, GTX 1650Ti
Hi @Msajt, Thanks for providing your computer details. The project should work on that system.
The warning about the video timestamp values is expected and should not impact video playback.
Out of curiosity, do you encounter the same issue with the project in the GitHub repository?
@cj-mills The example in your repository worked fine for me. I really don't know why the skeleton points aren't appearing, the gameObjects position appear in the right position but the balls and lines are ghosts. I used the Unity 2020.3.31f1 doing the barracuda example, could be it too?
I used posenet and unity recently, but I made with posenet.js sending messages to run unity webgl functions
@Msajt Hmm. I tested the example project with Unity 2020.3.31f1 and it worked as expected. It is difficult to say what the exact issue might be without having a look at your code or a screenshot. If the gameObjects are tracking the 2D body points, but you can't view them or the lines, perhaps the z-axis positions are incorrect?
I used posenet and unity recently, but I made with posenet.js sending messages to run unity webgl functions
Cool! I'll have to check that out. I was planning on exploring something similar but with TensorFlow.js for a future a tutorial given the interest in WebGL builds.
@cj-mills It was the z-index, I saw the game running on Scene Mode and the skeleton was there, thank you so much to enlight this. I have some examples using ml5.posenet + Unity in my repo if you want to check out
@cj-mills Hi, I'm very interested in your tutorial. I'm currently reasearching about pose estimator and I got your post in my search results. I followed your step and successfully with 1st edition. Since then, I began to work on your 2nd edition. But I came up with a problem. My goal is create an app that can compare 2 parallel video at the same time and put the score on the screen. I can make with the 1st edition, but with 2nd, somehow, the skeleton was magnified and its positions weren't right. I tried to modified the original positions of keypoints but still not working.
Here is my issue.
https://github.com/phungh67/PoseEstimator4th/blob/main/Issue.png
Hi @phungh67! I cannot be 100% sure without inspecting the modified project. However, I suspect the issue stems from the pose estimator script basing the final key point object positions on the origin coordinates and the absolute resolution of the source video. You might need to change that to account for the relative position and size of the video screen object in the scene. Maybe treat the bottom left corner of each video screen as the origin and offset the predicted pose coordinates based on the bottom left corner coordinates? I can't tell from the screenshot, but maybe the dimensions of the video screen objects do not match the resolutions of their respective videos?
Hi @cj-mills, thank you for your replying. I have figured out my problem. The reason is my dimensions of video screen and the "scale" in ProcessOutPut method. Now I will move to next stage. Thank you for your tutorial, without your help, maybe it will take months to research and develop form scratch.
@phungh67 Glad you found a solution!
Barracuda PoseNet Tutorial 2nd Edition Pt. 7 | Christian Mills
This post covers how to create pose skeletons and manipulate them using output from the model.
https://christianjmills.com/Barracuda-PoseNet-Tutorial-V2-7/