Is ThreeDPoseUnityBarracuda supports unity 2019.4.1f1?

[Kasidit0052]

Hello, I have cloned your repository, upgrade its to unity 2019.4.1f1 and follow your tutorial. However, Unitychan won’t move and the error log appears.

This is my computer info OS: Mac OS X (10.14.6) Graphic: Intel UHD Graphics 630 1536 MB Unity: 2019.4.1f1

Have you encountered this problem before? thank you

[yukihiko]

It also works with that version of Unity. But I think it's very slow on those specs. I heard that the error is solved by raising the version of Barracuda. I haven't tried it yet.

[isgbuddy]

I try to upgrade Barracuda the latest version. It report the same error.

[yukihiko]

Adds ,3 to the second argument of new Tensor(InitImg) and new Tensor(videoCapture.MainTexture) of WaitLoad and UpdateVNectModel method in VNectBarracudaRunner.cs. So, what do you think? But I don't think it's going to work with those specs.

[alezuky]

Hello. When I add 3 to the second argument, the error changes to:

KeyNotFoundException: The given key was not present in the dictionary. System.Collections.Generic.Dictionary2[TKey,TValue].get_Item (TKey key) (at <437ba245d8404784b9fbab9b439ac908>:0) Unity.Barracuda.PrecompiledComputeOps.PrepareModel (Unity.Barracuda.Model model, System.Collections.Generic.IDictionary2[TKey,TValue] inputShapes) (at Library/PackageCache/com.unity.barracuda@1.0.0/Barracuda/Runtime/Core/Backends/BarracudaPrecompiledCompute.cs:241) Unity.Barracuda.StatsOps.PrepareModel (Unity.Barracuda.Model model, System.Collections.Generic.IDictionary`2[TKey,TValue] inputShapes) (at Library/PackageCache/com.unity.barracuda@1.0.0/Barracuda/Runtime/Core/Backends/StatsOps.cs:44) Unity.Barracuda.GenericWorker+d__29.MoveNext () (at Library/PackageCache/com.unity.barracuda@1.0.0/Barracuda/Runtime/Core/Backends/GenericWorker.cs:166) UnityEngine.SetupCoroutine.InvokeMoveNext (System.Collections.IEnumerator enumerator, System.IntPtr returnValueAddress) (at <480508088aee40cab70818ff164a29d5>:0) UnityEngine.MonoBehaviour:StartCoroutine(String) VNectBarracudaRunner:Start() (at Assets/Scripts/VNectBarracudaRunner.cs:165)

Any idea?

Thanks!

[alezuky]

Just to update: I try to run the same version of the code, same Unity version (2019.3.13f1), same Visual Studio, same .NET, same Barracuda, same CultureInfo Settings, same Windows language settings in 2 different computers and it works in one and throws that error in the other one...

[yukihiko]

Thank you. Similar errors occur on my other PCs but they work fine. I don't know because the specs are low on my Mac.

[alezuky]

Another update - I could reproduce the error at the computer in which the code was working before:

It occured when I tried to generate the joints' coordinates from a video not playing in runtime.

I tried to approach each frame of the video by following the instructions on this issue: https://forum.unity.com/threads/how-to-extract-frames-from-a-video.853687/

Then, instead of using the frames in runtime, I copied the Render Texture of each frame and used it as the base to call the UpdateVNectModel function, from BarracudaRunner script and PoseUpdate, from VNect Model script. My goal is to save the joints' coordinates of each frame into an array. So, instead of approaching each frame through the Unity Update funcion, I am using a "for" loop.

Then, I have got one of this error:

ArgumentException: Off-axis dimensions must match Unity.Barracuda.TensorExtensions.Concat (Unity.Barracuda.TensorShape[] shapes, System.Int32 axis) (at Library/PackageCache/com.unity.barracuda@1.0.0/Barracuda/Runtime/Core/TensorExtensions.cs:229) Unity.Barracuda.ModelAnalyzer.ListTemporaryTensorShapes (Unity.Barracuda.Model model, System.Collections.Generic.IDictionary2[TKey,TValue] inputShapes, System.Collections.Generic.IDictionary2[System.String,System.Nullable1[Unity.Barracuda.TensorShape]]& shapesByName) (at Library/PackageCache/com.unity.barracuda@1.0.0/Barracuda/Runtime/Core/Backends/ModelAnalyzer.cs:373) Unity.Barracuda.PrecompiledComputeOps.PrepareModel (Unity.Barracuda.Model model, System.Collections.Generic.IDictionary2[TKey,TValue] inputShapes) (at Library/PackageCache/com.unity.barracuda@1.0.0/Barracuda/Runtime/Core/Backends/BarracudaPrecompiledCompute.cs:231) Unity.Barracuda.StatsOps.PrepareModel (Unity.Barracuda.Model model, System.Collections.Generic.IDictionary`2[TKey,TValue] inputShapes) (at Library/PackageCache/com.unity.barracuda@1.0.0/Barracuda/Runtime/Core/Backends/StatsOps.cs:44) Unity.Barracuda.GenericWorker+d__29.MoveNext () (at Library/PackageCache/com.unity.barracuda@1.0.0/Barracuda/Runtime/Core/Backends/GenericWorker.cs:166) UnityEngine.SetupCoroutine.InvokeMoveNext (System.Collections.IEnumerator enumerator, System.IntPtr returnValueAddress) (at <480508088aee40cab70818ff164a29d5>:0) UnityEngine.MonoBehaviour:StartCoroutine(IEnumerator) VNectBarracudaRunner:UpdateVNectModel(RenderTexture) (at Assets/Scripts/VNectBarracudaRunner.cs:289) ExtractsPartitions:ExtractPartitions() (at Assets/Developing/Scripts/ExtractsPartitions.cs:106) ExtractsPartitions:Update() (at Assets/Developing/Scripts/ExtractsPartitions.cs:44)

And one of this error for each frame of the video:

AssertionException: Assertion failure. Values are not equal. Expected: 3 == 4 UnityEngine.Assertions.Assert.Fail (System.String message, System.String userMessage) (at <480508088aee40cab70818ff164a29d5>:0) UnityEngine.Assertions.Assert.AreEqual[T] (T expected, T actual, System.String message, System.Collections.Generic.IEqualityComparer`1[T] comparer) (at <480508088aee40cab70818ff164a29d5>:0) UnityEngine.Assertions.Assert.AreEqual[T] (T expected, T actual, System.String message) (at <480508088aee40cab70818ff164a29d5>:0) UnityEngine.Assertions.Assert.AreEqual (System.Int32 expected, System.Int32 actual) (at <480508088aee40cab70818ff164a29d5>:0) Unity.Barracuda.PrecompiledComputeOps.Conv2D (Unity.Barracuda.Tensor X, Unity.Barracuda.Tensor K, Unity.Barracuda.Tensor B, System.Int32[] stride, System.Int32[] pad, Unity.Barracuda.Layer+FusedActivation fusedActivation) (at Library/PackageCache/com.unity.barracuda@1.0.0/Barracuda/Runtime/Core/Backends/BarracudaPrecompiledCompute.cs:625) Unity.Barracuda.StatsOps.Unity.Barracuda.IOps.Conv2D (Unity.Barracuda.Tensor X, Unity.Barracuda.Tensor K, Unity.Barracuda.Tensor B, System.Int32[] stride, System.Int32[] pad, Unity.Barracuda.Layer+FusedActivation fusedActivation) (at Library/PackageCache/com.unity.barracuda@1.0.0/Barracuda/Runtime/Core/Backends/StatsOps.cs:69) Unity.Barracuda.GenericWorker+d__29.MoveNext () (at Library/PackageCache/com.unity.barracuda@1.0.0/Barracuda/Runtime/Core/Backends/GenericWorker.cs:211) UnityEngine.SetupCoroutine.InvokeMoveNext (System.Collections.IEnumerator enumerator, System.IntPtr returnValueAddress) (at <480508088aee40cab70818ff164a29d5>:0) UnityEngine.MonoBehaviour:StartCoroutine(IEnumerator) VNectBarracudaRunner:UpdateVNectModel(RenderTexture) (at Assets/Scripts/VNectBarracudaRunner.cs:289) ExtractsPartitions:ExtractPartitions() (at Assets/Developing/Scripts/ExtractsPartitions.cs:106) ExtractsPartitions:Update() (at Assets/Developing/Scripts/ExtractsPartitions.cs:44)

Hope this can give us some insights about the problem.

[alezuky]

Hello.

I have sent a proposal for the problem solution last friday, but I don't see it here, so I think there was a problem when posting it in GitHub.

I suggested including the line:

graphicsFormat = UnityEngine.Experimental.Rendering.GraphicsFormat.B5G6R5_UNormPack16,

inside the constructor of the RenderTexture of MainTexture, so it gets like this:

    MainTexture = new RenderTexture(bgWidth, bgHeight, 0, RenderTextureFormat.RGB565, RenderTextureReadWrite.sRGB)
    {
        useMipMap = false,
        autoGenerateMips = false,
        wrapMode = TextureWrapMode.Clamp,
        filterMode = FilterMode.Point,

        graphicsFormat = UnityEngine.Experimental.Rendering.GraphicsFormat.B5G6R5_UNormPack16,

    };

Hope this works for other machines...

[alezuky]

Hi. Another update:

I tried the above solution on another computer, and I got the same Assertion Failure. I found out that this happened because that graphic format ( B5G6R5_UNormPack16 ) was not supported by the graphic card of that computer. I could check it by using the IsFormatSupported Unity function ( https://docs.unity3d.com/ScriptReference/SystemInfo.IsFormatSupported.html ). When the graphic format is not supported, Unity automatically substitutes it for another one.

The problem is that the current Barracuda Script is using a three-component graphic format to make the calculations. I understood that the three-components are the three base colors channels, like RGB, BGR... You can see more about the image components here: https://docs.unity3d.com/ScriptReference/Experimental.Rendering.GraphicsFormat.html .

The assertion failure points that the script was expecting to receive three components, but it receives four: "AssertionException: Assertion failure. Values are not equal. Expected: 3 == 4"

The assertion check is at BarracudaPrecompiledCompute Script: "Assert.AreEqual(X.channels, K.kernelDepth);"

I am concluding that some graphic cards do not support three-components channel images. I guess that the fourth component is the Alpha channel.

I saw at the Barracuda manual that it is possible to configure the script to work with four channels: "Texture as input: You can directly pass Texture2D, Texture2DArray, Texture3D or RenderTexture to Barracuda without accessing individual pixels on CPU: var channelCount = 3; // you can treat input pixels as 1 (grayscale), 3 (color) or 4 (color with alpha) channels var tensor = new Tensor(texture, channelCount); " Reference: https://docs.unity3d.com/Packages/com.unity.barracuda@0.3/manual/index.html

So, I believe we can modify the Barracuda source code to work with 4 channels instead of 3. The assertion that we are exploring is comparing the number of channels with the kernel depth. So, I think we should change the kernel depth to match the number of channels (from 3 to 4). This should solve the problem. Maybe, a more advanced solution would be to check what graphic formats are supported by the graphic card and make the Barracuda Scripts work with the number that best fits the graphic card configuration. This should make the script work for more computers. I also believe that making this change should not be complicated for someone who knows in details how Barracuda works, which is not my case - I tried some solutions, but could not modify the number of channels or kernels. So, I am giving this feedback here to see if someone could try it...

Hope this give us some direction.

[yukihiko]

Thank you for your feedback. Even if it's slow on my PC, Mac, or company, if I specify new Tensor (InitImg, 3), it doesn't work with an error. What is your PC environment? Are you using a Radeon GPU?

And I still don't understand. Does it mean that if the tensor is 4 channels, the input of the learning model also needs to be 4 channels? That's impossible.

[alezuky]

Hello.

Thanks for the feedback too.

I think we solved it this time. By specifying all the tensors in the Barracuda Runner Script to work with 3 channels, I could make the code work in a computer in which it was not working before.

In practice: At the VNectBarracudaRunner script: For all 6 "New Tensor" calls (3 at the UpdateVNectModel() function and 3 at the WaitLoad() IEnumerator), include ",3" after the image that is sent to the constructor. This will force the Tensor to be created with 3 channels.

Ans answering your question: The GPU of the computer where the code was not working before is a NVIDIA GeForce GTX 770, and the GPU of the computer where the code was working is a NVIDIA GeForce GTX 1060. Now it is working at the both computers.

[Cliwo]

@alezuky Working like a charm!!!!

Thank you!

[alezuky]

You're welcome!

Em sex, 4 de dez de 2020 07:43, Seungchan Jeong notifications@github.com escreveu:

@alezuky https://github.com/alezuky Working like a charm!!!!

Thank you!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/digital-standard/ThreeDPoseUnityBarracuda/issues/15#issuecomment-738711539, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFU2QYGVJZHBXBJ46EZSQ6TSTC4MNANCNFSM4QJCTONA .

[godspeed1208]

Adds ,3 to the second argument of new Tensor(InitImg) and new Tensor(videoCapture.MainTexture) of WaitLoad and UpdateVNectModel method in VNectBarracudaRunner.cs.

what does this mean? private IEnumerator WaitLoad() { inputs[inputName_1] = new Tensor(InitImg,3); inputs[inputName_2] = new Tensor(InitImg,3); inputs[inputName_3] = new Tensor(InitImg,3);

like this?

[alanprowtfq]

in the UpdateVNectModel function, in the 1rst line of code, like this: before: input = new Tensor(videoCapture.MainTexture); after: input = new Tensor(videoCapture.MainTexture, 3);