search

benaclejames / VRCFaceTracking

OSC App to allow VRChat avatars to interact with eye and facial tracking hardware
https://docs.vrcft.io
Apache License 2.0
607 stars 94 forks source link

Previewing Lip Tracker Freezes Parameters #38

Closed 200Tigersbloxed closed 3 years ago

200Tigersbloxed commented 3 years ago

Keeping this short because I don't know what else to say about this. When you go to the SRanipal tab in the Quick Menu, and you preview the Lip Tracker camera, it freezes the parameter values (as seen visually on the avatar and from the Debug panel).

For example:

It seems to happen to me only when I preview the Lip Tracker camera, but a simple restart fixes it. The log and console do not output any error about this at all, so I won't upload a log file unless requested. Any thing else needed and I'll be happy to get back to you!

benaclejames commented 3 years ago

I'll be removing the lip camera functionality in the next update for this very reason. The runtime really does not like me streaming live video from it. However, I may replace the image in the quickmenu for a live selfie view of your current avatar or simply a mirror to allow people to check if their face is tracking properly

200Tigersbloxed commented 3 years ago

I did some testing to see if it actually see if this was on the SRanipal's end. I set up a basic Unity 2019.4.16f1 project, imported the ViveSR SDK, and added the Lip Framework Prefab.

I then setup 2 scripts: LipCameraViewer.cs and LipDataViewer.cs

I tried to set them up as close as you did, and LipData is v2. The two classes should be obvious, but I'll briefly explain what they do. LipCameraViewer will render the latest image from the tracker to a RawImage in a canvas. LipDataViewer will get the LipData and LipShapes/LipWeights and then output the TongueRoll property to a UI Text inside the same Canvas.

From my testing, I can see that running an update frame to get camera output while getting data does NOT effect anything on the SRanipal end at all. The data values seemed to still work and the camera output didn't lag or anything like that.

benaclejames commented 3 years ago

Issue is, the mod has another step of invoking the functions on the il2cpp domain. This causes some instability and minor lag which I feel is probably to blame. My code essentially does the same as you've done in yours so I'm not quite sure what's doing it at this point

200Tigersbloxed commented 3 years ago

Issue is, the mod has another step of invoking the functions on the il2cpp domain.

Changed the Scripting Backend to IL2CPP and set the Api Compatibility Level to .Net 4.x. The same results occur. TongueRoll seems to work properly and the images are still rendering properly, There seems to be no issues. If I need to, I can write a quick mod to link into VRChat and do the same thing for testing, but it wouldn't explain why it works fine in the Editor and not in-game.

A guess could be because you're modding it in with MelonLoader rather than building from unity (which I don't expect you to be able to do haha), but I don't see why that would make a difference.

I'm not quite sure what's doing it at this point

Here's my guess. If you look at line 14 in LipTrackingMenuPage.cs ( url ), you can see it's setting the LipEnabled value to whatever b is. I think b is false which is causing all the parameters to stop working. This could explain why the parameters default to some value rather than freezing at what they last were. I don't know what event TrackingEnabled is, but it's odd how it wouldn't set the value back to true, maybe the true event never hits it because the class "ends?" If you can, I'd try registering the event somewhere else in a static class (like MainMod.cs) since the class LipTrackingMenuPage is non-static.