search

alvr-org / ALVR

Stream VR games from your PC to your headset via Wi-Fi
MIT License
5.57k stars 495 forks source link

Publish tracking data through VMC Protocol #2461

Closed grillo-delmal closed 1 month ago

grillo-delmal commented 1 month ago

The VMC Protocol is a protocol to send motion tracking data through OSC for avatar motion applications. It's implemented in software like VSeeFace and others.

Here is a test of it being used through the inochi-session app

https://github.com/user-attachments/assets/30a7f535-a077-46f8-857e-924c8bf60ed5

zmerp commented 1 month ago

I haven't tested myself, at least seems it doesn't break anything else.

grillo-delmal commented 1 month ago

https://gist.github.com/grillo-delmal/b7712e4935f017067ced3d0428536a01

here is a short python script if you want to see the data being communicated :) (requires pyliblo3)

grillo-delmal commented 1 month ago

I moved it to draft to correct the pose orientation, tested a model with vseeface and it's not good...

current state: vmc_current_state

grillo-delmal commented 1 month ago

Ok, got orientations fixed. Had to build some tooling to record the VMC communication and transform the output on the go.

One thing I learned is that VMC expects for the data to be sent in Unity orientation to work correctly. Also, the actual math to make this work, I had to apply different rotations to each bone, but all where some multiple of pi/2, so it wasn't that bad.

It didn't help that the model that I have available is a free one generated with VRoid, which is not intended for this purpose XD. But at least I'm happy with the end result ^^.

https://github.com/user-attachments/assets/b83def2b-175d-45fe-b778-b1744560d585

zmerp commented 1 month ago

@grillo-delmal are you sure what you are doing is right? shouldn't you just follow clippy suggestion?

zmerp commented 1 month ago

@grillo-delmal your link is unrelated lol. in any case the PR looks good to me

grillo-delmal commented 1 month ago

@grillo-delmal your link is unrelated lol. in any case the PR looks good to me

Thx for the catch :)

(deleted old message cos of the wrong link, here is what actually said xD)

@grillo-delmal are you sure what you are doing is right? shouldn't you just follow clippy suggestion?

yeah, I got those numbers by correcting ALVR raw output against VSeeFace result using this script , the collision ocurred when I reduced the amount of numbers in the value. I could use the constant and get the same result since the numbers are similar, but prefer to leave the number since it reflects more on the origin.