mrhegemon commented 2 years ago

Top Priorities

[ ] New UX
[ ] Fix face and hands
[ ] Video Segment to VRM
[ ] QA

Overall goal is to fuse our pose Skeletal IK from many sources such as multiple camera inputs and or pose capture and clients into a full body model.

Avatar System

[x] #7486
[x] #7592
[x] Rewrite animation state machine to be way simpler & data oriented
[x] Additive FK/IK blending instead of slerp
[x] #8790
[x] #7519
[ ] https://github.com/EtherealEngine/etherealengine/issues/6700
[ ] #5650
[ ] #8503
[ ] Regression testing for different avatars for bone matching, retargeting & IK
[ ] #7943
[ ] #8502
[ ] #7590
[ ] Pill colliders moving with each bone

Motion Capture System V1

[x] Mediapipe pose tracking
[x] hip position assessment
[x] hip rotation assessment
[x] https://github.com/EtherealEngine/etherealengine/issues/9133
[x] https://github.com/EtherealEngine/etherealengine/issues/9134
[x] landmark noise reduction using kalman or other filters https://github.com/freemocap/freemocap/tree/main
[x] landmark return to a rest pose or have some strategy for when data is not present
[x] ground assessment - may need a fancy strategy
[ ] https://github.com/EtherealEngine/etherealengine/issues/9045
[ ] https://github.com/EtherealEngine/etherealengine/issues/8563
[ ] Mediapipe face tracking
[ ] #9047
[ ] Allow users to change their video feed quality and crop
[ ] Crop pose video and pass to hands and face capture models

Motion Capture System V2

[ ] Build a pretty CV Motion Capture Control UX
[ ] #9048
[ ] https://github.com/EtherealEngine/etherealengine/issues/7983

Human Assessment

[x] foot placement
[x] torso mode (seated position, wheelchairs)
[ ] height assessment of avatar - may need various strategies such as xr inputs
[ ] wingspan assessment and fine grained body fitting
[ ] #7495
[ ] Build the Skeletal IK Settings UX

MoCap Recording & Playback

[x] Serialize (Record) the Skeletal IK
[x] Deserialize (Play back) the Skeletal IK
[ ] Build Client Record / Playback UX
[ ] Build Editor Record / Playback UX
[ ] Build the Admin UX

Later nice features

[ ] Custom camera position, FOV and Frustum, intrinsics and so on especially for multiple cameras
[ ] AR Magic Window #5303
[ ] MR Capture
[ ] https://github.com/EtherealEngine/etherealengine/issues/7983

NOTES

CV Tracking OPTION 1 : MediaPipe

Human js https://github.com/vladmandic/human https://google.github.io/mediapipe/getting_started/javascript.html

HOLISTIC BODY CAPTURE - WEB DEMO https://gist.github.com/mrhegemon/2832793cae560c6980d2ed40048cf1e4

SIMPLE POSE CAPTURE - WEB DEMO https://gist.github.com/mrhegemon/7931bf2cf8be283f3aef0b2bb7a5cbe4

CV Tracking OPTION 2 : Others

Light Tensorflow https://codesandbox.io/s/9rhlj

OpenPose https://cmu-perceptual-computing-lab.github.io/openpose/web/html/doc/index.html

AR Magic Window

https://andykong.org/projects/eyecatcher/ https://karan-ahuja.com/fun_projects/eyecatcher/

MediaPipe + Three js

Mediapipe + Unity

https://github.com/Kariaro/VRigUnity

BlazePoseWithUnity https://github.com/joonb14/BlazePoseWithUnity https://github.com/digital-standard/ThreeDPoseUnityBarracuda/tree/master https://github.com/digital-standard/ThreeDPoseUnityBarracuda/blob/master/Assets/Scripts/VNectModel.cs

Final IK Unity

http://root-motion.com/ https://assetstore.unity.com/packages/tools/animation/final-ik-14290 https://www.youtube.com/user/SnuffThePuppy/playlists

Web IK

Osso

https://twitter.com/sketchpunklabs https://www.patreon.com/posts/63305737 osso Tests https://www.youtube.com/@SketchpunkLabs/videos

OTHER EXAMPLES

https://www.kdnuggets.com/2020/08/3d-human-pose-estimation-experiments-analysis.html

8th wall https://www.8thwall.com/8thwall/face-effects-threejs/code/README.md

https://github.com/imerso/facecap

DIRECT POSENET TO VR https://gitlab.com/posenet-vr

Another - TDPToVR https://nponta-tanuki.booth.pm/items/2492481

https://github.com/TrackLab/ViRe

Real-time pose estimation vr tracker by web camera in VRChat [2021-0102] https://www.youtube.com/watch?v=cf5moO_1TWY

ThreeDPoseTracker https://mobile.twitter.com/search?q=%23ThreeDPoseTracker https://store.steampowered.com/app/1366950/Driver4VR/ https://www.youtube.com/channel/UCVEw7o4VIuX3ZvWoQx_n3Fg https://www.youtube.com/watch?v=5at0Gw_mEmc

Kaldo https://npm.io/package/kalidokit https://kalidoface.com/ https://kalidoface.com/

VRChat UX Flow https://docs.vrchat.com/docs/full-body-tracking

VRChat OSC https://docs.vrchat.com/docs/full-body-tracking https://docs.vrchat.com/docs/osc-avatar-parameters https://docs.vrchat.com/docs/osc-resources https://docs.vrchat.com/docs/osc-debugging

Rokoko + Move.ai

Potential hacks for improvement:

explore video vs image
crop to the frame of the most confident body
workerize a buffer of frames that can be assigned to threaded workers
explore offscreen-canvas as an alternate execution space for more multithreading
see what happens if we disable video rendering, and only show the skeleton (fps improvement?)
see if PWA or fullscreen makes any difference (increased browser priority?)

HexaField commented 1 year ago

Boneworks / Bonelab

Jumping mechanism is great Pop One
Gliding w/ arms Walkabout
Flying by pointing up and teleport button Echo Arena
Zero G flying VRChat