any plans on upgrading it to 3d posenet?

[hpssjellis]

@tejaswigowda I don't have the time at the moment but 3D Posenet sounds really interesting do you have any links to it using tensorflowjs? Even with 3D posenet converting to a BVH file will have some math difficulties as the structure is quite different between the two sets of data. It would be a great project for someone to work on and would really help with animating a scene using belner.org.

[tejaswigowda]

I have had success with handpose with 3d fingers with a different project. However 3d body pose angles are hard. Here is a start: https://tejaswigowda.com/mediapipe-pose2bvh (standard example, gives a 3d plot of points). I will start working on it. Happy to get any help!

https://github.com/tejaswigowda/mediapipe-pose2bvh

[hpssjellis]

@tejaswigowda That is awesome, but don't you have the same problem I had mentioned in my github https://hpssjellis.github.io/tensorflowjs-bvh/ that converting the posenet data to bvh is mathematically confusing. Is the 3D data better suited to the problem? Can you post how the 3d psoenet data looks?

This is the 2D data

{
   "score": 0.0009414869196274702,
   "keypoints": [
      {
         "score": 0.00006651878356933594,
         "part": "nose",
         "position": {
            "x": 249.84802904564316,
            "y": 304.3941908713693
         }
      },
      {
         "score": 0,
         "part": "leftEye",
         "position": {
            "x": -9.578838174273859,
            "y": -0.28868514571447096
         }
      },
      {
         "score": 0,
         "part": "rightEye",
         "position": {
            "x": 12.497390430497925,
            "y": -2.538142667271784
         }
      },
      {
         "score": 0.00022864341735839844,
         "part": "leftEar",
         "position": {
            "x": 331.26815352697093,
            "y": 357.0778008298755
         }
      },
      {
         "score": 0,
         "part": "rightEar",
         "position": {
            "x": 5.787214730290456,
            "y": -5.047182961618257
         }
      },
      {
         "score": 0.0013141632080078125,
         "part": "leftShoulder",
         "position": {
            "x": 367.18879668049794,
            "y": 299.87085062240664
         }
      },
      {
         "score": 0.00040268898010253906,
         "part": "rightShoulder",
         "position": {
            "x": 140.62266597510373,
            "y": 250.91234439834025
         }
      },
      {
         "score": 0.0010004043579101562,
         "part": "leftElbow",
         "position": {
            "x": 440.36047717842325,
            "y": 487.1903526970954
         }
      },
      {
         "score": 0.0005249977111816406,
         "part": "rightElbow",
         "position": {
            "x": 162.70720954356847,
            "y": 375.969398340249
         }
      },
      {
         "score": 0.0005192756652832031,
         "part": "leftWrist",
         "position": {
            "x": 506.8801867219917,
            "y": 409.49533195020746
         }
      },
      {
         "score": 0.0007147789001464844,
         "part": "rightWrist",
         "position": {
            "x": 387.9429460580913,
            "y": 470.69346473029043
         }
      },
      {
         "score": 0.004131317138671875,
         "part": "leftHip",
         "position": {
            "x": 405.50414937759336,
            "y": 482.1348547717842
         }
      },
      {
         "score": 0.0031223297119140625,
         "part": "rightHip",
         "position": {
            "x": 412.95435684647305,
            "y": 479.4740663900415
         }
      },
      {
         "score": 0.00125885009765625,
         "part": "leftKnee",
         "position": {
            "x": 231.22251037344398,
            "y": 366.92271784232366
         }
      },
      {
         "score": 0.0009179115295410156,
         "part": "rightKnee",
         "position": {
            "x": 166.69839211618256,
            "y": 366.6566390041494
         }
      },
      {
         "score": 0.0008487701416015625,
         "part": "leftAnkle",
         "position": {
            "x": 448.34284232365144,
            "y": 481.07053941908714
         }
      },
      {
         "score": 0.0009546279907226562,
         "part": "rightAnkle",
         "position": {
            "x": 171.4878112033195,
            "y": 453.39834024896265
         }
      }

[tejaswigowda]

Here is the 3d data:

{"poseLandmarks":[{"x":0.555702805519104,"y":1.0254617929458618,"visibility":1},{"x":0.5740898251533508,"y":0.9523078203201294,"visibility":1},{"x":0.5902284979820251,"y":0.9488478899002075,"visibility":0.9999569654464722},{"x":0.6060236692428589,"y":0.9454158544540405,"visibility":1},{"x":0.5316729545593262,"y":0.9589123129844666,"visibility":1},{"x":0.5152977705001831,"y":0.9601184129714966,"visibility":0.9998074173927307},{"x":0.4991629719734192,"y":0.9615576863288879,"visibility":1},{"x":0.6277121305465698,"y":0.9619799852371216,"visibility":0.9992175102233887},{"x":0.48071950674057007,"y":0.9844540953636169,"visibility":0.9970190525054932},{"x":0.5879846215248108,"y":1.0820200443267822,"visibility":0.9418684244155884},{"x":0.5297483205795288,"y":1.090785264968872,"visibility":0.9317114353179932},{"x":0.7159487009048462,"y":1.2764034271240234,"visibility":0.05079945921897888},{"x":0.406944215297699,"y":1.26321280002594,"visibility":0.04913078621029854},{"x":0.7584456205368042,"y":1.7053102254867554,"visibility":0.02981654182076454},{"x":0.34312504529953003,"y":1.6720788478851318,"visibility":0.023464657366275787},{"x":0.6876243352890015,"y":1.8120176792144775,"visibility":0.08006926625967026},{"x":0.38454490900039673,"y":1.7714799642562866,"visibility":0.05781161040067673},{"x":0.6697507500648499,"y":1.8757975101470947,"visibility":0.18155324459075928},{"x":0.39518699049949646,"y":1.8450589179992676,"visibility":0.0991208553314209},{"x":0.6533892750740051,"y":1.825445532798767,"visibility":0.2838806211948395},{"x":0.40981394052505493,"y":1.8128212690353394,"visibility":0.17175468802452087},{"x":0.6497206091880798,"y":1.805605173110962,"visibility":0.21190567314624786},{"x":0.4158953130245209,"y":1.789478063583374,"visibility":0.13006426393985748},{"x":0.6305559277534485,"y":2.0137038230895996,"visibility":6.513035799571298e-8},{"x":0.4521673321723938,"y":1.9987857341766357,"visibility":3.076540266988559e-8},{"x":0.6322956085205078,"y":2.46055269241333,"visibility":0.0015247863484546542},{"x":0.428826242685318,"y":2.4528613090515137,"visibility":0.003538086311891675},{"x":0.6082723140716553,"y":2.831228733062744,"visibility":0.00013552077871281654},{"x":0.43409714102745056,"y":2.850555181503296,"visibility":0.0001956353517016396},{"x":0.5969229936599731,"y":2.891045570373535,"visibility":0.0002251682453788817},{"x":0.43673548102378845,"y":2.8967597484588623,"visibility":0.0002982776495628059},{"x":0.5912813544273376,"y":3.0208709239959717,"visibility":0.00043393249507062137},{"x":0.4406220018863678,"y":3.0242300033569336,"visibility":0.0005420927191153169}]}

[hpssjellis]

hmmm, that looks less easy to work with than my post above. And I don't see a "z" value. I assume their is a key for the order of the data: nose then leftEye etc.

[tejaswigowda]

Here you go: https://tejaswigowda.com/mediapipe-pose2bvh/. With full finger and face tracking.

[hpssjellis]

@tejaswigowda that is really cool. I get the webcam and your robot, not sure if I am supposed to be seeing any BVH data. What is the next step?

[tejaswigowda]

Done. functionally. BVH writer needs debugging. But try it out; https://tejaswigowda.com/mediapipe-pose2bvh/

"Record BVH" is at bottom left.

[hpssjellis]

@tejaswigowda do you have a linkedin account, I would like to show this off. Mine is

Jeremy Ellis

are you Tejaswi Gowda 2nd degree connection 2nd Clinical Professor of the Internet of Things, Arizona State University + Computer Scientist at Foxy Ninja Studios

Should the first frame of the recorded file be t-pose? I tried to bind your bvh to one of the mixamo models and it was very difficult to do.

The first frame at t-pose would allow me to delete the mixamo armatures, bind your armatures to the mesh and then animate the object fairly easily.

Well done by the way, this is awesome.

[hpssjellis]

@tejaswigowda On second look the armatures edit mode is in t-pose settings so I kind of was able to make a model using blender.org. A few of the body parts got messed up but I think that is more my mistake than what you have done. Really awesome work.

[tejaswigowda]

Thank you. Made the first frame a t-pose. Going to bed now :)

[tejaswigowda]

Feel free to port it on linkedin

[hpssjellis]

@tejaswigowda you mentioned 3D posenet. This work is still using the 2Dposenet, would it be helpful if I could get 3D Posenet working? Is it already ported to TFJS? I know nothing about it.

[tejaswigowda]

It is using 3d mediapipe pose. from here:https://codepen.io/mediapipe/pen/jOMbvxw

Results look like this

Look at holisticResults.poseLandmarks in console.

[hpssjellis]

@tejaswigowda I notice the Armature structure does not have facial bones. Is that on purpose or is that coming later? I think posenet gives facial armatures

[tejaswigowda]

face blendshapes are already mapped.

https://github.com/hpssjellis/tensorflowjs-bvh/assets/1267401/f5721693-0978-487b-b272-a9208a3ee4a3

[tejaswigowda]

Working on tongue and iris mapping

[hpssjellis]

The models that load onto blender have no face armatures. Is that a blender issue?

[tejaswigowda]

the face data is not in the bvh, working on exporting it. It'll probably be a json.

[tejaswigowda]

What are you using this for? (if you don't mind my asking)

[hpssjellis]

I might have used an older version. I just got the face motion working. It is all very cool. Does it use any of the work I did or is it all from mediapipe and your own code?

[tejaswigowda]

None of your code. Started from scratch.

Have a bunch of mocap solutions I work on, both academically and commercially -- IMU, UWB, rgb, optitrack, streaming, xr -- all this kind of stuff.

But something like this should be in the open domain. Will seed a lot of other ideas.

[tejaswigowda]

(full disclosure: copilot is a huge help!)