This repository provides an ideal way to use PoseNet for installations; it runs totally offline and can stream poses over websocket to any consuming appication such as Unity, OpenFrameworks or TouchDesigner, or a browser on another computer or mobile device. Much of the useage is inspired by the wonderful work being done at RunwayML
PoseNet allows for real-time human pose estimation in the browser with any webcam, powered by machine learning and tensorflow.js.
This application is a re-write of the camera demo with a few improvements ideal for a scenario where it is to be run as part of an installation:
The app is built in react, material-ui, and typescript, and was bootstrapped with create-react-app.
Make sure to have yarn installed. If you don't, follow the instructions to do so here
Install all dependencies:
yarn setup
Download all the models:
yarn download-models
Start the application and the node.js server (responsible for broadcasting pose messages over websocket):
yarn start
To just start the application (if you don't want to broadcast the poses over websocket):
yarn start-client
Open http://localhost:3000, start a video capture and start estimating poses. Then click to connect to the local node server hosted at localhost:8000; this will broadcast all estimated poses via a websocket through that server.
On your client application, consume from the websocket at ws://localhost:8080.
When poses are estimated, they are broadcast over the websocket to all subscribers. The object that is broadcast looks contains:
{
"image":{
"width":640,
"height":480
},
"poses":[
{
"score":0.36304738942314596,
"keypoints":[
{
"score":0.8328775763511658,
"part":"nose",
"position":{
"x":-13.688090340035862,
"y":239.52947203318277
}
},
{
"score":0.8907157778739929,
"part":"leftEye",
"position":{
"x":-4.325634065221568,
"y":229.78147989908854
}
},
{
"score":0.4457024037837982,
"part":"rightEye",
"position":{
"x":-15.426258524910349,
"y":224.52771606445313
}
},
{
"score":0.8607040047645569,
"part":"leftEar",
"position":{
"x":16.80914206582992,
"y":229.8184631347656
}
},
{
"score":0.174045130610466,
"part":"rightEar",
"position":{
"x":-11.459165479316086,
"y":221.1838175455729
}
},
{
"score":0.962192177772522,
"part":"leftShoulder",
"position":{
"x":37.98613301261527,
"y":302.2597198486328
}
},
{
"score":0.45834219455718994,
"part":"rightShoulder",
"position":{
"x":-3.9096296967053026,
"y":298.102495320638
}
},
{
"score":0.7816433906555176,
"part":"leftElbow",
"position":{
"x":27.73797957623591,
"y":401.3367919921875
}
},
{
"score":0.14949043095111847,
"part":"rightElbow",
"position":{
"x":-0.5075093566394243,
"y":390.36883748372395
}
},
{
"score":0.13570570945739746,
"part":"leftWrist",
"position":{
"x":-22.4962658491291,
"y":409.99115022023517
}
},
{
"score":0.041653428226709366,
"part":"rightWrist",
"position":{
"x":-10.491187924244365,
"y":427.8301574707031
}
},
{
"score":0.25177454948425293,
"part":"leftHip",
"position":{
"x":9.068478443583505,
"y":473.309304300944
}
},
{
"score":0.11538269370794296,
"part":"rightHip",
"position":{
"x":-1.3889761752769596,
"y":473.36097513834636
}
},
{
"score":0.019853923469781876,
"part":"leftKnee",
"position":{
"x":-2.5295049948770494,
"y":469.2986684163411
}
},
{
"score":0.01517724059522152,
"part":"rightKnee",
"position":{
"x":-2.33144766385438,
"y":470.7204767862956
}
},
{
"score":0.024063894525170326,
"part":"leftAnkle",
"position":{
"x":-5.524195436571466,
"y":468.3872568766276
}
},
{
"score":0.012481093406677246,
"part":"rightAnkle",
"position":{
"x":-7.836129610655738,
"y":468.0144561767578
}
}
]
},
...
],
}
Run the command to create an executable of the application and server:
yarn build-and-distribute
Copy client/posenet.exe and server/posenetServer.exe to the destination directory.
TODO: add a flag that allows changing the target OS.
If you want to develop the code, it is strongly recommended to use VSCode, as that provides an ideal IDE for working with typesript.