search

shuhaowu / remote-for-pianoteq

GNU Affero General Public License v3.0
15 stars 4 forks source link

Feature request - head tracker #7

Open davidedelvento opened 2 years ago

davidedelvento commented 2 years ago

See https://forum.modartt.com/viewtopic.php?pid=983525 for discussion

creaktive commented 2 years ago

I love the idea! I checked and the JSON-RPC interface does support the sending of head position parameters. There are several JavaScript libraries for head tracking using webcam, too (https://github.com/brownhci/WebGazer, https://github.com/auduno/headtrackr). I believe this should be a separate project, so as to avoid code bloating. I'll look into in when I have some free time :)

shuhaowu commented 2 years ago

Yes this is indeed a great idea! I think we can have it in this project, just in a separate HTML page so it is "independent" and doesn't "bloat" the main code. This way, you can even run the main controller html page on a phone, and put the head tracker html page on a separate computer with beefier processing.

creaktive commented 2 years ago

OK I made a PoC using face tracking via webcam: https://sysd.org/pianoteq-pov/ Works in Chrome & Firefox but not Safari (which refuses to call the JSON-RPC interface on localhost due to security concerns). FWIW, the source code is here.

davidedelvento commented 2 years ago

@creaktive this is very nice, thanks!

One needs a really beefy machine though. In my case, I had to select the lowest camera resolution and least performant Pianoteq settings (22kHz sampling, 4096 samples, ok for listening but not much for playing), otherwise I'd hear pops and cracks. That way it works fine for me. The sound changes moving the head!

There are two problem with the code, though. The sound was changing but not realistically, so I opened the mic view in binaural mode and the head does not rotate: just moves right and left or up and down. Which is very nice but incomplete set of movements that one want to track in Pianoteq. Looking at the blue dots on the camera, it appears that the code does track the rotation differently from the left/right movements, so it must be possible to include that too.

The second problem is that always, in chrome, after a few minutes, the website hangs forever, hence the position of the head in Pianoteq locks.

So great job, this is impressive. If you can fix these two things, it'll be perfect!

creaktive commented 2 years ago

@davidedelvento I added the head rotation and improved the head position tracking, please reload the link I shared earlier :) Regarding the performance and hanging issues: the face tracking algorithm is intended to run on a GPU, with a fallback to CPU. Using GPU, performance is decent even on my 6-years-old iPhone... Unless I'm missing something, of course! Can you please check the reference I'm building upon; whether it slows down/hangs for you?

davidedelvento commented 2 years ago

This is awesome.

Not sure if you changed anything in the backend settings, but it looks different (before it was tracking only eyes/ears/nose/mount, now it's meshing the whole head) and most importantly it does not cause any problem on my desktop anymore (I have a whimpy Intel GPU, which this backend appears to support, not sure about the "previous" one, assuming it were different).

From the phone, it appears to be working correctly, however if I use http://192.168.0.12:8081/jsonrpc link status is always red and (obviously) it does not work. I inadvertently discovered that if I "drop" the protocol and use 192.168.0.12:8081/jsonrpc link status becomes green, but it does not work either. Not blaming you this could be firewall settings on my desktop and/or router (and/or phone). It'd be nice to be able to use the phone, for machines that are underpowered or don't have a webcam.

creaktive commented 2 years ago

Glad to hear :D Indeed, I did swap the backend, since the whole-head mesh works much smoother (previous model was very jumpy, and obtaining a decent value of the head angle was too complicated). Being more stable is an extra bonus :) There's a simple hack that will allow head tracking to run on a mobile device: download this, unpack it anywhere, and then download server.py and put it in the same place you unpacked the ZIP. Execute server.py on the machine where your instance of PianoTeq is running, and then connect to it from your mobile device. Next, alter the JSON-RPC endpoint so that hostname/port matches the ones you used to connect to server.py. For instance, if you are accessing it as http://192.168.0.12:8000, then your endpoint is http://192.168.0.12:8000/jsonrpc. This should do the trick!