search

hiukim / mind-ar-js

Web Augmented Reality. Image Tracking, Face Tracking. Tensorflow.js
MIT License
2.13k stars 394 forks source link

How to select tracked face and crop video stream #459

Closed HieronymusLex closed 5 months ago

HieronymusLex commented 9 months ago

I have a use case where I'd like to:

  1. detect faces in the video feed and let the user cycle through which user should be the one that is tracked
  2. crop the video to only display the tracked face plus some padding
  3. (optional) on target lost, automatically select any of the other available faces in the frame

Face tracking only seems to track a single face at the moment and I can't see an equivalent parameter to the image tracking maxTracks? Therefore if it is only possible to track a single individual, I was thinking of managing the tracking outside of MindAR using a TF model to track individuals, then crop the video feed and supply that to MindAR. However I can't see a way of modifying the video feed without monkey patching navigator.mediaDevices.getUserMedia or perhaps creating a virtual camera from the cropped video stream and supply that deviceId when instantiating MindARThree

Is there a better way of achieving this?

hiukim commented 5 months ago

that sounds complicated, but I guess you could try to hack the source code and re-compile the library.

You can modify the input video here: https://github.com/hiukim/mind-ar-js/blob/fb5db87b1e5076ba86fde82101955da70b629a1c/src/face-target/three.js#L204

I believe it's probably some post-processing you need to do to adjust the position. because the threejs canvas is full screen, and your passed a chopped video feed for detection, so the detected pose will be off.

HieronymusLex commented 5 months ago

I'll have a play around with the input video and see if I can achieve what I'm after