motion capture tools

[ngokevin]

http://aframe.io/blog/motion-capture https://github.com/dmarcos/aframe-motion-capture-components/pull/32

Changes Proposed

Motion Capture Features

• Press the camera icon in the top left or press m shortcut to toggle motion capture tools above scene graph. Opening the tools will inject the motion capture components.
• Specify recording name to save as (within Indexed DB) with a recording button to start recording. When we press the recording button with the controller toggle on, then we press any button on controller to start recording and then press any button five times quickly to stop recording. There will be a 5 second countdown before recording, shown both in the 2D UI and in-VR camera. There will also be a 2D button to stop recording too.
• Select a recording from Indexed DB via dropdown. All recordings will be saved locally. Then we can press replay button to start recording. During replay, we can either use the Inspector camera (and even modify the scene live), the first-person player camera, or the spectator camera (hit q outside Inspector).
• To easily transfer recordings between computers, we can export recording by saving as a file, upload button to gist.github.com and get a URL back. Or delete recordings.
• We can add recordings into our recording DB from file system or from a URL.
• The Inspector will populate Indexed DB with a sample recording with head and two controllers if the database is empty. Developers will have to set IDs rightHand/leftHand to see controllers, however, I added a note in the UI when this sample recording is selected. Or perhaps when we select a recording, I should always should which IDs it is controlling.

[ngokevin]

Available to test https://ngokevin.github.io/aframe-inspector/example/controllers.html

[dmarcos]

I did a quick first run. Problems:

1. When the recording starts the inspector closes, what makes everything a bit confusing. I know that the scene is paused and might be difficult to resume / pause with the inspector opened. We could maybe at least to add any recording feedback while recording is happening with the inspector closed. A blinking red dot, a timer? It would be interesting to have a button to stop recording and then the inspector opens back up again when pressed. I would probably hide the Inspect Scene button as well.
2. When a recording is playing I can't start recording. Replaying should stop and recording start

[ngokevin]

1. The Inspector should close because the Inspector adds helpers and extras. We want to be sure we are recording with the pure scene. The Inspector closing adds a level of feedback that the recording has started (versus nothing on the screen changing). -- I will make the Recording button do a countdown for now, and we can add more feedback later. -- What I intend is to have the Inspect Scene button act as the Stop Recording button. I need to make the Inspector overlay a button on top.

2. OK.

[ngokevin]

I also added a few more tasks for myself in the PR description via the checkboxes after doing a run because sometimes I'd rather not have to open the Inspector on each refresh. People can import the components, take recordings with the Inspector, but be able to auto-replay them on refresh via like query param or component configuration.

[ngokevin]

Ready for another try-out. https://ngokevin.github.io/aframe-inspector/example/controllers.html

Camera switching may be weird sometimes because the Inspector examples immediately boot into the Inspector, and vrdisplayactivate will kick you into VR while in the Inspector. Sometimes recording will work fine, but best to exit VR and re-enter the Inspector and record. In normal use case, it is not a big problem because entering the Inspector will exit VR.

[ngokevin]

[ngokevin]

Also finding it useful being able to change the speed of the recording. I wrote a script to double the speed by dividing the timestamps in half, speeds up each test iteration for me (e.g., watching a 20 second recording). Maybe later will add a UI tool for that.

[ngokevin]

@fernandojsg r?

[fernandojsg]

@ngokevin I've just fetched it, npm install and npm start but I get:

ERROR in ./~/react-file-reader-input/lib/index.js                                                                                
Module not found: Error: Cannot resolve module 'prop-types' in C:\code\aframeinspector\node_modules\react-file-reader-input\lib 
 @ ./~/react-file-reader-input/lib/index.js 29:17-38                                                                             
webpack: Failed to compile.                                                                                                      

Did you forget to add some dependency in package.json? Or am I doing something wrong? :\

[ngokevin]

OK, can you try again? I think I needed to move a devDep to a dep in react-file-reader-input

[fernandojsg]

@ngokevin I've tried again and that error is gone, but as long as I click on the motion capture button and the inspector injects the aframe-motion-capture component I get the following error on the console:

[inspector] Injecting <script src="https://rawgit.com/ngokevin/aframe-motion-capture/mocapinspector/dist/aframe-motion-capture-components.min.js"></script> for motion capture.
SceneGraph.js:107 Uncaught TypeError: Cannot convert undefined or null to object
    at Function.keys (<anonymous>)
    at SceneGraph.js:107
    at treeIterate (SceneGraph.js:105)
    at treeIterate (SceneGraph.js:130)
    at EventEmitter.SceneGraph._this.rebuildOptions (SceneGraph.js:134)
    at EventEmitter.emit (events.js:96)
    at Events.emit (Events.js:16)
    at MutationObserver.<anonymous> (Main.js:61)

/cc @dmarcos

[ngokevin]

Is this A-Frame master? Will try again later.

[dmarcos]

When testing this:

1. I click on the play button for the Sample Recording and I get the following errors on on console on Chrome.:

2. In Firefox. I see an Indexed DB error and the Sample Recording does not show up in the list.

[dmarcos]

When trying to record:

• In both Chrome and Firefox: I introduce test as a recording name and play record. It enters fullscreen (testing without headset) I move with the keyboard and press escape. When back to the inspector my recording does not show up in the list and I see the same errors as in my previous comment (constructor of null in Chrome and IndexedDB errors on Firefox)

[ngokevin]

OK will patch it up soon. I think it was due to an A-Frame regression (constructor is null in buildData), was working flawless for me otherwise.

[ngokevin]

OK fixed the A-Frame error, try again.

[ngokevin]

nvm need to fix things...code has rotted since rebase

[ngokevin]

OK, all fixed up again. Had to fix several bugs with the motion capture components (changes to A-Frame tracked-controls system which motion capture replayer wraps), and since A-Frame no longer cloned data on getAttribute, motion-capture-recorder had to clone so it wasn't save the same object for all of its poses.

[fernandojsg]

IMHO too much verbose info on the component. I would get rid of the long explanation on the toggle recording checkbox and on the controllers sample info, maybe adding then to a tooltip next to the button if needed, because right now is taking more than half of my screen's height.

I would change the color of the Record and Replay instead of using underline as the first reaction would be to try to click it thinking it's a link.

I'll be out without access to a Vive so I can't test the functionality with it, @dmarcos could you try it?

In the meanwhile I've been trying with the laptop without any headset and it's confusing when you click the record button as it just go fullscreen without any other action. Should we just remove the record button if not controls or headset detected? And just let the user to replay recordings?

[ngokevin]

I wouldn't remove the text, they are critical instructions and it's a checkbox that changes behavior. It'd be confusing to remove the instructions and just have a bunch of shape icons. The tools are collapsible if you don't want to see.

The tools are primarily for headset but still works for 2D. Unticking the checkbox will start recording without controller trigger. I can change color of headings but I use this heavily so it works, put maybe a month into it, and it'd help everyone who is still putting the headset on and off every code change.

[fernandojsg]

Ok to the comments text. I still don't undestand the behaviour in 2D as I don't see any feedback that I'm recording or so, it just go fullscreen and after exiting I can't find the recorded session. Am I doing something wrong?

[fernandojsg]

I mean, if it's not still supported the 2D mode for recording, we could just merge it and then we could fix/disable the 2d mode recording in another PR, and then improve the support for it just not to delay this PR as it's already being delayed a lot (Sorry for that :)

[fernandojsg]

@ngokevin any thoughts on my previous comments?

[ngokevin]

@fernandojsg Sorry, I'm back! I added some text display to the camera Press any controller or keyboard button to record, and support pressing keyboard button to start. This gives an indicator for both 2D and VR mode. Should be good now!

[fernandojsg]

@ngokevin welcome back! :+1: I've tried it and it works great in 2D! I've just found a small bug that make it throws an error when using the keyboards to move around the scene.

[ngokevin]

ok fixed

[fernandojsg]

Supercool!!! 🎉🎉🎉🎉🎉🎉🎉🎉🎉🎉🎉🎉🎉🎉🎉🎉

[ngokevin]

horray, i'm gonna cut a release of inspector and later on do a blog post

[fernandojsg]

@ngokevin cool! :+1:

[Utopiah]

Bit confusing to get a fixed point of view when the recording is finished, might we less disorienting to leave the camera moving with the HMD but display a message like "remove the HMD and go edit in the inspector" after.

Anyway that's great, can't wait to read the blog post!

[nikgraf]

Really amazing!