Is it possible to link Omnitone to Google VR View?

[PoC22]

Rather than video, is it possible to show a static 360º image (say via VR View) and play an Ambisonic file that is linked to the viewpoint of the 360º VR View photograph?

[hoch]

It's on the project roadmap. Will update this issue when it lands.

[PoC22]

Thank you.

The scenario I'm looking at is producing a 90 minute audio podcast in Ambisonic sound and then linking it to a static (say Ricoh Theta S) 360º image that people can look around in using VR View (already have VR View working).

My assumption is that the Ambisonic file would be 128 or 160 kbps AAC format due to length of the recordings!

[hoch]

I don't think we need tight integration between VR view and Omnitone to achieve this. It seems like only thing that needs to be shared is the rotation matrix, feeding from VR view to Omnitone.

[PoC22]

So, not too difficult to achieve?

[hoch]

No. I think it is on the easy side. I will do some fiddling and report back here soon.

[PoC22]

New ambisonic microphone and recorder appearing next month, so will be interested to see how this match-up goes.

[hoch]

Could you share the name of devices that you're planning to use? I would like to try as well.

[hoch]

The work has been started. The integration was fairly simple, and I am coordinating with the VR view team so Omnitone can be the default ambisonic decoder within.

[PoC22]

Excellent! Looking forward to the results!

Re: Devices - Ambeo VR Mic and Zoom F4 (due November).

[PoC22]

Are there any further updates on this yet?

[hoch]

If you can't wait for the official update, take a look at this PR: https://github.com/googlevr/vrview/pull/67#pullrequestreview-6099344

The update is planned and ongoing. Sorry that I cannot give you the exact timeline.

[reech]

We have tried the PR with 360 images instead of video. http://www.ambyss.com

[hoch]

@reech There seems to be an issue with the integration testing.

@borismus Boris, what was our next step for the integration? Should I provide some unit tests for Omnitone within VRView?

[reech]

@hoch It is just a proof of concept which will stay open until official PR. For now the repo is private since it is really bad and messy, and I leave it up to you guys to get it done the right way. Anyway, here are some experiences, pitfalls and suggestions:

1. For best user experience I believe there should be 3 possibilities for the Omnitone integration in the VRView constructor:

• Video with stereo or ambisonic audio integrated in the mp4

• Image with external stereo or ambisonic audio source

• Video with external stereo or ambisonic audio source (ex: music videos, nature, non-descriptive videos)

example:

var vrView = new VRView.Player('#vrview', { video: 'link/to/video.mp4', is_stereo: true, ambidecoder: true, ambisource: '/audio/ambisource.wav' });

I suggest an extra ambisonic audio source parameter in VRView constructor, as well as be able to turn on or off ambisonic decoder. These extra parameters has to be handled correct in scene-info.js module and handled further in world-renderer.js; world-renderer.js which in turn is imported in embed/main.js. The omnitoneDecoder has to be passed between the world-renderer.js to the rotation matrix in the render-loop in embed/main.js. Should omnitoneDecoder maybe be an extra module? globals?

2. VRView.setContentInfo() problems

This method I believe should also have the ambisonic source, ambisonic decoder on/off, etc as the VRView constructor, otherwise this method only works without ambisonics. The problem with this approach is the audiocontext in:

var omnitoneDecoder = Omnitone.createFOADecoder(audioContext, audioElement);

Since browsers fail with more than one audiocontext(mixer), I solved it with audioContext.close() and re-initiated it. I couldn't fetch the original audiocontext from the DOM, if it even exists there. Better solutions?

[hoch]

Thanks for the notes. Some follow up questions:

Since browsers fail with more than one AudioContext(mixer)

Do you have a repro case for this? It seems like a browser bug. Chrome has a hard limit of 6 and FF does not have this limit. Closing a context and recreating one sounds okay. Note that you close the context, that will end of context lifetime and you cannot restart it.

Out of curiosity, why do you need multiple AudioContexts? (and multiple Omnitone decoder?)

[reech]

@hoch I tried to reproduce the error I had with reaching audiocontext limit, but it seems to work now without closing and recreating. My bad, sorry. I don't need several AudioContexts or Omnitone decoders, just one, but I believe it should be possible to change the audio source (as is possible with the video or image source) with the VRView.setContentInfo() method in order to have one player and many examples/thumbs. Maybe it's enough to just change the audioElement.src instead of recreating the Omnitone decoder like I did. I will try that

[hoch]

Maybe it's enough to just change the audioElement.src instead of recreating the Omnitone decoder like I did.

Hmm. That's actually good point. My concerns are: 1) The change will abrupt and you will here glitches. 2) You don't have control over the timing of the transition.

You might want to set up 2 Omnitone decoders and crossfade each other, if you really want to have a smooth change.

[reech]

@hoch To change the audioElement.src works fine with just one decoder, the same is done in the Omnitone only demo I see now: https://github.com/GoogleChrome/omnitone/blob/master/examples/demo.html If it should be an audio fadeout/fadein on changing sources like with the fade-to-black when changing video source I am not sure.

It would be really nice to have omnitone as the default decoder in VRView, and hopefully also possible to change audio and video sources for one player with many examples, like here (just with custom audio-sources as well): http://googlevr.github.io/vrview/examples/gallery/index.html

Looking forward to it, and I guess many more with me.

[hoch]

@reech The new FOARenderer in v0.2.x now behaves just like an AudioNode. You can create your own MediaElementSourceNode and feed the stream into Omnitone. Also the output Omnitone renderer needs to be connected to the destination. That way you can have a full control over the input and the output.

See here: https://github.com/GoogleChrome/omnitone/blob/master/examples/foa-renderer.html

[hoch]

Unfortunately the VRView integration will not be pursued anymore. My plan is to make Omnitone more flexible/open for projects like Three.js and other WebVR projects.

Closing this issue as "Won't fix".

[topherbuckley]

Is there any documentation as to how to link omnitone with Three.js, WebVR, etc.? I tried linking it with VRView and couldn't get it to work, which is likely due to the fact that I couldn't properly link the rotationMatrix, but I'm not 100% sure yet.

I was trying to download a copy of demos on your "https://googlechrome.github.io/omnitone/#home" site for testing/fiddling purposes, but it seems there may be some automated block for using wget or other scraping tools? I just wanted some way to verify that my setup is able to use omnitone in unison with something visual (Three.js, WebVR, VRView, etc.). The only examples in the git repo are audio only it seems. Any recommendations as to how I can test the pairing of omnitone with one of the above mentioned visual libraries/handlers?