GoogleChrome / omnitone

Spatial Audio Rendering on the web.
https://googlechrome.github.io/omnitone
Apache License 2.0
855 stars 114 forks source link

Apply ambisonics to an audio file then to a video file result to a non playable video #84

Open bbaudel opened 6 years ago

bbaudel commented 6 years ago

Here is the code using it. I will add a sample online soon (tested with Omnitone 1.0.6 and 1.2.1) : https://github.com/gopro/forgejs/tree/alps-700-omnitone-fix https://github.com/gopro/forgejs-samples/tree/alps-700-omnitone-fix

bbaudel commented 6 years ago

Here is the sample to execute on a localhost: https://www.dropbox.com/s/ok3c101ej0r71mk/sample.zip?dl=0

Here is the sample online: http://www.kolor.com/private_forgejs/projects/ambisonic/ to begin by an image scene with ambisonics audio (first scene) http://www.kolor.com/private_forgejs/projects/ambisonic/#first-scene&uid=scene-2 to begin by a video with ambisonics (3rd scene)

bbaudel commented 6 years ago

Notes on the samples

hoch commented 6 years ago

So sorry for the late response! (Was on vacation when the issue was filed...)

I am not sure if I understood the problem correctly. This is what I did:

  1. Go to http://www.kolor.com/private_forgejs/projects/ambisonic/
    • The audio plays correctly. (scene 1)
  2. Press ">" on the right.
    • The audio still plays correctly. (scene 2)
  3. Press ">" on the right again.
    • No audio from the player. (scene 3)

But if I access the scene 3 with the direct link, it plays the audio correctly.

Did I get your problem right?

bbaudel commented 6 years ago

Yes exactly. It can be linked to our audio implementation. For image scene with ambisonics we instanciate an audio tag and use a global webaudio context which are destroyed and "unlinked" when leaving the scene. For video scene with embedded ambisonics we instanciate a video tag and use a global webaudio context which are destroyed and "unlinked" when leaving the scene.

hoch commented 6 years ago

Have not looked at the code yet...

Can you keep the audio context and the Omnitone renderer? Why do you have to re-instantiate the whole thing for every scene?

The Omnitone renderer is simple and transparent, so all you need to do is the reset the rotation matrix to put it back to the initial state. There will be a tail of 256 samples from the convolution, but that's less than ~7ms. It should be trivial to reconstruct the renderer from scratch, but still there is a cost and you might want to avoid that.

For each scene, all you have to do is to create MediaElementAudioSourceNode for audio or video tag and feed it into the renderer.

Keep it mind that an AudioNode must be connected to the node belongs to the same audio context.

bbaudel commented 6 years ago

Thanks for your feedback.

I've tested with a SoundManager class which create a FOARenderer and connect it to the "global" audio context. Then the Sound and Video classes now only connect the sound element source to the FOARenderer input when they are instanciated.

But I've exactly the same result.

Note: on some sounds or videos I'll be happy to switch between FOA and HOA renderer also, it's why I was creating and instanciating the Ambisonics renderer on each sound or video where ambisonics are required on my first test.

hoch commented 6 years ago

Could you try it again with the latest version of Canary? I think it might have been a bug in Chrome's implementation and I fixed the issue a couple of days ago.

bbaudel commented 6 years ago

Same result with a few tests with the latest Canary version.

hoch commented 6 years ago

Okay, I think this is related to an implementation bug in Chrome. Have you had a chance to try this out in FireFox?

hoch commented 6 years ago

My bet is changing the source of MediaElement dynamically does not go well with the attached MediaElementSourceNode (which is a Chrome bug). For the workaround/hack, I suggest you to do this:

  1. Create VideoElement
  2. Set src.
  3. Create MESN with the video element.
  4. Connect to the Omnitone.

When changing the source:

  1. Disconnect MESN from Omnitone.
  2. Give it ~20ms.
  3. Change the new src in the element.
  4. Connect to MESN the Omnitone again.
hoch commented 6 years ago

One more finding: if you change .src on MediaElement while the element was being played, you'll get an error. I had to call .play() after I set the source, otherwise I get the silent output.

hoch commented 5 years ago

@bbaudel The example above does not work correctly any more because of the recent Autoplay policy change. Could you check it out? I would like to revisit this because the problem might have been fixed in Chrome.