search

IIIF / iiif-av

The International Image Interoperability Framework (IIIF) Audio/Visual (A/V) Technical Specification Group aims to extend to A/V the benefits of interoperability and the growing ecosystem of clients and servers that IIIF provides for images. This repository contains user stories and mockups for interoperable A/V content – contributions are welcome.
http://iiif.io/community/groups/av/
Apache License 2.0
13 stars 3 forks source link

Control of multi-channel audio #38

Open jronallo opened 8 years ago

jronallo commented 8 years ago

Description

Control of multi-channel audio, such as spatial information as to where the audio should come from. BL have a considerable number of ambisonic recordings (https://en.wikipedia.org/wiki/Ambisonics )

Proposed Solutions

Media-fragment spec has track fragment, no notion of channel. Could track be sanely used as channel? There appears to be explicit alignment between tracks in the webaudio spec and tracks in media-fragment spec so it would seem wrong to try to make a map to channels. Level 3 server could separate them into individual streams. Web Audio API supports arbitrary number of channels already. Could redirect/remix as necessary. Might be able to work from video element (?) http://webaudio.github.io/web-audio-api/#MediaElementAudioSourceNode Requires suitable CORS headers on the source video HTML5: https://www.w3.org/TR/html5/embedded-content-0.html#media-resources-with-multiple-media-tracks

Workaround is to encode channels as separate tracks and list them as such.

Additional Background

Source: BL workshop notes Interest: 50%

azaroth42 commented 8 years ago

Client side concern? Unless the UC is to download different channels separately, which I think is a different issue?

jronallo commented 8 years ago

Seems like this is similar to Dolby Audio and other audio with multiple channels. There is some support for Dolby in Microsoft Edge for playback, but I don't know how much of an API the browser supports for dealing channels. Other browsers could support multi-channel audio with the appropriate codecs in the future (if they don't already, though I can't find evidence that they do).

Is there any modern codec that supports Ambisonics? Otherwise you're probably left with a couple options. Remix the audio into a modern multi-channel codec that tries to approximate Ambisonics and that can be played back. Use the Web Audio API to take separate audio and mix them to try to reproduce the Ambisonic feel.

azaroth42 commented 8 years ago

I would propose defering work on multi-channel until we have the basics working and implemented

jronallo commented 8 years ago

Sounds good to me me to defer too.