search

IIIF / iiif-av

The International Image Interoperability Framework (IIIF) Audio/Visual (A/V) Technical Specification Group aims to extend to A/V the benefits of interoperability and the growing ecosystem of clients and servers that IIIF provides for images. This repository contains user stories and mockups for interoperable A/V content – contributions are welcome.
http://iiif.io/community/groups/av/
Apache License 2.0
13 stars 3 forks source link

Playback (audio and video) at different speeds #18

Open jronallo opened 8 years ago

jronallo commented 8 years ago

Description

Playback (audio and video) at different speeds. Multiplier on the speed. Open question as to whether its client or server.

Issue: If there are multiple video/audio on a single canvas at the same time, then the multiplier applies to the canvas not the stream, otherwise the time point of the canvas is wrong.

Variation(s)

Whether to resample audio or not. (control in scope?)

Proposed Solutions

Client application concern, not interop.

Additional Background

Source: BL workshop notes Interest: 100%

azaroth42 commented 8 years ago

What are the use cases that would require this functionality?

fractos commented 8 years ago

Somewhat notional, but for audio I can imagine a "sample server" which would allow arbitrary segments of audio streams to be selected, stretched or compressed and made available as wav/mp3/ogg/flac. This could have use cases in audio and video production software design. I'm sure there's an analogue example for video, but it makes more sense (to me) around music production.

jronallo commented 8 years ago

I'm not seeing the use case for a video and audio for the video to play at different speeds. This seems to be a pre-processing issue where they ought to be synced up.

If the video and audio are separate files then the client could control their playback at different speeds. These could all be controlled together at the canvas level or vary the playback speed in the client from one media file to the next.

The use case for the sample server could also be met with the client changing the playback speed of the sample. No need for the server to do that.

This seems to be either a client issue or out of scope.

fractos commented 8 years ago

I don't have access to the BL workshop notes right now, but I suspect that my comment doesn't fit this particular use case anyway. I can't seem to find one that actually matches the ability to separately manipulate audio and video speeds. They are connected in the BL notes?

azaroth42 commented 8 years ago

The use case could be that the audio and video come from different institutions and hence there isn't a pre-processing step.

jronallo commented 8 years ago

In the case where the audio and video come from different institutions (or are even separate files from the same institution) and their normal playback speeds would have the audio out of sync with the video, then the client could adjust playback speed. This could be a control in a client application to do that. That's feasible now.

What if the creator of the manifest knows about this issue? Can we provide a way through a Presentation manifest to indicate that the audio ought to play at 1.3x in order to sync up correctly? So there's no server-side change that needs to happen to the audio (and in this case the creator of the manifest may not have any ability to change the speed of the audio on the server), but the manifest does give a client the hint that for proper playback and syncing the audio ought to play at a particular speed. If we want to support something like this then is is a Prezi-API issue too?

azaroth42 commented 7 years ago

Propose closing as non API. The last case in @jronallo's comment is #33.

zimeon commented 7 years ago

I think this is actually a (small) subset of #33 as this might be described as "linear time alignment". In current thinking this is simply handled by a time stretch/squash via painting of the video and audio resources onto the canvas timeline (and thus a client issue). Could also be a feature of an AV bitstream API.