thehabes
commented
8 years ago When referring to Audio pieces within the manifest, it was easy for our annotations to be somewhere "on" a piece of audio. We stored audio resources as an oa:Annotation and made the resource of that annotation the actual mp3 audio file, which made it very easy to connection annotations and annotation lists to.
It would be nice if a/v resources could fit into the specs this way. Making them an annotation was a way to cheat. If something existed like sc:Sound that worked like an sc:Canvas in the specs, it would work rather smoothly (see the example below on way we did this to make it work).
So the main point of discussion that comes from this is how exactly would an audio/video resource be described and treated? What would be a proper motivation and @type? What's the consistent and proper way to talk about how an annotation is "on" a resource with the dimension of time (and remember, time is probably an interval not an exact moment).
Ex:
//The audio resource { "@id": "/some/audioResource", "@type": "oa:Annotation", //this was the hacky part, we would like this to be sc:Sound or something "label": "sound file", "motivation": "performance", //what should this be? "resource": { "@id": "media/audio/audioFile.mp3", "@type": "dctypes:Sound", "format": "audio/mpeg" } }
{ "@id": /some/audio/annotation/ID, "@type": "oa:Annotation", "label": "first five seconds", "on":[ "/some/audioResource#t=0,5.00"]
}
For the variation aspect of this, we box an area of a sheet music canvas when the notes are playing. So when the first five seconds of the music notes are boxed by this annotation, all we had to do was let this annotation know it was "on" two resources at once:
{ "@id": /some/audio/annotation/ID, "@type": "oa:Annotation", "label": "first five seconds", "on":[ "/some/audioResource#t=0,5.00", "/some/music/sheet#xywh=546,485,186,382"] }
So when the canvas was loaded to the screen, so was the music. Since both resources were active, when this annotation was hit, it knew to be drawn to the screen given the dimensions during a specific TIME interval the music was playing. If the canvas is clicked on, the annotation knows to load the audio to a specific time. If the audio is loaded to a specific time, it knows to make a specific drawn annotation active, and in this way we connected audio, time and drawn visuals.
jronallo
Description
Ability to refer to a point in time. Standard syntax for addressing points. Want to say this chord at this point. SMPTE time codes. Might also need sample level access on audio files -- this 10 sample window. Frequency content of the window. SMPTE, Samples and Time. HTML5 video not great at supporting sample accuracy, get nearest key frame based on time. Could annotate with sample accuracy, but just works as metadata.
Variation(s)
Additional Background
Related IIIF: Image API Region, info.json, # Fragment on Canvases Media fragments support NPT, SMPTE (https://www.w3.org/TR/media-frags/#naming-time) and note no problem combining t=123.45&xywh=1,2,3,4 (https://www.w3.org/TR/media-frags/#processing-name-value-lists ) Media fragment spec does not specify mandatory or canonical order of parameters. If using in IIIF we’d probably want to mandate (or at least recommend a canonical order) to avoid the creation of duplicate URIs for the same thing.
Film folks need frame specific references. Can build a more advanced player that supports it. Frames would need to be relative to original scan of the film. Transcoding gets … "exciting". e.g. freeze on a frame where the original physical film has a scratch.
SMPTE allows reference to a frame in the format: hh:mm:ss:ff eg: 00:01:30:02 is the third frame in 1 minute 30 seconds. (SMPTE spec is closed, but https://en.wikipedia.org/wiki/SMPTE_timecode describes)
Source: BL workshop notes Interest: 100%