The International Image Interoperability Framework (IIIF) Audio/Visual (A/V) Technical Specification Group aims to extend to A/V the benefits of interoperability and the growing ecosystem of clients and servers that IIIF provides for images. This repository contains user stories and mockups for interoperable A/V content – contributions are welcome.
Not all clients can play all video formats. We ought to provide information to allow clients to determine whether they can play a particular video or not. In HTML5 video the canPlayType() method is available to do this testing. It responds with different values based on the certainty of whether it can play the video. Besides determining that it can't play a video (empty string) it can respond with "maybe" if it would need to start playing the video to determine whether it can play it or "probably" if it is confident the media appears to be playable.
In cases where only a media type like "video/mp4" is provided browsers usually return "maybe" as the highest level of certainty. In order to have the possibility of getting a higher level of certainty and get a result of "probably" the codecs parameter needs to be included like: 'video/mp4; codecs="avc1.42E01E,mp4a.40.2"'
Proposed Solutions
We could provide a way to specify in an info.json both the MIME type and codecs of the video. Only providing the MIME type does not allow for as much certainty.
One of the reasons why a client may be able to play a video but be uncertain if only given a MIME type is that some formats include different levels of adherence to the specification. For instance MP4s have different levels which would result in different codecs (the part after the avc1 in "avc1.42E01E").
Variations
One of the issues with only providing a single MP4 codec value is that levels are determined via a matrix of adherence which can include the size (dimensions of the frames) as one of their limits among other things.
We could list just the highest level the server supports, but them some clients may still be able to request a lower level and be able to play the video.
Description
Not all clients can play all video formats. We ought to provide information to allow clients to determine whether they can play a particular video or not. In HTML5 video the canPlayType() method is available to do this testing. It responds with different values based on the certainty of whether it can play the video. Besides determining that it can't play a video (empty string) it can respond with "maybe" if it would need to start playing the video to determine whether it can play it or "probably" if it is confident the media appears to be playable.
In cases where only a media type like "video/mp4" is provided browsers usually return "maybe" as the highest level of certainty. In order to have the possibility of getting a higher level of certainty and get a result of "probably" the codecs parameter needs to be included like: 'video/mp4; codecs="avc1.42E01E,mp4a.40.2"'
Proposed Solutions
We could provide a way to specify in an info.json both the MIME type and codecs of the video. Only providing the MIME type does not allow for as much certainty.
One of the reasons why a client may be able to play a video but be uncertain if only given a MIME type is that some formats include different levels of adherence to the specification. For instance MP4s have different levels which would result in different codecs (the part after the avc1 in "avc1.42E01E").
Variations
One of the issues with only providing a single MP4 codec value is that levels are determined via a matrix of adherence which can include the size (dimensions of the frames) as one of their limits among other things.
We could list just the highest level the server supports, but them some clients may still be able to request a lower level and be able to play the video.
Additional Background
https://developer.mozilla.org/en-US/docs/Web/API/HTMLMediaElement/canPlayType
https://en.wikipedia.org/wiki/H.264/MPEG-4_AVC#Levels