jronallo commented 8 years ago

Description

Custom media players for file types that aren't directly supported, or subhosted via youtube. Something distinct from just a video file, but actually an iframe.

Variation(s)

Annotations on custom content (e.g. annotate youtube) -- requires enough API to get time update events
Control the embedded player from surrounding UI (eg, side-by-side linking to common points in two alternate videos) -- requires enough API to start/stop and seek. Description: Annotate content where the content can't be exposed to the annotation tool. Control of a player that has the bits, and annotation part can't see it. Esp for commercial content providers. [related to iframe player]
Player doesn't understand the content, but can be annotated.
Additional Background

Source: BL workshop notes Interest: 90%

Technical discussion

Issue: postMessage() to communicate with iFrame; maybe standardize interface? Would be super valuable. Look at youtube API. Likely need a way to describe the interface supported. This may need a separate spec; see quick survey of state of art. Initial implementations could probably hardcode use of a few site-specific APIs such as YouTube’s to demonstrate basic effectiveness.

[If folks are interested in further followup on iframe embedding standardization, see https://github.com/iframe-player/iframe-api-docs where @brion will be collecting details and trying to organize something. Feel free to add ideas there as issues. Will try to make sure editable pages are up or can add people.]

Use of profile for the API to use, e.g. for youtube? Multiple remote services, like multiple formats. Toss in a YouTube link, and a local MP4, and let the player decide which to load based on availability.

Scraping YouTube for the raw mp4 would be so much easier in theory… but unreliable, and won’t work with content protection, etc. Would also violate their ToS.

Stability of iframe URLs, parameters? (Seem to mostly be stable for YouTube and Vimeo.) Extra requirements and oddities -- youtube says “must” be 200x200px (see https://developers.google.com/youtube/iframe_api_reference) What’s that mean? :D

Comparing with http://wellcomelibrary.org/iiif/b16748967/manifest example:

         "rendering": [
            {
              "@id": "https://s3-eu-west-1.amazonaws.com/wdl-video-open/mp4/80e62970-d26a-403e-8aa8-480de6d22495.mp4",
              "format": "video/mp4"
            },
            {
              "@id": "https://s3-eu-west-1.amazonaws.com/wdl-video-open/webm/80e62970-d26a-403e-8aa8-480de6d22495.webm",
              "format": "video/webm"
            }
          ],

Let’s imagine extending that to also include a remote link and an API reference, something like:

            {
              "@id": "https://www.youtube.com/embed/dQw4w9WgXcQ?enablejsapi=1",
              "format": "text/html",
              "profile": "https://developers.google.com/youtube/iframe_api_reference",
            }

Or more simply with a fake type, where the player would know how to add the api-specific iframe parameters:

            {
              "@id": "https://www.youtube.com/embed/dQw4w9WgXcQ",
              "format": "x-video/youtube"
            }

We might want to look to the service pattern to embed the YouTube @id & profile as above (see image annotation resources in the Presentation API). [This could avoid needing a separate info.json for every youtube video you want to embed?]

Note use case given where YouTube video was used preferably but a local file needed to be provided for China. Providing both the embed link and the .mp4, in preferred order, should work same as including an .mp4 and a .webm: player can know that YouTube won’t work locally and skip over it.

John Dyer of MediaElement.js was working on an HTML5 Video Wrapper for the YouTube and Vimeo embedded player APIs (seems to still be maintained as part of MediaElement.js). This is very similar data model, taking the embed URL in a element just like a raw mp4 or webm file reference. Blog post: http://johndyer.name/html5-video-wrapper-for-youtube-and-vimeo-api-mediaelement-js/, Example: http://mediaelementjs.com/examples/?name=youtube
The shim itself: https://github.com/johndyer/mediaelement/blob/master/src/js/me-shim.js

Popcorn.js has something similar: http://popcornjs.org/popcorn-docs/media-wrappers/

azaroth42 commented 8 years ago

Implementation issue, I think? I think our primary (but not exclusive) focus should be on situations when the stack is controlled. If (favorite video hosting site) doesn't let you directly request time ranges for a video stream, then to me the best way of convincing them to implement that would be demonstrating how valuable it is rather than by trying to work around their current technology as a third party

jwd commented 8 years ago

I agree this is not first priority to deal with, but there are many situations in the A/V world where a content provider does not want to (or is not able to due to licensing) provide direct access to the bits via HTTP, and an embeddable player that is tightly integrated with their authorization and streaming infrastructure is the best we're going to get...

workergnome commented 7 years ago

I think of youtube embeds (and the like) as web applications, not videos. They're applications that display videos, but they're not a video. I think it might be worth thinking if there's a way to reference them as annotations or as content on a canvas/timeline...but as HTML, not as video.

tomcrane commented 7 years ago

@workergnome See also https://github.com/IIIF/iiif-av/issues/46:

{
  "type": "Annotation",
  "target": "http://example.org/canvas/1",
  "body": {
    "id": "http://youtube.com/video",
    "service": {
      "id": "http://youtube.com/video/api",
      "profile": "uri-for-youtube-api",
      "more": ["params", "go","here", "too"]
    }
  }
}

It then becomes an impl detail for the client. Client recognises the service profile "uri-for-youtube-api" and fires up its YouTube plugin to render the content.

The anno could easily be a temporal fragment of the canvas (or spatial, or both) and could be extended to reference a fragment of the YouTube target as well, if you don't want the whole.

    "selector": {
      "type": "FragmentSelector",
      "conformsTo": "http://www.w3.org/TR/media-frags/",
      "value": "t=30,60"
    }

The Presentation API then can say all it needs to very succinctly, and is not tied to implementation changes in Youtube/Vimeo/SoundClound/X/Y/Z APIs. It's no more implementation work for the client developers to do either (Mirador, UV, clientX, clientY will still need to implement each third party support regardless of how much of each of their descriptions are imported into the Presentation API - preferably none). Institutions that host all their video on YouTube will pick a viewer that has support for YouTube content, i.e., recognises "profile": "uri-for-youtube-api", and handles it. Viewers can share components that turn a YouTube profiled anno into a rendered video.

workergnome commented 7 years ago

I like it. Are we worried about versioned APIs for, say, Vimeo? Can't decide if tying the profile to a version is a +1 or a -1.

Also, have you looked into http://oembed.com? I know both Youtube and Vimeo support it, and while it's not explicitly LOD, it might be a useful reference for this sort of thing.

jronallo commented 7 years ago

@workergnome Could look at how other players are handling this. Some players have support for playing YouTube and Vimeo embeds: https://github.com/Selz/plyr#embeds

And video.js has had this working via plugins: https://github.com/videojs/video.js/wiki/Plugins#playback-techs-support-media-from-different-sources

tomcrane commented 7 years ago

I think you can't put a version in the profile, because you can't control when YouTube changes its APIs. You'd have to keep updating your manifests with whatever YouTube was doing. I think it's enough to say "this video is on YouTube, here's its YouTube URI" and let the client take it from there, interacting with YouTube as it sees fit.

@edsilv is looking at oEmbed for the Universal Viewer. The oEmbed interaction happens when someone embeds a viewer configured to look at a particular IIIF resource. When the viewer spins up, reads the manifest, and sees that it needs to show a YouTube video, the oEmbed interaction will have already happened, you've got your code running in the embed target so it can pull in whatever plugin it needs at that point. In the video example here - http://oembed.com/#section5 - instead of a YouTube embed it would be a Universal Viewer embed chunk of HTML, that only later starts talking to YouTube, using the techniques @jronallo mentions in the previous comment.

workergnome commented 7 years ago

@jronallo: Yeah—it looks that a IIIF viewer using either of those libraries could be implemented with the provided metadata that @tomcrane suggested.

@tomcrane: I'm thinking how you say "this is a youtube video" in a way that doesn't require parsing the provided URL. Sort of a moot point, since neither Vimeo nor Youtube provide versioned APIs in any referencable way I can see.

I mention oEmbed because it's a programmatic way to get a bunch of the metadata that would be useful to a viewer trying to display a video, like poster frames and the like. I like the idea that UV could provide a oEmbed Profile—I was thinking of it more as a programmatic data source a viewer could use, or as a set of parameters that would be both easy to include into a service description and that would reuse existing metadata from providers. It's a good example of how this problem has been solved before.

tomcrane commented 7 years ago

@workergnome in the annotation example above (originally from #46), it's the service profile that the client uses to determine that it is dealing with a video hosted on YouTube:

"service": {
      "id": "http://youtube.com/video/api",
      "profile": "uri-for-youtube-api",
      "more": ["params", "go","here", "too"]
    }

In my maybe over-simple scenario I'm suggesting that the YouTube URL of the video, and the profile string to tell the client that it is a YouTube video, maybe all that you need. The client might interact with the API, but the publisher of the manifest (via the annotation) doesn't have to tell it how to do that.

Looking at the links @jronallo posted, I see that video.js plugin does it with a fake video/youtube type:

https://github.com/videojs/videojs-youtube

So maybe

{
  "type": "Annotation",
  "target": "http://example.org/canvas/1",
  "body": {
    "id": "http://example.org/video1",
    "type": "Video",
    "format": "video/youtube"
  }
}

If the client can play a YouTube video, via whatever library it wants to use, then it can process that. Not sure about the fake media type though. Feels a bit wrong. A service profile is an opaque URI that IIIF servers and clients can agree has a particular significance without pretending to be something else.

bvibber commented 7 years ago

I agree a version isn't needed, as long as the URLs stay consistent the important part is to know which provider's current API to load up and feed it to.

For content types -- consider text/html for iframe embeds, as that will be technically correct and can signal the need to check for an API service profile. (In theory I believe you can embed the iframe without knowing about the YouTube API details, that's mainly needed for programmatic control...)

azaroth42 commented 7 years ago

Redescribe for youtube (etc) case as more useful to more people

bvibber commented 7 years ago

Note that while you can easily embed any URL into an iframe, since the surrounding viewer app needs to be able to control playback, an appropriate API must be known or the video won't start/stop/seek the way it's supposed to. So rather than supporting format: "text/html" I think it may be better to go with explicit 'fake' mime types, or else services attached to a default file (as with images).

The viewer would recognize either the format (video/youtube) or the service profile, and know that it needs to load the YouTube API JavaScript, initialize an iframe, and queue up the video. A non-web viewer could instead choose to use a native YouTube player widget, such as on Android or iOS. In either case, the actual URL to the iframe doesn't need to be known -- the id URL can be treated as an opaque identifier.

A couple possible modelings...

Using a fake mime type:

                {
                  "id": "https://www.youtube.com/watch?v=3j-ktiYTTds",
                  "type": "Video",
                  "format": "video/youtube"
                },

Using a service:

                {
                  // no fallback -- loading this URL into an iframe without API would fail
                  "id": "https://www.youtube.com/watch?v=3j-ktiYTTds",
                  "type": "Video",
                  "format": "text/html",
                  "service": {
                    "id": "https://www.youtube.com/watch?v=3j-ktiYTTds",
                    "profile": "https://developers.google.com/youtube/iframe_api_reference" // or other agreed-upon API URL
                  }
                }

One could also have multiple service URLs in the same entry:

                {
                  // a default mp4 or webm:
                  "id": "https://upload.wikimedia.org/wikipedia/commons/transcoded/2/26/Knowledge_for_Everyone_%28no_subtitles%29.webm/Knowledge_for_Everyone_%28no_subtitles%29.webm.360p.webm",
                  "type": "Video",
                  "format": "video/webm",
                  "service": [
                    {
                      // IIIF A/V API for auth or, in future, multi formats/resolutions
                      "id": "https://commons.wikimedia.org/iiif/av/Knowledge_for_Everyone_%28no_subtitles%29.webm",
                      "profile": "http://iiif.io/api/av/0/profiles/level0.json"
                    },
                    {
                      // Youtube API
                      "id": "https://www.youtube.com/watch?v=3j-ktiYTTds",
                      "profile": "https://developers.google.com/youtube/iframe_api_reference"
                    },
                    {
                      // MediaWiki iframe embed API
                      "id": "https://commons.wikimedia.org/wiki/File:Knowledge_for_Everyone_%28no_subtitles%29.webm",
                      "profile": "https://www.mediawiki.org/api/iframe-embed"
                    }
                  ],
                },

I'm not sure if that makes sense for picking alternate services, or if services are supposed to be specific to one version. It may be best to model the above via types:

        "body": {
            "type": "Choice",
            "items": [
              {
                // "plain HTML5 video" -- raw mp4 or webm file upgradeable via IIIF A/V API
                "id": "https://upload.wikimedia.org/wikipedia/commons/transcoded/2/26/Knowledge_for_Everyone_%28no_subtitles%29.webm/Knowledge_for_Everyone_%28no_subtitles%29.webm.360p.webm",
                "type": "Video",
                "format": "video/webm",
                "service": {
                  "id": "https://commons.wikimedia.org/iiif/av/Knowledge_for_Everyone_%28no_subtitles%29.webm",
                  "profile": "http://iiif.io/api/av/0/profiles/level0.json"
                },
              },
              {
                // Youtube API
                "id": "https://www.youtube.com/watch/?v=3j-ktiYTTds",
                "type": "Video",
                "format": "video/youtube"
              },
              {
                // MediaWiki iframe API
                "id": "https://commons.wikimedia.org/wiki/File:Knowledge_for_Everyone_%28no_subtitles%29.webm",
                "type": "Video",
                "format": "video/mediawiki"
              },
           ]
        }

IIIF / iiif-av

Custom media players for file types/delivery platforms that aren't directly supported #27

Description

Variation(s)

Additional Background

Technical discussion