Create an audio IIIF-AV Example Manifest

[joncameron]

Done Looks Like

• [x] an example IIIF av audio manifest is published
• [x] documentation is written if necessary about approach taken
• [x] issues are generated for player to use manifest.

[little9]

Is this specifically about creating an example audio manifest?

[adamjarling]

@joncameron To follow up on Jamie's note, should we put relevant notes under this ticket, or create a separate ticket about playing an Audio file from an example audio manifest? I have some info from Brian which might be helpful to log in comments.

[adamjarling]

@mcwhitaker Does the sample manifest below look like it's acceptable to play an audio track? @bkeese mentioned that the manifest files for video and audio should be very similar, so we created this trimmed-down version.

Note the difference is "type":"Audio" within "items". Brian also mentioned audio "items" could potentially be in the same array alongside video "items".

{
   "id":"http://dlib.indiana.edu/iiif_av/canvas/1",
   "type":"Manifest",
   "label":"Beginning Reponsibility: Lunchroom Manners [motion picture] Coronet Films",
   "description":"Lunchroom manners are taught by following a boy through his lunchroom experience",
   "height":360,
   "width":480,
   "duration":660,
   "content":[
      {
         "id":"...",
         "type":"AnnotationPage",
         "items":[
            {
               "id":"...",
               "type":"Annotation",
               "motivation":"painting",
               "body":[
                  {
                     "type":"Choice",
                     "choiceHint":"user",
                     "items":[
                       {
                          "id":"https://dlcs.io/iiif-av/wellcome/1/0128dccf-e2b8-4b0d-b41a-2d9edc6952f5/full/max/default.mp3",
                          "type":"Audio",
                          "label":"Florence Nightingale : greetings to the dear old comrades of Balaclava."
                       },
                        {
                           "id":"http://dlib.indiana.edu/iiif_av/lunchroom_manners/high/lunchroom_manners_1024kb.mp4",
                           "type":"Video",
                           "label":"High"
                        },
                        {
                           "id":"http://dlib.indiana.edu/iiif_av/lunchroom_manners/medium/lunchroom_manners_512kb.mp4",
                           "type":"Video",
                           "label":"Medium"
                        },
                        {
                           "id":"http://dlib.indiana.edu/iiif_av/lunchroom_manners/low/lunchroom_manners_256kb.mp4",
                           "type":"Video",
                           "label":"Low"
                        }
                     ]
                  },
                  {
                     "id":"http://dlib.indiana.edu/iiif_av/lunchroom_manners/lunchroom_manners.vtt",
                     "type":"Text",
                     "format":"text/vtt",
                     "language":"en"
                  }
               ],
               "target":"http://dlib.indiana.edu/iiif_av/canvas/1"
            }
         ]
      }
   ]
}

[adamjarling]

@little9 I pushed up a feature branch which includes some bare bones audio support, in a separate player from your video player.

https://github.com/avalonmediasystem/avarax-poc/tree/feature-play-audio

I'll be at Code4Lib the next 1.5 days, but back Thursday afternoon.

@davidschober This is the story I mentioned earlier in our meeting. Guessing it will get broken out or re-defined at some point, but some work is being done for audio support in this sprint...

[davidschober]

@joncameron I renamed to audio iiif-av

[mcwhitaker]

@adamjarling I suppose so. In Avalon, we don't choose between an audio or a video, which is what your manifest is representing; we choose between derivatives. But what you have here should be possible within a IIIF-AV manifest.

For POC, I think this is fine, since we are hard coding the choice made; but for Avalon in general we have 2 qualities for audio -- medium and high.

I would use something like this instead if we are trying to reflect Avalon's use cases:

{
   "id":"http://dlib.indiana.edu/iiif_av/canvas/1",
   "type":"Manifest",
   "label":"Florence Nightingale : greetings to the dear old comrades of Balaclava.",
   "description":"Florence Nightingale : greetings to the dear old comrades of Balaclava.",
   "height":50,
   "width":480,
   "duration":660,
   "content":[
      {
         "id":"...",
         "type":"AnnotationPage",
         "items":[
            {
               "id":"...",
               "type":"Annotation",
               "motivation":"painting",
               "body":[
                  {
                     "type":"Choice",
                     "choiceHint":"user",
                     "items":[
                       {
                          "id":"https://dlcs.io/iiif-av/wellcome/1/0128dccf-e2b8-4b0d-b41a-2d9edc6952f5/full/max/default.mp3",
                          "type":"Audio",
                          "label":"High"
                       },
                       {
                          "id":"https://dlcs.io/iiif-av/wellcome/1/0128dccf-e2b8-4b0d-b41a-2d9edc6952f5/full/max/default.mp3",
                          "type":"Audio",
                          "label":"Medium"
                       }
                     ]
                  }
               ],
               "target":"http://dlib.indiana.edu/iiif_av/canvas/1"
            }
         ]
      }
   ]
}

[adamjarling]

@mcwhitaker Great, thanks. Any updates you can continue to share which reflect real use cases and anticipated manifest structure will be super helpful.

[mcwhitaker]

I created the manifest for the Mahler symphony that we have in pawpaw and made the media files available: http://dlib.indiana.edu/iiif_av/mahler-symphony-3/mahler-symphony-3.json

[mcwhitaker]

@adamjarling For your reference, the Avalon navigation tree for the Mahler item is here: https://pawpaw.dlib.indiana.edu/media_objects/2j62s484w

[adamjarling]

@mcwhitaker Perfect. I'll update the manifest and code tomorrow.

[adamjarling]

@mcwhitaker Will the updated Mahler symphony example manifest structure be similar to how video manifests are structured as well? @little9

[mcwhitaker]

@adamjarling Yes, they are similar. The only example I have not put together is one where we have video and audio in the same manifest, but it will be similar to this one, only with the format and type tags reflecting the media type. I don't find an example of this in pawpaw but can create one for the proof of concept if you need one.

[adamjarling]

@mcwhitaker Yes, as we build out extended features into the Hyrax player, it'll help to have as many examples of real, use-case IIIF manifests, as soon as possible (but not to rush you obviously if decisions are still in flux). If we see similarities in structure between audio and video and shared audio/video manifests, it will help us abstract manifest parsing functions to a shared class. @little9

[bkeese]

It might be helpful at this point to consider manifests that contain both audio and video references (and images, 3D, ...). Audio and video manifests are just special cases of the generic IIIF manifest. Structure representation should be the same regardless of the type of media referenced.

[adamjarling]

@bkeese I think this is finally starting to make sense to me. A manifest representing both audio and video would essentially look like?

{
  ...
  "sequences": [
    { 
      ...
      "canvases": [
        {
          audio canvas,
          video canvas,
          any type of canvas (image, etc.)
        }
      ]
    }
  ] 
}

[bkeese]

Yep.

[davidschober]

@adamjarling this is looking done, correct? Do you want to check off those boxes and close it out? I'm not sure if we documented the approach.

[adamjarling]

@mcwhitaker @bkeese In our standalone player, is it same to assume that supplied manifests will always contain a 'sequence[] > canvases[] > content[]' hierarchy in the json? Or do we want to allow, say the 'content[]' node to live in the root level of the manifest as well ( ex. https://raw.githubusercontent.com/avalonmediasystem/avalon-poc-standalone/master/dist/lunchroom_manners_v2.json )?

[mcwhitaker]

@adamjarling I think this is a decision that we have to make. Basically, the "sequences" allow for multiple master files for the same media object (like the Mahler audio that we have been using as an example). But a media object with just one master file does not need it this tag.

We can decide that we will always have the "sequences" tag and assume that media objects with just one master file will be represented as a set of only one sequence. I see how that can simplify your code.

@bkeese Am I missing anything in this reasoning?

@joncameron @davidschober I believe a discussion with developers for a decision is important here if Brian agrees with me.

[bkeese]

@mcwhitaker I agree. And I think it is reasonable to always include the sequence layer.

[adamjarling]

Great, I think that will make our code less complex, if it's a reasonable assumption how manifests will be structured. @little9