Hypertext audio/video navigation proposal ('hypertoc')

danbri commented 4 years ago

This is a proposal from Google to make it easier (e.g. in a voice assistant context) to navigate within audio and video content, providing a kind of "table of contents" with cues and links.

Example:

 "@context": "https://schema.org/",
  "@type": "HyperToc",
  "associatedMedia": 
 "@type": "AudioObject",
 "encodingFormat": "audio/mpeg",
 "bitrate": "320 kbps",
 "duration": "...",
"contentUrl: "https://ia800201.us.archive.org/27/items/sherlockholmes_otr/391106%20The%20Adventures%20of%20Sherlock%20Holmes%20-%20The%20Adventure%20of%20the%20Missing%20Submarine%20Plans.mp3",
"contentSize": "67M"
},
   "tocEntry": [ 
     {
     "@type": "HyperTocEntry",
    "url" : "...",
    "utterances": ["advert"]
    }
     {
     "@type": "HyperTocEntry",
    "url" : "...",
    "tocUtterances":[ "mycroft"]
    }
 ]
}

This would involve:

adding a type HyperToc (as a CreativeWork type)
adding tocEntry property of a HyperToc
adding a type HyperTocEntry which has an "url" representation of an offset into the media file (we could discuss other structures), as well as a list of textual "tocUtterances" that can be used in a voice assistant to navigate to this entry. Finally an optional "tocContinuation" pointer to other entries (using @id in json-ld, etc.), so that consuming applications could know which audio/video clip to continue playing if the end of an entry is reached. So new properties would be: tocUtterances (Text) and tocContinuation (a HyperTocEntry).

There are a few details left to work through but I wanted to surface our thinking here for discussion.

Related vocabulary includes Clip, PodcastEpisode, .

danbri commented 4 years ago

Given that this is a little drafty, I will file it in the brainstorming repo for now.

danbri commented 4 years ago

We (Google) have both an opensource proof-of-concept demo (parsing using https://github.com/google/schemarama) ready to share, and other signs of implementation seriousness, so I will move this back into the main repository.

github-actions[bot] commented 3 years ago

This issue is being tagged as Stale due to inactivity.

mkanzaki commented 1 year ago

Given that Dublin Core has dct:tableOfContents, would it be possible to extend the scope of tocEntry to cover gereric CreativeWork, and maybe to allow textual value?

I'm now implementing a large bibliographic database with schema.org vocab. While table of contens go to schema:description with leading label "目次:" (table of contents:) on the initial stage, some user comments suggested that a dedicated property for the toc would be more useful for the data consumption.

danbri commented 1 year ago

Sounds like a good idea! And your database sounds interesting too - will it be public?

On Thu, 8 Dec 2022 at 01:17, KANZAKI, Masahide @.***> wrote:

Given that Dublin Core has dct:tableOfContents, would it be possible to extend the scope of tocEntry to cover gereric CreativeWork, and maybe to allow textual value?

I'm now implementing a large bibliographic database with schema.org vocab. While table of contens go to schema:description with leading label "目次:" (table of contents:) on the initial stage, some user comments suggested that a dedicated property for the toc would be more useful for the data consumption.

— Reply to this email directly, view it on GitHub https://github.com/schemaorg/schemaorg/issues/2766#issuecomment-1341837104, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABJSGKNN47ASE22E3EDFGTWMEZK3ANCNFSM4TYJW4VQ . You are receiving this because you authored the thread.Message ID: @.***>

mkanzaki commented 1 year ago

Thanks! Yes, it will be public next spring :)

mkanzaki commented 1 year ago

I found it a bit disappointing that Schema.org v17 did not extend the scope of tocEntry. What steps are necessary to make progress? Here is an implementation example from the database mentioned above: https://ld.webcatplus.jp/snorql/?describe=https%3A%2F%2Fld.webcatplus.jp%2Fdata%2F169876

danbri commented 1 year ago

Let’s fix that!

On Sun, 28 May 2023 at 14:54, KANZAKI, Masahide @.***> wrote:

I found it a bit disappointing that Schema.org v17 did not extend the scope of tocEntry. What steps are necessary to make progress? Here is an implementation example from the database mentioned above:

https://ld.webcatplus.jp/snorql/?describe=https%3A%2F%2Fld.webcatplus.jp%2Fdata%2F169876

— Reply to this email directly, view it on GitHub https://github.com/schemaorg/schemaorg/issues/2766#issuecomment-1566149079, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABJSGJL7UEHJKUGRU33PETXINKJXANCNFSM4TYJW4VQ . You are receiving this because you authored the thread.Message ID: @.***>

github-actions[bot] commented 1 month ago

This issue is being nudged due to inactivity.

schemaorg / schemaorg

Hypertext audio/video navigation proposal ('hypertoc') #2766