w3c / media-and-entertainment

Repository for the Media and Entertainment Interest Group
56 stars 15 forks source link

Channels 2.0 #107

Open AdamSobieski opened 7 months ago

AdamSobieski commented 7 months ago

Introduction

Channels, in particular digital channels, can be much more than just video containers. In these regards, I am pleased to share some preliminary ideas and brainstorming towards channels 2.0.

Channels 2.0

Channel Descriptions

Channels could provide channel descriptions utilizing an extensible standard format (e.g., XML or JSON) to describe their features and capabilities to smart televisions and other devices, these resources referring to one or more real-time video containers.

Multimedia

Channel descriptions could reference a number of images and video clips, e.g., logos, at various resolutions and pixel densities.

Homepages

Channels could each have a homepage. As for whether these pages would be fully-customizable HTML pages or templated pages with image or video backgrounds and positioned text remains to be determined.

Customers could navigate to subpages about the particular content that they were watching from a channel's homepage. Similarly, these subpages would be either fully-customizable HTML pages or templated pages with image or video backgrounds and positioned text.

Channel Menus

Channels could each have a main menu. This menu could be available on a channel's homepage and made available while watching the channel, e.g., by pressing the up arrow.

Settings and Configurations

Channels could each provide settings and configuration for customers.

Similarly, individual (interactive) shows could have settings and configurations.

Sub-channels

Channels could have multiple sub-channels inside of them. There could be different types of sub-channels. Sub-channels could have attributes. Some sub-channels could be statically available while others might be dynamically available, or available only for a time.

For each channel, one sub-channel could have an attribute indicating it to be the main, selected, or default sub-channel. These sub-channels would be the video content displayed when customers utilized the traditional channel browsing technique with channel-up and channel-down buttons.

Grid-based Navigational Guides

Some subchannels would have attributes indicating them as being visible and as having scheduled content. Channels with one or more of these sub-channels could provide grid-based navigational guides for browsing them.

This guide could be navigated to from a channel's homepage. This guide could be accessed via a channel's main menu.

Smart televisions' grid-based navigational guides for viewing the content scheduled on multiple channels could version to be tree-based, each channel having a main, selected, or default sub-channel of content while also being expandable to reveal sub-channels, each of these similarly having a description and, perhaps, scheduled content.

User Feedback

Channels could provide elegant user experiences for customers to provide feedback pertaining to content.

Personalization

Channels could offer semi-personalized content, possibly via a special sub-channel. In this way, customers could receive recommended, personalized contents.

Content providers could interrupt personalized content to provide other content, e.g., breaking news content, intended for entire audiences.

Internationalization

Channels could provide content in multiple languages. Translations could involve video, e.g., digital lip sync, in addition to audio.

Interactive Video

Channels could provide customers with interactivity with respect to their content and advertisements.

See also: https://github.com/WICG/proposals/issues/33 .

Multi-device Interoperability

Channels could support multi-device interoperability.

Today, customers can use their mobile devices to scan on-screen QR codes. In the near-future, customers might be able to use a remote control to click on a hyperlink or hotspot to send content to or otherwise navigate to content on their mobile device.

Legacy Channels

Default behaviors with respect to new features and capabilities such as homepages, menus, and settings and configuration would need to be specified for legacy channels. In theory, channel descriptions (i.e., XML or JSON resources) could be automatically generated for legacy channels.

Use Cases and Scenarios

Artificial Intelligence

Multimodal dialogue systems could be implemented via interactive live-streaming video channels or via stored and persisted such content, e.g., files.

Children's Entertainment

Children's entertainment, e.g., Sesame Street and Blue's Clues, often presents audiences with opportunities for thinking, reflection, and learning. Shows' formats could be enhanced, e.g., with "games", utilizing the advancements to technology under discussion.

Education and Training

With new features and capabilities, in particular interactivity, educational television content for learners of all ages could be enhanced, e.g., including questions, quizzes, and exams within or between segments of instructional content.

Adaptive educational content, content varying based on learners' responses, can be provided via these technologies.

Government Transparency

With sub-channels, more government events, hearings, and proceedings, e.g., local, state, and federal, could be provided for audiences on public-interest channels.

In the United States, C-SPAN focuses primarily on the federal government, providing three simultaneous channels of content. C-SPAN focuses on the House of Representatives, C-SPAN 2 on the Senate, and C-SPAN 3 on various other government events, hearings, and proceedings.

With location-based personalization, those sub-channels most relevant to individual customers could be presented for their browsing, navigation, and consumption, e.g., content relevant to their cities, counties, states, and federal governments.

In the near future, local, state, and federal public meetings could be simulcast on enhanced digital television alongside Web-based video live-streaming, storage, and search services and solutions.

Music

With respect to music video scenarios, different kinds of music could be each be streamed simultaneously via sub-channels. Presently, a separate channel is required for each kind of music from a content provider.

News

Via channel-specific homepages, main menus, or via browsing sub-channels’ contents using grid-based navigational guides, customers would be able to check their local weather forecasts at any point in time while remaining on a news channel.

A channel’s sub-channels could be differing kinds of news content, e.g., weather, national, world, local, business, technology, entertainment, sports, science, health. There could also be a default "combination" sub-channel which would be understood as selecting and blending together segments of content from other specialized sub-channels.

Sports

With these new features, e.g., sub-channels, new possibilities for sports entertainment include, but are not limited to, customers being able to select from available camera angles and to browse replays.

Surveys and Opinion Polls

Surveys and opinion polls could be conducted using interactive video.

Implementation Discussion

Some preliminary implementation ideas are offered for discussion purposes.

Channels, Containers and Tracks

Each channel would be described by a channel description. A channel description would be a resource (e.g., XML or JSON) for describing its features and capabilities, referencing pertinent multimedia content such as images and video clips, describing its homepage, describing its channel menu, and indicating its (dynamic) set of containers.

Each container would be described by attributes and metadata.

Each container would contain a (dynamic) number of tracks. These tracks could be of kinds including audio, video, text, and data.

Hyperlinks or Hotspots

Each container could contain a track designated as being for hyperlinks or hotspots. Images or video clips could be provided in another track, these having the same width and height as their intended visible hyperlinks or hotspots, to be displayed while visible hyperlinks, hotspots, or menu items were displayed and/or hovered over.

Each visible hyperlink or hotspot would occur both spatially and temporally, e.g., in a rectangular screen region and available for a duration.

Visible hyperlinks or hotspots could:

  1. open and add a container to the channel,
  2. close and remove a container from the channel,
  3. pause, resume, rewind, or fast-forward a container in the channel, if it provides seeking capabilities,
  4. navigate the playback device to a container in the channel,
  5. signal a user's intent to open or navigate to other (Web) content, on another device, e.g., a connected mobile device,
  6. store a value to a new or existing setting, configuration, or data variable,
  7. ping a service provider,
  8. perform multiple actions simultaneously.

Menus

A menu can be considered to be a mutually exclusive set of visible hyperlinks or hotspots with (optionally or required) a default option in the event that a specified duration of time elapsed. As menus can be implemented with video content and overlayed hyperlinks or hotspots, the above capabilities may suffice to provide a number of key scenarios.

Towards providing accessibility features for hyperlinks, hotspots, and menus, menu items could have identifiers (e.g., option-1, option-2, option-3, and so forth), these being unique per menu, and, using these identifiers, menu items could be labeled and described by text content in tracks for purposes including hover-over effects and accessibility.

Invisible Hyperlinks or Hotspots

Other important elements to consider are invisible hyperlinks or hotspots. These would function in some, but not all, of the ways that visible hyperlinks or hotspots, intended for customer interaction, could but would function in a seamless manner and invisibly to customers. Invisible hyperlinks or hotspots would not, for instance, be of use for signaling a user's intent to open or navigate to other (Web) content, on another device (5, above).

Invisible hyperlinks or hotspots would have an additional capability:

  1. perform one or more actions conditionally, based upon the value of a setting, configuration, or data variable.

Invisible hyperlinks or hotspots would, then, be able to conditionally branch playback depending upon the values of devices', channels', or shows' settings, configurations, or data variables. Invisible hyperlinks or hotspots could be implemented with tracks in containers, e.g., those same tracks designated as being for visible hyperlinks and hotspots.

By adding the capabilities for invisible hyperlinks or hotspots to be able to conditionally act, e.g., branch, based on the availability and readiness of channels' containers, interactive content could prefetch content and also display content while awaiting other content. Prefetching is one technique for ensuring the seamless and smooth playback of interactive video content.

Time and Timers

Visible and invisible hyperlinks or hotspots could open, start, pause, resume, reset, and close named timers, these being special types of named variables. Alternatively, a more expressive syntax for conditional expressions could be considered and a keyword provided for referencing the current time, e.g., with millisecond resolution. This special keyword could have its value assigned to variables and also have variables' values subtracted from it in conditional expressions.

Conclusion

Thank you. I am looking forward to discussing these or any other ideas towards advancing the concepts of channels.

Channels 2.0.pptx

AdamSobieski commented 7 months ago

Here is some preliminary extensible markup representing a channel description. While a work in progress, it includes channel metadata (e.g., name and description), multimedia (e.g., logos), extensible menu data, and settings and configuration data.

This example shows a simple case where a described channel refers to one main video container.

<channel version="0.0" xmlns="...">

  <description>
    <data rel="name">
       <hypertext>
         <source type="text/plain" lang="en">Example</source>
       </hypertext>
    </data>
    <data rel="name-long">
       <hypertext>
         <source type="text/plain" lang="en">The Example Channel</source>
       </hypertext>
    </data>
    <data rel="description">
       <hypertext>
         <source type="text/plain" lang="en">This is a description of The Example Channel.</source>
       </hypertext>
    </data>
    <data rel="logo">
      <picture>
        <source width="..." height="..." media="..." theme="..." type="image/png" src="..." />
        <source width="..." height="..." media="..." theme="..." type="image/png" src="..." />
        <source width="..." height="..." media="..." theme="..." type="image/png" src="..." />
      </picture>
    </data>
    <data rel="background-image">
      <picture>
        <source width="..." height="..." media="..." theme="..." type="image/png" src="..." />
        <source width="..." height="..." media="..." theme="..." type="image/png" src="..." />
        <source width="..." height="..." media="..." theme="..." type="image/png" src="..." />
      </picture>
    </data>
  </description>

  <menu include-default-items="true">
    <item name="additional-item-1">
      <data rel="name">
         <hypertext>
           <source type="text/plain" lang="en">Additional menu item</source>
         </hypertext>
      </data>
      <data rel="icon">
        <picture>
          <source width="..." height="..." media="..." theme="..." type="image/png" src="..." />
          <source width="..." height="..." media="..." theme="..." type="image/png" src="..." />
          <source width="..." height="..." media="..." theme="..." type="image/png" src="..." />
        </picture>
      </data>
      <data rel="action">
        <script>
          <source type="text/javascript" src="..." />
        </script>
      </data>
    </item>
  </menu>

  <settings include-default-items="true">
    <setting name="custom-setting-1" kind="boolean">
      <data rel="name">
         <hypertext>
           <source type="text/plain" lang="en">A custom setting</source>
         </hypertext>
      </data>
      <data rel="description">
         <hypertext>
           <source type="text/plain" lang="en">This is a description of the custom setting.</source>
         </hypertext>
      </data>
    </setting>
  </settings>

  <containers default="main">
    <container name="main">
      <data rel="content">
        <video>
          <source width="..." height="..." media="..." type="video/mp4" src="..." />
          <source width="..." height="..." media="..." type="video/mp4" src="..." />
          <source width="..." height="..." media="..." type="video/mp4" src="..." />
        </video>
      </data>
    </container>
  </containers>

</channel>

I hope that the extensible example markup, above, clarifies upon the ideas towards advancing channels.

chrisn commented 6 months ago

Thanks Adam, and sorry for taking so long to reply. Some of this would seem to fall in scope of existing TV industry specifications such as DVB-I or ATSC, which are channel-oriented, moreso possibly than W3C.

I haven't done a full analysis though between the use cases you're describing and what's currently specified, e.g., whether there's a concept similar to sub-channels already in TV standards, or how DVB Companion Screens and Streams (CSS, not to be confused with W3C CSS!) might be used for the multi-device interoperability you describe.

Also related is this DVB study: https://dvb.org/news/dvb-seeks-input-on-object-based-media/

Some things I expect can be built today in interactive TV applications, e.g, the news, sports, and surveys use cases just by building using standards such as HTML and ECMAScript.

The data tracks and hyperlinks and hotspots examples sounds like they could be supported using Media Timed Events. Related work is MPEG ISO/IEC 23001-15 Carriage of web resources in ISOBMFF.

As an aside, this gave me the idea that could also be interesting to look at how some of these use cases could be supported in ActivityPub based video sites such as PeerTube. For example, PeerTube's documentation describes the extensions they've made to ActivityPub.

AdamSobieski commented 6 months ago

Thank you for the hyperlink to the DVB study and for the information about interactive video, MPEG ISO/IEC 23001-15: Carriage of web resources in ISOBMFF. Yes, I think that Media Timed Events could be of use for implementing both visible and invisible hyperlinks or hotspots.

Thinking about Web resources being embeddable in videos, perhaps index.js resources could be retrieved from within certain kinds of videos and run in sandboxed nested frames. As interesting, technical questions would include: (1) should the processing of videos' scripts' pause or stop when their videos are paused or stopped? (2) should videos' scripts be able to interact with their videos' playback control events? (3) should videos' scripts be able to interact with timed events in their videos' tracks?

It may also be the case that custom elements could be useful: <interactive-video>. Such custom elements could provide reusable client-side scripting logic across video instances, reusable-index.js, and process Web-enhanced video formats and their tracks.

Yes, I think that "channels 2.0" ideas and brainstorming could be useful for PeerTube – excellent project, peer-to-peer video streaming and decentralized social-media integration!