UI for linear advertisements

[littlespex]

Requirements

Provide a set of components for displaying and controlling linear advertisements. The controls layer would provide functionality to:

• Play/Pause ad playback (SSAI and CSAI)
• Prevent UI based seeking
• Provide a click-thru element
• Display ad details like "Ad X fo Y"

Proposal 1

Create a <media-ad-controller> element which bootstraps the ad SDK and act as a wrapper element for a separate set of ad controls. The ad ui would use the standard media-* controls, i.e. buttons like play and fullscreen, and the <media-ad-controller> would intercept the MEDIA_X_REQUEST events and perform any ad specific logic.

<media-controller>
  <video slot="media"></video>
  <media-control-bar>
    <media-play-button />
    <media-mute-button />
    <media-volume-range />
    <media-time-range />
    <media-pip-button />
    <media-fullscreen-button />
  </media-control-bar>
  <media-ad-controller>
    <media-ad-info />
    <media-control-bar>
      <media-play-button />
      <media-mute-button />
      <media-volume-range />
      <media-time-range />
      <media-pip-button />
      <media-fullscreen-button />
    </media-control-bar>
  </media-ad-controller>
</media-controller>

Pros

• Keeps the ad logic and controls contained to the <media-ad-controller> instead of having each component be aware of what type of media is playing and switching its logic accordingly.
• A completely custom "ad view" can be achieved entirely in mark down by simply adding the desired controls as children of <media-ad-controller>. With a single control bar for both modes, the playback mode would need to be made available in a way that CSS and JS could be written to customize the control for the given mode.
• In the future, if the library is ever delivered in an "unbundled" manner (individual ES6 modules), having that ad logic in a standalone component would allow for the ad code to be omitted entirely when not needed.

Cons

• More verbose markdown at the page level.
• The ad controller will need to be aware of the video element (ad SDK usually require a reference). Since the media element is not a child of <media-ad-controller> the reference would need to be injected some how:
  • Perhaps in a similar way to how state is passed down?
  • Sibling query selector?

Proposal 2

Similar to the first proposal but with the <media-ad-controller> wrapping the entire UI.

<media-controller>
  <media-ad-controller>
    <video slot="media"></video>
    <media-control-bar>
      <media-play-button />
      <media-mute-button />
      <media-volume-range />
      <media-time-range />
      <media-pip-button />
      <media-fullscreen-button />
    </media-control-bar>
  </media-ad-controller>
</media-controller>

Pros

• Less verbose markdown at the page level.
• Similar future "tree shaking" benefits as proposal one

Cons

• No ability to create custom ad views in markdown. At minimum the <media-ad-controller> would need to have a mode attribute that could be used to create CSS selectors, i.e:

  media-ad-controller[mode="ad"] media-time-range { 
   pointer-events: none; 
  }

Other considerations

The current set of web components revolve around a stable public API: HTMLMediaElement. There is no similar standard for ad playback. When using a raw playback engine likehls.js or dash.js the ad implementation would need to be done separately. This could easily be encapsulated using <media-ad-controller> as a base class, and creating a ad-vendor specific element such as <ima-ad-controller>. For full featured players like Shaka and Video.js, ad APIs are provided directly by the SDK, or via a plugin. In those cases there would need to be a element that translates the player specific APIs into a normalized API that <media-ad-controller> could use. Similar to the way <youtube-video> provides an API to mirror HTMLMediaElement.

[mmcc]

We talked about this in Video Dev Slack, but a 3rd proposal is to mix the two a bit more. I like the higher level goals in Proposal 1, but I agree that having sibling elements with this much of an effect on each other feels a little awkward. My 2 cents would be that we go towards something like:

<media-controller>
  <media-ad-controller>
    <video slot="media"></video>
    <media-control-bar>
      <media-play-button />
      <media-mute-button />
      <media-volume-range />
      <media-time-range />
      <media-pip-button />
      <media-fullscreen-button />
    </media-control-bar>
    <media-ad-control-bar>
      <media-play-button />
      <media-mute-button />
      <media-volume-range />
      <media-time-range />
      <media-pip-button />
      <media-fullscreen-button />
    </media-ad-control-bar>
  </media-ad-controller>
</media-controller>

The only special feature of the <media-ad-control-bar> would be that it would namespace the underlying children's events as they bubble up, allowing the <media-ad-controller> to behave in the same way as proposed, but with the additional benefit gained from fully wrapping the underlying media element.

Thoughts?

[heff]

I'm liking the general idea and conversation direction here. 👍

With custom elements, can a slotted element (media) be inside a child or does it need to be a direct descendent? I couldn't find a quick answer, but that's obviously a key detail here. Hypothetically, we could have media-ad-controller wrap media-controller if that makes it possible and simplifies some logic.

Ads...so much to wrap your brain around. Does this approach apply equally to

• Server-side ads
• Ads that replace the source content in the media element
• Ads that overlay the source content

Does it matter? Are all of those still a concern?

[heff]

It doesn't appear slots in children work today. So in the first few examples, either media-ad-controller would need to live in the media slot itself (duplicating the API of a media element) or pass the media to the media-controller some other way.

But that's pushing me to want to explore wrapping media-controller with media-ad-controller.

<media-ad-controller>
  <media-controller>
    <video slot="media"></video>
    <media-control-bar>...</media-control-bar>
  </media-controller>
  <!-- ad control bar -->
  <media-control-bar>...</media-control-bar>
</media-ad-controller>

What are the downsides of that? Feels a little odd to not have media-controller be the consistent top-level element, but I'm not totally sure of the specific problems that creates.

An upside is media-controller and all of the other core elements would have less need to expand to be ad-aware.

[mmcc]

But wouldn't that approach make the event bubbling stuff a bit weird? What if we did something similar to what I proposed for media-ad-control-bar for media-controller? Just having that base element extend media-controller feels like it could both simplify the markup and the component itself.

[heff]

But wouldn't that approach make the event bubbling stuff a bit weird?

Can you say more about weird?

What if we did something similar to what I proposed for media-ad-control-bar for media-controller?

Yeah that's definitely an option too. A sub class that bolts on ad functionality to the controller. We'd still need to distinguish between control bars in that case.

[mmcc]

Can you say more about weird?

Doesn't it feel weird that a parent element would affect the bubbling of events to a child? I'm not even 100% sure how we'd even handle that, but just looking at that markup and thinking of the implication makes me squint a little.

We'd still need to distinguish between control bars in that case.

Totally, I think I'd still fall back to the special media-ad-control-bar outlined.

[heff]

Doesn't it feel weird that a parent element would affect the bubbling of events to a child? I'm not even 100% sure how we'd even handle that, but just looking at that markup and thinking of the implication makes me squint a little.

I'm probably not seeing the whole picture but I'm not sure why the parent element (ad controller?) would need to affect bubbling. I think the ad controller would listen to the events it needs from media-controller, but wouldn't need to do anything with the other events.

Relationship-wise, it seems like the ad controller needs to know about media events, but the media-controller doesn't need to know about ad events. The media-controller gets controlled by the ad-controller as needed, maybe similarly to how the IMA ad plugins work with the video element. The video element does its thing, and the ad framework works around it.

It's pretty much exactly like when you wrap optimus prime in bigger optimus prime.

[littlespex]

@heff Good catch with the child slots issue. Proposal 2 will not work as written. The media-ad-controller will need to wrap the entire UI, either by containing the original media-controller as a child, or extending it as a base class.

I'm probably not seeing the whole picture but I'm not sure why the parent element (ad controller?) would need to affect bubbling.

If the ad controller only wraps the ad specific controls (proposal 1), it would be able to intercept the MEDIA_REQUEST_X events fired by its children and perform ad specific logic, and then prevent the events from bubbling up to the standard media-controller.

If the ad controller wraps the entire UI, there would need to be a new set of controls that fire ad specific events so that the ad controller could differentiate between the content actions and ad actions. This wouldn't be too difficult as the new ad controls could simply extend the existing controls and override the event emission with the new ad specific events.

All this being said, I think the much bigger problem is: Where does the ad SDK integration live, and how do the web components interact with it.

Let's use Video.js and an example. To get IMA ad playback we might use a plugin such as videojs-ima. This would provide a source for the ad events needed to render the ad ui, adstart, adend, etc, but how would the web components consume those events? The player instance lives in JS, and the events are 'emitter' style events, not DOM events. Would we need to have custom media element components for every player framework that dispatch a set of standard ad DOM events?

[heff]

Got it, I see now. Thanks for clearing that up.

One thing I'm feeling through all of this is media-controller probably needs a set of defined ad events, to at least be able to proxy ad state. But otherwise, I think we might need to break up the different ad situations into their own problem sets, because they might need to be solved differently. Let me try to enumerate them...

Server-side ads

The media element itself needs to be able to emit standardized ad events at minimum, if not other functions like blocking scrubbing, handling clicks on the ad, and sending ad click reports.

Client-side ads

Client-side ads via an ad-supporting media element, e.g. <theo-player> or <video-js>

The media element needs to emit standardized ad events, or it can just handle the ad UI internally (not ideal, but an option).

Client side ads via a custom media-chrome element integration

All the options we've discussed... but new questions

• Is a downside that you could simply delete the ad-controller element to remove ads?
• If the video el lives in the shadow dom, can you still pass it to the IMA sdk that's not in the shadow dom?

After laying that out...I think ad playback might need to live in the media element, while ad UI can live in media-chrome. Everything gets pretty complex if we try to let the ad integration live in the UI layer. Whereas if we can separate ad playback from ad UI, then media-chrome can still play to its strengths. And we're going to have to support media-els-with-ads either way, unless we're going to force you to use the media-chrome ad integration, and I don't love that.

At its simplest it might look like <video-ima> as a media element that also emits standardized ad events, but doesn't worry about UI. Then media chrome passes state like media-ad-playing, that the UI can be built around.

The downside would be that you can't just easily add ad support at will to any media element like <hls-video> or <youtube-video>. Is there a way to make that easy? Or is that ok, at least for v0 of media-chrome? Am I missing anything else major?

[heff]

If the ad controller wraps the entire UI, there would need to be a new set of controls that fire ad specific events so that the ad controller could differentiate between the content actions and ad actions.

Had more thoughts on this one, incase UI level integrations are still a direction we need to go. I don't know that we would need ad specific UI events here. The ad-controller could add event listeners to media-play-request and other events, and preventDefault to block further actions when needed (similar to how the IMA SDK works). This is a benefits of having every request funneled through the controller now rather than UI elements talking directly to the media element. Then the ad-controller can rely on its own current media-ad-playing state to determine if UI events should apply to the ad or the media.

There does seem to be a key long term implication to consider here, which is whether or not <media-controller> base media states (e.g. media-duration, media-current-time) are updated to reflect the ad state or maintain the primary media's state. For example if media-current-time is updated to reflect the ad's current time, then external elements now need to be "ad-aware". For example an external text transcript that's following along with the video. Or just a captions element. I think the answer is we don't want that.

(this is very complicated... sorry for the all the long exploratory comments)

Pulling from various ideas, here's a proposal...

1. Add first-class ad states to media-controller

<media-controller media-ad-playing media-ad-duration="100" media-ad-current-time="10" ...>

2. media-controller listens for ad-events on the media element, and updates ad state accordingly

<media-controller media-ad-playing>
  <jw-player></jw-player> <!-- media element with internal ads -->
</media-controller>

3a. A UI-level client-side ad integration can wrap media-controller and update its ad state accordingly.

The ad integration element can listen for a media-play-request on media-controller and preventDefault to block media playback and play an ad.

<media-ima-ads>
  <media-controller media-ad-playing>
    <video></video>
  </media-controller>
</media-ima-ads>

3b. A UI-level client-side ad integration could also wrap the media element, taking the the place of the media and exposing the html5 video API.

<media-controller media-ad-playing>
  <ima-video slot="media">
    <hls-video></hls-video>
  <ima-video>
</media-controller>

4a. For all cases, controls could then be built with a special control bar (Matt's idea) that translates events/states

<media-controller>
  <video></video>
  <media-control-bar> <!-- control bar for media -->
    <media-play-button />
  </media-control-bar>
  <media-ad-control-bar> <!-- control bar for ads (special) -->
    <media-play-button />
  </media-ad-control-bar>
</media-controller>

4b. For the media-controller wrapping ad integration case, controls could also optionally live outside the media-controller, and not require a special control bar.

<media-ima-ads>
  <media-controller media-ad-playing>
    <video></video>
    <media-control-bar /> <!-- control bar for media -->
  </media-controller>
  <media-control-bar /> <!-- control bar for ads (not special) -->
</media-ima-ads>

Thoughts on that?

[littlespex]

Thinking about the overall design again with @heff, another approach that doesn't require new elements or DOM hierarchy would be:

The media element (slot="media") encapsulates the content vs ad complexity, leaving the media controls and media-controller as is. The media-play-button fires the "mediapauserequest" event and the media-controller calls this.media.pause(), as it does currently. The custom media element would be responsible determining whether to call pause() on the video tag or the ad manager.

The custom media element would need to fire a new event signaling the start/end of ad playback, an adbreak event with a boolean data payload, or perhaps separate adbreakstart and adbreakend events. The controller would listen for that event and set a boolean attribute on itself so that ad specific CSS rules can be written for the child elements:

<style>
  media-controller[media-ad-break] media-progress-range {
    pointer-events: none;
    background-color: #FFCC33;
  }
</style>
<media-controller media-ad-break>
  <player slot="media" />
  <media-control-bar>
    <media-progress-range />
  </media-control-bar>
</media-controller>

The display of ad data could be handled in a couple of different ways. For a "YouTube style" display, when the ad break starts the custom media element would fire a durationchange event and populate its currentTime and duration properties with ad specific values:

For a "Hulu style" display, the custom media element would need to provide additional pieces of state to allow for a separate rendering of ad time:

• media-ad-current-time
• media-ad-duration

It would be the custom media element's responsibility to determine if media-ad-duration represents the duration of an individual ads, or the duration of the entire break. The custom media element would also need to populate media-ad-url for a "Learn More" button / click-thru element.

The last piece would be a single new API method on the custom media element for skipAd().

[heff]

Thanks Casey, this all sounds good. I think we could handle both cases and let the UI dev choose, if we provide the details of the adbreak, and always rely on media-duration/currentTime to represent the current time.

• media-adbreak-duration="90.0"
• media-adbreak-ad-durations="30.0 30.0 30.0"

After a mediaadbreakstart event, the media element updates these to represent the current ad.

• media-duration
• media-current-time

With then some added ad-specific details

• media-ad-url

Most cases I think would just choose the simple 1-ad path. It does seem to be a key decision whether we always split the ad metadata from the primary video metadata. Including things like, does media-duration get updated before or after mediaadbreakstart, seeing as it would also fire the 'durationchange` event?

[littlespex]

I think we will have to provide the ad metadata separately. In addition to the time info, metadata about whether the ad is skippable and clickable can vary from ad to ad. Furthermore, for live streams the total number of ads in a break isn't always known at the beginning of the break. This would leave us with the following list of changes needed for a production ready ad UI:

Methods

Custom Media Element

• skipAd(): Provided by the custom media element. Called by the media-controller when it receives the mediaadskiprequest event.
• clickAd(): Provided by the custom media element. Called by the media-controller when it receives the mediaadclickrequest event.

Events

Custom Media Element

• mediaadbreakstart: Fired by the custom media element at the start of an ad break. The media-controller would then add the media-ad-mode attribute. This event could also provide a payload with info about how many ads in the break.
• mediaadbreakend: Fired by the custom media element at the end of an ad break. The media-controller would then remove the media-ad-mode attribute.
• mediaadstart: Fired by the custom media element when an individual ad begins playback. The event payload would need to provide:
  • skippable: Boolean flag indicating whether or not the ad can be skipped. The media-controller would ad the media-ad-skippable attribute.
  • clickable: Boolean flag indicating whether or not the ad has a click thru url. The media-controller would ad the media-ad-clickable attribute.
• mediaadend: Fired by the custom media element when an individual ad ends playback.

  NOTE: This event may not be needed.

Media controls

• mediaadskiprequest: Fired by the skip ad button. Listened to by the media-controller, which would then call skipAd() on the custom media element.
• mediaadclickrequest: Fired by the "Learn More" button or click through element. Listened to by the media-controller, which would then call clickAd() on the custom media element.

Attributes

Custom Media Element

• media-ad-break: Added to the media-controller on media-ad-break-start, and removed on media-ad-break-end. Used to write break specific CSS rules.
• media-ad-skippable: Added to the media-controller on media-ad-start, removed on media-ad-end. Used to write ad specific CSS rules. (i.e. show the skip button)
• media-ad-clickable: Added to the media-controller on media-ad-start, removed on media-ad-end. Used to write ad specific CSS rules. (i.e. show the "Learn More" button)

Media controls

• media-ad-break-current-time: The current time in relation to the entire ad break
• media-ad-break-duration: The total duration of all ads in the break
• media-ad-break-ad-count: The total number of ads in the break
• media-ad-current-time: The current time of the currently playing ad
• media-ad-duration: The duration of the currently playing ad
• media-ad-break-cues: A list of ad cue points for drawing progress bar ad indicators

Slots

We will need at least one new slot, or perhaps a specific custom element, that acts as the container for ad rendering. Most ad SDKs require this element to be passed in to ad manager.

[heff]

I like this.

clickAd() - I like that this is putting it back on the media element to handle the click (and I assume track it). I had assumed it would be saying what to do on a click, like giving a URL to follow. This also makes we wonder if we should be setting up a rule that "the media element should not handle its own clicks". That kind of makes sense since the controller is already handling clicks on the media to toggle play/pause, and you know, we're building the user interface here. But also maybe that's not necessary.

media-ad-mode attr - how would you feel about a name more like media-ads-playing or media-ad-break?

mediaadstart - In our QoE sdks we've used adplay and adplaying, because there's still that difference, and I think that could still be relevant for the UI, like showing a loading spinner.

Here's a question...do we need a mediaadend event, or does the UI just care about the next adplay or adbreakend? I like the completeness of including mediaadend but feels like a good question to ask for the sake of keeping things simple. Maybe the question is, what media-attrs change when mediaadend is fired?

We'll need something additional to know how many ads are in the break, when we can know that. e.g. media-ad-break-ads-count, media-ads-in-break

This is feeling really good. 👍

[littlespex]

I'm open to any naming changes, and I would go with media-ad-break. For mediaadend, I can't think of a UI reason for this event that couldn't be met by adplay or adbreakend. I will update the previous comment with these changes, as well as a note about a new slot for an ad container.