Improve <video> and <audio> elements

[FluorescentHallucinogen]

From what I saw, most developers prefer third-party players (video.js, Plyr, etc.) over native <video> and <audio> elements for two reasons:

• style player
• add controls

Off the top of my head:

1. The speed of the audio/video playback can be set only via playbackRate property: document.getElementById('player').playbackRate = 0.5. Why not to expose it as UI control?

2. Whether the audio/video should start playing over again when it's finished can be set only via loop property: document.getElementById('player').loop = true or via loop attribute: <video src="test.webm" loop>. Why not to expose it as UI control?

3. There's no UI for switching sources (e.g. audio sources in different languages, video sources in different resolutions).

4. The <source> element has no srclang and label attrubutes unlike <track> element. You can't do something like:

<video width="320" height="240" controls>
  <source src="forrest_gump_480.mp4" type="video/mp4" label="480p">
  <source src="forrest_gump_1080.mp4" type="video/mp4" label="1080p">
  <source src="forrest_gump_en.ogg" type="audio/ogg" srclang="en" label="English">
  <source src="forrest_gump_no.ogg" type="audio/ogg" srclang="no" label="Norwegian">
  <track src="fgsubtitles_en.vtt" kind="subtitles" srclang="en" label="English">
  <track src="fgsubtitles_no.vtt" kind="subtitles" srclang="no" label="Norwegian">
</video>

[una]

See also VTT and caption styling (::cue)

This is definitely an interesting area to explore and would require some research on existing patterns.

[una]

Would you want to discuss this at the telcon? Is there anything to resolve on here specifically?

[Malvoz]

• style player

See related: https://github.com/WICG/controls-list/issues/5 about styling controls specifically.

[gregwhitworth]

@Malvoz this is perfect thank you and this also sounds great @FluorescentHallucinogen.

A few things here - Open UI is more than happy to investigate any of these ideas. That said, it requires someone to champion the effort. The video and audio elements you're actually in a good starting position in that much of the API's defined actually standardized much of the parts since you invoke them. @Malvoz @FluorescentHallucinogen either of you want to take on the task of beginning to build out a research and proposal page? There are enough major video players that a solid anatomy and behavior set could be defined.

This then could begin to solve aspects of your issues by denoting that there should be new attributes or psuedo elements (eg: removing the download button).

Ironically, I've referenced the media elements as somewhat low-hanging fruit as you either get ALL controls or none of them when you may just want to change out a sub-menu rather than rebuild the entire thing.

[una]

Do we have any research on this area? Do any design systems include

[Malvoz]

either of you want to take on the task of beginning to build out a research and proposal page?

@gregwhitworth I don't see myself finding the time to dive into this anytime soon, unfortunately.

https://github.com/WICG/open-ui/issues/213#issue-730662606

• add controls

One of the challenges is adding custom controls that integrate well with the native controls. For example avoiding overlapping default controls (a low-level primitive may be derived from discussions such as discourse@WICG: Position an element relatively to another element from anywhere in the DOM), or responding to a responsive UI where (some of the) controls may be hidden behind a collapsed menu at certain sizes.

Another reason why developers may choose not to use native media elements is because they're not doing too well in the accessibility department, see the assessment by @svinkle at https://scottvinkle.me/blogs/blog/how-accessible-is-the-html-video-player.

[gregwhitworth]

Another reason why developers may choose not to use native media elements is because they're not doing too well in the accessibility department, see the assessment by @svinkle at https://scottvinkle.me/blogs/blog/how-accessible-is-the-html-video-player.

Thanks for sharing this. @svinkle did you file bugs on the browser vendors for these?

One of the challenges is adding custom controls that integrate well with the native controls.

This is one of the key aspects of Open UI, is to ensure that the native controls are defined/standardized (also this will also solve many of the issues identified there since there will be a concrete UI doc to follow) to ensure that they are extensible and solve the majority of usecases.

I don't see myself finding the time to dive into this anytime soon, unfortunately.

That's understandable, we all are in similar boats and thus end up in this scenario where we have to pick and choose the items we spend our time on.

[FluorescentHallucinogen]

It seems all my pain points as a regular developer I've described in the first comment: https://github.com/WICG/open-ui/issues/213#issue-730662606.

Things like adding the srclang and label attrubutes to the <source> element require changes in the HTML spec.

Also, I have some crazy ideas like adding new <tracklist> HTML element and exposing "Cast" button for Presentation API and "VR" button (see https://crbug.com/605200) to native player. But I'm not sure is it out of scope of this project or not.

I think the good idea is to ask developers of third-party players what is missing in the native <video> and <audio> players.

@sampotts Do you have anything to add? This is related to what we discussed recently in Slack. Why did you start working on Plyr?

[Malvoz]

It'd be nice if media elements supported tiny screen/element sizes without compromising usability. For example, default controls may be hidden from the user if there isn't enough space (taking video in Chrome as an example):

[sampotts]

@sampotts Do you have anything to add? This is related to what we discussed recently in Slack. Why did you start working on Plyr?

Hey 👋

Without going into too much detail, at the time (2013 or so) there were a handful of options for custom video and audio players; mainly video.js and mediaelement. Both were using strange markup at the time - things like <span> for buttons, <div>s for progress, no <input type="range">, missing ARIA, etc. Also, the styling out of the box didn't suit my needs. I originally built Plyr for myself as I need to use something in a project and wanted a nice looking player that had a small footprint - it started out as 3kb "minzipped"! It's now up there with the big players if GitHub stars are anything to go by 🤷🏼‍♂️.

It would have been great to been able to style up the native players somehow and then extend the functionality rather than the design as well. Adding an extra layer on top of the players only makes for a fragile solution as I'm sure we all know. I can only imagine the politics that goes on between vendors but the dream would be to have a standard player implementation (with features such as speed, language, looping, etc) that can be easily customized via CSS using the shadow DOM like other elements such as <input type="range">. I know the styling is not what's being discussed here though.

I'd love to work with browser vendors on this although I'm sure they have folks looking after that side of things.

[jimmywarting]

Oh, i would love to see a standardized UI for the video element!

I quite like the native UI better then full custumized UI and not having to rebuild every single control with js, html and css just so you can add that missing quality, language & subtitle picker, and when someone mention <tracklist> i also thought about prev/next/first/last/shuffle.

I could give you a hole list of custom controls for why you would want to go all custom (as i have built a own player with cast support)

List of possible controls

- [ ] poster image (shown durning PiP and casting) - [ ] video title (only mouse movement and paused) - [ ] source picker (hls, dash, mp4...) - [ ] subtitle picker - [ ] quality picker - [ ] live indication (just a red dot indicating if it's live or not) - [ ] Current Time (needs to be adjustable if it's paused on the screen and playing on a cast device) - [ ] Remaning Time - [ ] duration (how long the video is) - [ ] jump to prev/next video - [ ] rewind and fast forward - [ ] jump 30 sec back & fort (could be customized if video is long or short) - [ ] volume up - [ ] volume down - [ ] mute - [ ] volume meter changer - [ ] cast button (aka Presentation API - airplay, chromecast) - [ ] enter fullscreen - [ ] exit fullscreen - [ ] large player - [ ] small player - [ ] play - [ ] pause - [ ] stop? - [ ] enter picture in picture - [ ] leave picture in picture - [ ] download video - [ ] playlist (tracklist) - [ ] video buffered (shows how much & what range-parts you have downloaded - definitely needs to be customizable if you control whats is downloaded, good eg for webtorrent) - [ ] a button for taking things offline (store it in cache, indexeddb, or sandboxed filesystem) - [ ] a button to download the video or any of the other qualities - [ ] a 3-doted menu just to put other stuff in - [ ] a chapter time bar (like the thing [youtube](https://www.youtube.com/watch?v=pvkTC2xIbeY) have done) - [ ] [preview thumbnails](https://support.jwplayer.com/articles/how-to-add-preview-thumbnails) for the time bar - [ ] repeat - [ ] shuffle - [ ] a stalling spinner (loading/buffer indication) - [ ] copy video adress - [ ] copy video adress at specific time - [ ] copy embed video code - [ ] watermark (logo) - [ ] going vr/360 viewing - [ ] play speed control - [ ] settings (menu) button - [ ] option to add/pick a own local subtitle file to the video element? - [ ] [frame accurate seeking](https://bitmovin.com/demos/frame-accurate-seeking) - [ ] something [vlc have but no other web player: delay](https://wiki.videolan.org/VLC_HowTo/Adjust_subtitle_delay/) adjust sound, and subtitle (would be more of a hotkey) adjusting the delay gave me thinking of other stuff too. if you are playing a WebRTC media stream then you might want other controllers - [ ] changing play [playoutDelayHint](https://stackoverflow.com/questions/56510151/change-playout-delay-in-webrtc-stream/58341171#58341171) - [ ] call button - [ ] hang up - [ ] toggle mic button - [ ] toggle webcam button - [ ] record video (mine, theirs or both?) - [ ] button to toggle chat pannel little of topic on those last ones. I would wish you could change some stuff in browser settings or at one video element (where you pause the video and toggle a kind of settings panel) and having it sync towards other video elements on other websites as well - [ ] possible a UI to customize subtitle (font family, size, color, background) - [ ] hotkey's management ... you see, many developer don't use native UI b/c it's missing some stuff so we need to re implement everything and that takes time. we need a way to add controllers to the video element. hard to add in everything, but that's why it needs to be customizable and why Open UI exist.

[Malvoz]

Re https://github.com/WICG/open-ui/issues/213#issuecomment-761208580:

• [ ] copy video adress at specific time

A similar suggestion was made in https://github.com/whatwg/html/issues/6323. Edit: well, the capability to share the copied video address at a specific time was suggested.

[FluorescentHallucinogen]

Recently @beaufortfrancois has finally implemented native controls for playback speed of <video> and <audio> elements in Chromium! 🥳

See https://twitter.com/quicksave2k/status/1415638025183371265.

@beaufortfrancois What about implementing native controls for other things like loop and etc. from https://github.com/openui/open-ui/issues/213#issue-730662606?

[beaufortfrancois]

Which native controls are you thinking about? Loop for instance is already part of the context menu.

[FluorescentHallucinogen]

@beaufortfrancois

Loop for instance is already part of the context menu.

Oh, I've checked on Android and haven't found "Loop" item. Also I haven't found "Picture and picture" as well as your new "Show all controls" that available on desktop.

Any plans to bring it on Android?

The second problem is that "Loop" feature is hidden behind context menu. Users need to right-click or long-tap on the video to find it.

What about to expose it as a button (maybe as a part of 3-dot menu since this is not such an often needed feature)?

The third problem is that on Android the <audio> element has no context menu (long tap on the audio player just doesn't work).

[FluorescentHallucinogen]

@beaufortfrancois

Which native controls are you thinking about?

UI for switching sources (e.g. audio sources in different languages, video sources in different resolutions/formats).

[FluorescentHallucinogen]

E.g. Safari <video> player has UI for switching audio tracks e.g. when playing HLS stream with multiple audio.

<video
  controls
  src="https://bitdash-a.akamaihd.net/content/sintel/hls/playlist.m3u8"
  type="application/x-mpegURL">
</video>

Surprisingly, Chrome for Android can play HLS streams natively too!

But Chrome <video> player has no UI for switching audio tracks. :(

And for some reason Chrome on desktop doesn't support HLS. :(

BTW, another pain point for developers is the playback of HLS (HTTP Live Streaming) and MPD (MPEG-DASH) streams.

It would be really great if browsers supported playback of HLS and MPD natively on all platforms without the need for libraries like hls.js and dash.js. Something like this:

<video
  controls
  src="https://bitmovin-a.akamaihd.net/content/sintel/sintel.mpd">
</video>

@beaufortfrancois cc.

[Malvoz]

Adding to the list of wants, transcripts for media elements: https://github.com/whatwg/html/issues/7499.

[github-actions[bot]]

There hasn't been any discussion on this issue for a while, so we're marking it as stale. If you choose to kick off the discussion again, we'll remove the 'stale' label.