Videos on iOS devices are missing important buttons

[owi92]

iPhones are missing the fullscreen and volume buttons (iOS 15.6, tested chrome and safari):

iPads are missing the volume button (iOS 16.0, tested chrome and safari):

[karendolan]

1. Our site requests that the Paella volume button not show up when the player cannot control the device volume, as on some mobile devices.
2. Our site also requests that if @owi92 gets the wish regarding fullscreen:
   • that it be configurable because it is problematic for us to allow devices to take the player out of the Paella wrapper context for the reasons mentioned on the last comment of this pull: https://github.com/polimediaupv/paella-core/pull/320
   • that it be configurable to disable full screen for all iOS devices. We locally added this isEnabled() override to the fullScreenButton plugin to ensure it would not be visible under 3 conditions: browser does not allow, embed does not allow, disabled for all iOS devices.

     // #DCE Override https://github.com/polimediaupv/paella-basic-plugins/blob/main/src/plugins/es.upv.paella.fullscreenButton.js
     async isEnabled() {
     const enabled = await super.isEnabled();
     // #DCE OPC-892 related to disabling full screen when not supported,
     // also check if full screen button is enabled when embedded in an iFrame
     // #DCE OPC-910 Also disable fullscreen for all IOS
     const isIOS = DceUtils.testIfIOS();
     const isFullScreenEnabled = (
     window.document.fullscreenEnabled ||
     window.document.webkitFullscreenEnabled
     );
     return (
     enabled
     && isFullScreenEnabled
     && this.player.isFullScreenSupported()
     && !isIOS
     );
     }

[LukasKalbertodt]

volume button not show up when the player cannot control the device volume, as on some mobile devices.

Oh interesting, I never considered that. While I personally dislike this (I wish I could separately control the volume of all my mobile apps), I certainly see the reasoning behind it (to reduce user confusion with two separate volume controls).

browser does not allow, embed does not allow, disabled for all iOS devices.

The first two I fully understand but why the third condition? Why disallow fullscreen on all iOS devices?

EDIT: oops, I'm sorry, I only now saw the linked PR. I'm not fully sure yet what you mean by "iOS fullscreen takes the video out of the context of the player". But even with the listed disadvantages (which might be solvable in another way?), I would argue that watching a video in full screen is still a very important requirement and just saying "nop, iPhone users cannot" seems like a non-solution to me?

[oas777]

No problem with the suppressed audio control. With respect to fullscreen, my understanding is that DCE wants more control over the usage and its context. That's fine as long as it doesn't prevent others from having more viewing area, a. k. a. fullscreen.

@karendolan, can we compromise on customizing this?

[karendolan]

@karendolan, can we compromise on customizing this?

@oas777 @LukasKalbertodt Yes, I would love a config option to allow a site like ours to choose to avoid letting the native player take the video out of the Paella context.

        "es.upv.paella.fullscreenButton": {
            "enabled": true,
            "side": "right",
            "order": 104,
            "tabIndex": 11,
            "description": "Fullscreen",
       +     "enabledInIOS": true
        },

As I said, right now in iOS devices (aka iPad) , we only allow the Paella player, itself, to go fullscreen by using the fullscreen button on the browser. This ensures the same experience as in other devices which can use both the Paella player's full screen button, or the browser's fullscreen button, to make the Paella player go fullscreen, in the same way.

[karendolan]

I'm not fully sure yet what you mean by "iOS fullscreen takes the video out of the context of the player". But even with the listed disadvantages (which might be solvable in another way?),

In the iPad, going fullscreen means the iPad takes the HLS video HTML element and plays it directly in the native player in full screen mode, with no Paella player wrapper. A big issue is that Closed Captions are no longer accessible (unless they are also included as a reference in the HLS video manifest) because they are a mediapackage attachment that the Paella Player plays along with the HLS video.

[karendolan]

Our site has an open ticket to add transcripts to the HLS manifest, to solve the native fullscreen mode. But we are concerned about how to toggle between the native fullscreen HLS transcripts and the Paella player's Caption plugin, or give up the Paella player's Caption plugin. The native HLS fullscreen mode means we also cannot collect user stats, easily.

[karendolan]

@oas777 @LukasKalbertodt sorry for too much traffic on this ticket. I wanted you to know that we were not able to duplicate iPad fullscreen issue using stage.opencast or the demos at paellaplayer.upv.es just now. We'll check if it's just an issue for our videos and post an update here later. Meanwhile, it'd be great to have the config option.

[ferserc1]

Sorry, I've been unattended on github issues for a while because I've been quite busy with some major improvements and planning for paella-core v2.

As a summary of the behaviour on Apple devices I have to say that currently (iPad OS 17.2, iOS 17.2) there is nothing that can be done to fix the following issues:

• Fullscreen on iOS devices: iOS does not allow fullscreen mode on DOM tree elements, it only allows it on a specific video. I understand that it may be desirable to set fullscreen mode to be enabled on a video when not using two streams, although many player features (subtitles, navigation bar plugins, multi stream, etc) would be lost. However, the fullscreen plugin is not implemented in this repository, so you should report this issue in the corresponding repository, which is https://github.com/polimediaupv/paella-basic-plugins.
• The audio JavaScript API in both iOS and iPadOS is read only. In addition to this, the device itself prevents multiple media sources from playing simultaneously. For example, on an iPad, if you play a video in the Netflix app, turn on picture in picture mode and go to youtube to play another video, the first video will stop. So it doesn't make sense for each app to control its volume, at least according to Apple's criteria.

Please try to heed the recommendations for contributions, as indicated in the template for reporting issues. None of the issues discussed here belong to paella-core. I can understand that sometimes it is difficult to identify if an issue is in one repository or another, but in this case all the issues refer to button plugins, and the only button plugin defined in paella-core is the play button.

[LukasKalbertodt]

I finally got around to looking into the fullscreen issue again.

(I'm commenting on this issue again, instead of creating a new one at paella-basic-plugins as suggested above, because the fullscreen button plugin does not seem to contain any of the logic we are talking about here. That seems to live here in paella-core as these three methods).

A few random notes:

• On iPadOS, starting with Safari 16.4, the fullscreen API on DOM nodes (that feature allows us to show both videos and Paella's controls in fullscreen) works now without vendor prefixes. So there everything should just work. Link
• On iOS, the fullscreen API indeed still only works for individual video elements using a built-in player (i.e. no captions, ...).
• On iOS, with Safari 17.2 beta, experimental support for fullscreen API on DOM nodes was apparently added, see here. But yes, that's not usable yet.

I got an idea though and have a suggestion: always show the fullscreen button, but on devices where proper fullscreen is not supported, use "fake fullscreen" instead. By which I just mean, add the following CSS to the player outer div:

position: "fixed",
top: 0,
right: 0,
bottom: 0,
left: 0,
z-index: 123456,

It's obviously not as good as normal fullscreen. But I tested it on my iPhone 13 Mini and it works quite well. It's certainly a big usability improvement for when Paella is embedded into another page, like Tobira or LMS. In Opencast itself, Paella already takes all the viewport, so there it doesn't make any difference obviously. But again, for Tobira and LMSs, it's a big win in my book.

What do you think? If you want I can prepare a PR for that.

[oas777]

Hesitant because of "hack", but always in favour of a "big win" - but how much does it cost in terms of your time or Oles?

[LukasKalbertodt]

Very little time, I suspect. Should be easy to implement, probably. And I wouldn't really call it "hack". What I suggested simply makes Paella span the whole browser viewport. Unlike a normal fullscreen, it does not hide UI by the browser or the operating system though.

I'm of course also interested in what the Paella devs think about this.

[LukasKalbertodt]

Since this was a small change I just created a pull request and a test deployment so that you all can easily test what I'm talking about.