Element X iOS Specific Picture in Picture

[stefanceriu]

Your use case

What would you like to do?

• take advantage of the system picture in picture behaviors while using ElementCall (especially when embedded in EX)

Why would you like to do it?

• because it provides a better user experience than whatever custom solution we might come up with

How would you like to achieve it?

• allow picture in picture to be enabled from the web side on currently active speaker tile(?)
• iOS will intercept that and take the right steps to hide the main EC window so that the user can interact with the rest of the application
• web will change that particular video tag's source to be whoever is currently speaking and the iOS PiP will automatically update (as per our exploratory testing)

Have you considered any alternatives?

No response

Additional context

No response

[toger5]

how do we want to communicate this between app and webview? I can imagine that EX will know when it gets pushed into the background and then informs the app that it should start PIP now?

[stefanceriu]

EX will not get pushed into the background per se but the PiP will be presented on top of it. When that happens the system posts a AVPictureInPictureControllerDidStart notification which we can intercept and "minimise" the parent EC WebView

[toger5]

What is the initial event that we need to use to start the PIP in EC in the first place?

[stefanceriu]

I would assume that would sit completely on the Element Call side as a button on the video or similar.

[toger5]

I would expect, that i can just use native multitasking from ios and still see the PIP. (this is also what we need to reach feature parity with EXA)

[stefanceriu]

I'm no longer sure we're talking about the same thing, what do you mean by native multitasking? How would you imagine this working?

There are different use cases for which we clearly need product input:

• does the user want to interact with the rest of the app and minimize element call? Element call currently controls all of the chrome.
• does the user want to minimize element x as a whole? would they expect element call to automagically enter pip mode?

YouTube for example automatically enters and exists PiP when backgrounding and foregrounding the app but using a completely custom interface for when minimizing within the app. We're trying to avoid coming up with a custom solution.

[toger5]

I think that switching apps (minimizing apps) and automatically entering pip mode for the call is the primary use case. I dont see why we would want to show a pip view above the ongoing call. So making the user manually start a pip view before they switch to another app/screen on their device seems unnecessary.

This is also what we have on android.

So I would expect:

• EX detecting that it will be moved to the BG.

  • EX sending an event to ElementCall
  • ElementCall starts a native PIP view
  • EX gets put into BG but the native PIP view is still visible to the user

  I dont see a case, where one would benefit from making:

  • start pip view
  • moving away to another part of the OS (another app for instance) separate actions.

[pixlwave]

Hey @toger5, I'm jumping on this issue as I've been assigned to it now.

I think that switching apps (minimizing apps) and automatically entering pip mode for the call is the primary use case.

We're specifically looking for PiP to implement https://github.com/element-hq/element-internal/issues/601: "This project enables the navigation within the app when the user has an active call in Element Call."

For this we would need a way for the user to put EC into a PiP, that signalled to us so that we can minimise the webview and let them continue to navigate around the app.

• EX detecting that it will be moved to the BG.
• EX sending an event to ElementCall
• ElementCall starts a native PIP view
• EX gets put into BG but the native PIP view is still visible to the user

This isn't specifically required by the project but would 100% be nice to build at the same time as it is only 1-extra signal to get it once we've build the above.

[pixlwave]

cc @robintown 👆

[pixlwave]

Small update: We already get the signalling as Stefan mentioned above, so we would just need the button.

[robintown]

Cool, so the button itself should be trivial given a design. What's maybe more difficult is creating a single \<video> element which has its source swapped out to always reflect the active speaker. The solution we're using currently seems to recreate the \<video> element whenever the speaker changes. (needs more fine-grained reactivity) I'm going to take a quick look at this now.

[robintown]

Wasn't so difficult after all: https://github.com/element-hq/element-call/pull/2553

[stefanceriu]

Wasn't so difficult after all: #2553

Whoa that's super cool! 👏

[robintown]

@pixlwave Did I get it right that you're aware of my new PR https://github.com/element-hq/element-call/pull/2558 which uses the webkitPresentationMode API, and can test whether that one requires transient activation? That's the only blocking tech question left, I think, because it determines whether EX can use native UI for the back button or whether EC should be responsible for this.

[pixlwave]

Just testing it now 😁 (got waylaid yesterday with other things). It seems promising so far 🤞

[robintown]

@pixlwave Got a new PR for you to try out: https://github.com/element-hq/element-call/pull/2563

This one exposes a global JS API to control the picture-in-picture view, available as controls.enableCompatPip() and controls.disableCompatPip() on the window. You should always see video in the PiP as long as someone (not necessarily the speaker) has their camera on, and when this is not the case it should fall back to a screen share or else the following placeholder video stream:

speaker-without-video.webm

It uses the webkitPresentationMode option of course. I don't have an iOS device to test with, but it seemed to work in Chromium when I swapped the calls out for requestPictureInPicture.

[pixlwave]

I've just given this a go and can confirm that controls.enableCompatPip() works great. It took me a while to realise I needed to call disableCompatPip() after the user has put the PiP back, but once I did I was able to relaunch it. Would it be possible to make disableCompatPip happen automatically when the presentation state changes back to inline?

What doesn't work is the static video. I just end up with a blank call in this instance (the PiP doesn't start, so the screen isn't hidden):

I don't see any logs in the console when this happens.

[robintown]

Talked with Doug and it seems the remaining gaps are:

• Make sure a loaded video is always in the PiP (we think this may solve the switching issue)
• Crop the placeholder video to 16:9
• Find some way to hide the controls possibly
• If there's no way to hide the controls, then detect pause events and just start the video again
• Return something from the enableCompatPip method to indicate whether the PiP has actually been activated

[pixlwave]

Can confirm that the static video now works:

I sent debug logs about 5 mins ago for the error when returning out of PiP.

Find some way to hide the controls possibly

Related, it seems worth pointing out that the static video appears to have played and finished so has a play button and the jump forwards button is now enabled.

[robintown]

@pixlwave New version is up on the same branch with all the currently known issues addressed, hopefully. I dug into the controls stuff and found that, as I suspected, I've already done everything I can to disable controls on the HTML element. So I've made it so that the video will immediately resume if you try to pause it - and the placeholder video will immediately pause if you try to play it.

Ideally I'd like to keep the placeholder video in the paused state rather than looping it, because looping a 1-frame video caused a lot of CPU usage in my testing due to all the events it emits. I could make it artificially longer if we care about showing a stop button there for consistency, but this still wouldn't disable the seek buttons.

I've removed the controls.disableCompatPip method because it should now automatically detect when the element goes back to the inline presentation mode, and also controls.enableCompatPip will now throw an error if the call is not yet running, which you can catch.

[robintown]

My confidence on having fixed the bug where the PiP just disappears when it would switch video is about… 50%. Be prepared for a couple more iterations if it hasn't worked out :)

[pixlwave]

I've made it so that the video will immediately resume if you try to pause it

This works perfectly! 👏

I've removed the controls.disableCompatPip method because it should now automatically detect when the element goes back to the inline presentation mode

This doesn't appear to happen from my testing. When I close the PiP the web view remains showing a single video source with no controls to hang up. Calling enableCompatPip then doesn't work any more 🙃

controls.enableCompatPip will now throw an error if the call is not yet running, which you can catch.

Could we do the opposite and have it return a specific signal if the call is running? For example on both of the screens below tapping the back button doesn't return anything or throw, so I imagine the method maybe isn't available on the window at this point? I'm imagining there are others too.	Error setting up call	Error during call

[pixlwave]

Just tried out switching tiles and it doesn't end the call, but what seems to happen is it remembers which speaker was the one with the PiP presentation mode. When it switches to someone else the PiP stops and when it switches back to the original speaker the PiP resumes again.

The first time the PiP starts was me, the rest is happening automatically with the speaker/tile changes (sorry for the lack of audio which would help here, the iPhone screen capture doesn't seem to get it from the web view):

https://github.com/user-attachments/assets/69379d15-fde0-4014-a8ce-c40115648016

Edit: For what it's worth in Stefan's demo html we only got the PiP to remain open by switching the src on the video tag (but this was with video files).

[robintown]

Gah okay, I need to have a think then about what semantics this behavior from the native PiP element would indicate, and how to actually work with them.

I'm disappointed but not surprised that the automatic inline detection isn't working, since I found the event name for that in the Webkit source but not in any current documentation :sweat_smile:

And on the enableCompatPip return value issue, I think I've just discovered that the call view model is not being fully disposed after the call, so from the perspective of enableCompatPip the call is still running. I need to fix that regardless of whether we change it to a simple return value.

[robintown]

I really can't make sense of the speaker switching behavior yet, so I've added some extra logging. Could you please run through the same scenario again, ideally with another video, and submit logs through the feedback page of settings? Thanks!

[pixlwave]

Sent as Doug PiP Logs

Corresponding video file below. The video that shows in the PiP is my normal account and the video that closes the PiP when "talking" (I piped an MP3 into the relevant call to switch) was m1-element.

https://github.com/user-attachments/assets/a2d3e8fe-e1b4-4624-9156-91035e4f6dd5

[robintown]

@pixlwave New build implementing only the changes needed for the web view PiP approach that you discovered: https://pr2573--element-call.netlify.app/ The controls are renamed to controls.enablePip and controls.disablePip.

[pixlwave]

This looks really slick now (at least when using placeholder tiles)! 👏

https://github.com/user-attachments/assets/f6703627-447f-4220-b902-5a1dda9d6c54

[manuroe]

We discussed it with @pixlwave . We have everything now. Thanks a lot! PiP is so cool 👯👯👯