search

element-hq / element-call

Group calls powered by Matrix
https://call.element.io
GNU Affero General Public License v3.0
585 stars 89 forks source link

All iPhones/iPads getting burning hot, high battery drain due to unsupported VP8 codec #2579

Open jacotec opened 2 months ago

jacotec commented 2 months ago

Steps to reproduce

Users are refusing to use Element for video calls after migrating to Element-X / Element-Call for video conferences / calls as their iPhones are getting burning hot during video calls, battery drain of 10%(+) per 15 min.

We never saw this in legacy element with 1:1 calls or Jitsi, nor with BigBlueButton or Nextcloud Talk.

Investigating further the root cause is the use of VP8 which is not supported in any way by Apple hardware at all. There are plenty of reports in the web which are documenting the same issue when using VP8, as VP8 needs to be decoded/encoded fully in software and can't use hardware acceleration.

Outcome

What did you expect?

Also Apple participants should be able to use Element Call without suffering extreme temperature dissipation and battery loss.

Widespread supported standard codecs like H.264 or H.265 should be at least offered as an alternative to all clients to support hardware acceleration

What happened instead?

Apple devices getting hot, losing battery quickly in Element calls. People refusing to use Element-X for calls due to this issue.

Operating system

IOS, iPadOS, MacOS

Browser information

Element-X, Element-Desktop

URL for webapp

Element call 0.6.0

Will you send logs?

No

jacotec commented 2 months ago

Workaround for self-hosted Element-Call deployments (Docker):

In element-call/volumes/elementcall/backend/livekit.yaml add this section:

room:
  enabled_codecs:
    - mime: video/h264
    - mime: audio/opus

This will force livekit to use H.264 and all is back to normal in the Apple world, iPhones not getting hot anymore, battery use for 15 minutes video call on an iPhone 14 is down to 3% now.

fkwp commented 2 months ago

@jacotec thx for the research :-) very good datapoints. We already have this topic on our radar. Currently we are finishing getting-the-basics-right; in that order:

We are aware of the various codes out there and its complicated dependencies. Where h264 makes a lot of sense in the Apple eco system (energy performance) its looks different on Android. Here https://livekit.io/webrtc/codecs-guide is a good summary with all the pros and cons

Sadly VP8 is currently the only codec which at least works on every platform. Hence, that is our starting point. Going forward we want to include the preferred codec already on the m.call.member state event layer. With the above this is a next/later topic.

In the meantime your proposed workaround (restricting codecs on the SFU level) seems to state a good compromise assuming that calls originated on your SFU belong mainly to your user-base. (Btw. If you join a call in a federated room say #webrtc:matrix.org which has already a call running, your client will use a different SFU likely then using VP8) -- Stay tuned.

jacotec commented 2 months ago

@fkwp Wow, I really thought good old h264 is not only partially supported in Android. But it looks like VP8 does NOT work on any platform also, an iPhone 8 Plus does only work one-sided with VP8 (It could not send video) but it's fine with h264.

I'm by far not a WebRTC expert, but I'm asking myself why Nextcloud Talk and Bigbluebutton do work with any device and without using excessive resources ...

For me I'm fine with the workaround for now as it works on all known mobile devices as well as on PC and Mac (E-Web/Desktop) without issues.

I hope you'll find the "golden solution"!

BTW: What was the native 1:1 call in the old Element using? Never had an issue with this ...

pixlwave commented 2 months ago

an iPhone 8 Plus does only work one-sided with VP8 (It could not send video)

Ohhhhh, that would explain why I failed to use my iPhone 8 as an extra call participant when working on PiP in EX.

fkwp commented 2 months ago

I just confirmed with a real hardware device (Moto e20) and It fails encoding h264 media. So to get this sorted out we need similar to the foci_preferred concept a video_codec_preferred concept...

jacotec commented 2 months ago

I just confirmed with a real hardware device (Moto e20) and It fails encoding h264 media. So to get this sorted out we need similar to the foci_preferred concept a video_codec_preferred concept...

That's really strange. I've read lot across the web in the last days, and most sites say for WebRTC h.264 is the de-facto-standard, followed by VP8.

h.264 is officially fully supported from Android 5 (i think Android 4 could decode but not encode) ... and from what Google says this e20 is running Android 8.

So, what should be the reason that h.264 should not be supported?

However, I also read that WebRTC can offer more than one codec. What I don't know: Does Livekit as SFU support running different codecs to each client? Or will still all iPhones burn once only one "bad" Android is in the call?

fkwp commented 2 months ago

Or will still all iPhones burn once only one "bad" Android is in the call?

I think so -- we really need to investigate this properly.

The e20 is on Android 11. And it seems it's only the encoder which is not working. Decoding h264 is fine. But I don't want to rush a solution now... let's do the homework properly.

jacotec commented 2 months ago

Or will still all iPhones burn once only one "bad" Android is in the call?

I think so -- we really need to investigate this properly.

The e20 is on Android 11. And it seems it's only the encoder which is not working. Decoding h264 is fine. But I don't want to rush a solution now... let's do the homework properly.

Sure!

Nevertheless, strange that this phone behaves like it's described for Android 4 (decoding: yes, encoding: no). I have an old Xioami phone which I've only used for some Android testing and which houses the WhatsApp account for my Mautrix-Bridge (i personally don't want to have WA on any operating phone). I'll see how this behaves.

jacotec commented 2 months ago

@fkwp Tested it on that old Redmi 9A. Android 10. This phone is slow like hell, feels like an old 486 running Windows. Takes forever to even open Element-X-Android ... but EC with h.264 works perfect in both directions.

Can't imagine that it does not work on Android 11...?

Quackdoc commented 2 months ago

Support is on a per phone basis, however I would say I havent seen an android device without H264 support in nearly a decade now bugs aside (and some do for sure have some bugs). some phones may opt to exclude h264 support since it is licensed, however most devices have working platform support.

H264 licensing may pose an issue in some cases, IIRC the patents for H264 Main profile have all expired so that should be safe to use regardless. H264 also has significantly higher quality when doing real time encoding as well.