search

nextcloud / talk-ios

📱😀 Video & audio calls through Nextcloud on iOS
GNU General Public License v3.0
147 stars 87 forks source link

Add more video & audio codecs to Nextcloud Talk iOS #1198

Open gymnae opened 1 year ago

gymnae commented 1 year ago

Is your feature request related to a problem? Please describe. Janus, part of the standalone signaling server, can be configured to use modern and efficient codecs like VP9,AV1,H265 etc. And the Nextcloud Talk infrastructure can work perfectly fine with this.

Using Nextcloud Talk in a Browser, even mobile Safari, one can make use of these codecs.

The issue appears form Desktop to iOS - iOS to iOS works, but i’m not sure which codecs are being used

Describe the solution you'd like Add codec support which is on par with mobile Safari, mainly this would be h265, h264 and VP9 Right now, receiving video is blank (grey backdrop without user icon), whereas video sent from iOS works.

Describe alternatives you've considered Not using the iOS app any more, instead only mobile safari

gymnae commented 1 year ago

I think all there needs to be done is to change this https://github.com/nextcloud/talk-ios/blob/4849cd89348dada5d176ab7d13de90c402f16a86/NextcloudTalk/NCPeerConnection.m#L150 to have it not say h264, but leave it open to the server to negotiate. Then all codecs could be used, even vp9

SystemKeeper commented 1 year ago

Hey, thanks for your issue and your pull request on the signaling repo. Unfortunately it is not that easy. The line you mentioned above does only shuffle the codecs around so the H264 is in front of every other supported codec:

image

It does not actually change anything regarding the supported codecs. I only took a quick look at your request and found an older issue on the webrtc lib regarding this:

We will try to keep the iOS WebRTC library more simple from now on, and we have no plan on adding H265 support anytime soon. The injectable HW codecs used by AppRTCMobile should be seen more as an example of how an app can provide custom video codecs. I would recommend implementing your own injectable video codec and add H265 support there if you want to get it.

(See https://bugs.chromium.org/p/webrtc/issues/detail?id=8265)

So I don't think it's as easy as it seems. I have not looked into how/if other codecs could be supported. Since only H264 and VP8 are mandatory according to the standard (https://developer.mozilla.org/en-US/docs/Web/Media/Formats/WebRTC_codecs), there's probably a low chance that Google will implement anything else soon.

gymnae commented 1 year ago

Hi, thank you for the time to reply to my issue. I’m no coder, so I’m just fishing in the dark, thus appreciate that you explained the situation to me. If Google is behind the library, maybe VP9 is possible, since it’s „theirs“ and mobile Safari does support it, at least in my trials.

Edit: I might be running into this bug: https://bugs.chromium.org/p/webrtc/issues/detail?id=13573#c52

SystemKeeper commented 1 year ago

mobile Safari does support it, at least in my trials.

Did you enable it? Because it’s considered experimental and only opt-in in the safari settings right now?!

gymnae commented 1 year ago

mobile Safari does support it, at least in my trials.

Did you enable it? Because it’s considered experimental and only opt-in in the safari settings right now?!

I checked, it was checked ;) But I don't remeber setting it on purpose. Disabling it yields then the same result als Talk iOS. Reading more about VP8 vs VP9 - there might not be such a huge benefit to it anyway.

EDIT: I just tested with AV1 in front of the list and h264 removed, it works fine with Talk iOS on my iPhone XS. Not sure if it will hit the battery harder. It works also from browser to iPhone