search

jitsi / jitsi-meet

Jitsi Meet - Secure, Simple and Scalable Video Conferences that you use as a standalone app or embed in your web application.
https://jitsi.org/meet
Apache License 2.0
23.3k stars 6.76k forks source link

JitsiMeetSDK on iOS 18 Beta - Missing VPN addresses in ICE candidates #14948

Closed corby-github closed 3 months ago

corby-github commented 3 months ago

What happened?

We are trying to get our app ready for iOS 18 and noticed some issues with our meetings using the JitsiMeetSDK via CocoaPods.

The JitsiMeetSDK pod version 8.6.3 and 9.2.2 are not able to establish audio & video WebRTC channels on iOS 18 (Developer beta and public beta versions 1-4). Same experience on iPhone and iPads running iOS 18

Our app uses a VPN to connect users and relies the private addresses blocks to be exchanged with the Video Bridge. If the private addresses are sent to the VB, the channel is open and audio & video works.

Keeping everything the same in the app/code/OS, we found that downgrading to 7.0.1 fixed the issue.

Note that 8.6.3 is in current production version and it works fine on iPads and iPhones running the latest iOS release, 17.5.1

Platform

Browser / app / sdk version

JitsiMeetSDK 9.2.2 & 8.6.3 (see description)

Relevant log output

No response

Reproducibility

More details?

This problem does not manifest itself on meet.jit.si since only public addresses are used there.

corby-github commented 3 months ago

Server side tested with Jitsi 2.0.9584, Jitsi 2.0.9457, Jitsi 2.0.9364. (all had the same issue as described)

corby-github commented 3 months ago

This issue is likely deeper than JitsiMeetSDK itself... JitsiWebRTC version 106.0.0 was the last version that worked properly. The next iteration JitsiWebRTC version 111.0.2 and on do not work properly.

By the way, the private addresses expected to be sent to the JVB are not RFC1918 IPv4 private address.

I also tested this on the release from today, 2.0.9646.

saghul commented 3 months ago

Is the ICE candidate offered by the JVB only reachable via the VPN candidate?

corby-github commented 3 months ago

Yes. Thats correct.

saghul commented 3 months ago

Quick check in case it's the OS filtering something: does Safari gather candidates for the VPN in iOS 18?

corby-github commented 3 months ago

On Safari, those addresses are excluded (I can only see that in p2p mode. Nothing shows up in jvb mode). But that is actually not new in Safari. I checked a few other devices running iOS 16.3 and iOS 17.5.1 against our current production servers and those also fail (while the app works)

I also know on macOS it's been that way for many years. It's for this reason we use Electron on macOS, because those candidates are not available.

However, the app behaves differently. It seems like the app either send all the candidates (including the ones on the VPN) and it works, or none -- not even the not even the main adapter IP addresses from the Wi-Fi or LTE.

If they were fully filtered by iOS 18, wouldn't the lower Jitsi SDK 7.0.1 also not see them? 🤷‍♂️ (That is, unless the older SDK is using older iOS functions that are deprecated but not removed.)

Also, see my comments on the community - I have seen it work. First join worked over the VPN, end conference and rejoin... did not work.

saghul commented 3 months ago

Let's continue the discussion here since I suspect we won't be able to get away without code changes, though it's not clear to me where they need to happen.

However, the app behaves differently. It seems like the app either send all the candidates (including the ones on the VPN) and it works, or none -- not even the not even the main adapter IP addresses from the Wi-Fi or LTE.

This is really weird and sounds like an iOS+WebRTC bug somehow.

If they were fully filtered by iOS 18, wouldn't the lower Jitsi SDK 7.0.1 also not see them? 🤷‍♂️ (That is, unless the older SDK is using older iOS functions that are deprecated but not removed.)

Yeah I suspect older WebRTC perhaps used a different API...

I wanted to tell you to enable WebRTC logging, but I realized it's only possible if you compile the app / sdk yourself :-/

corby-github commented 3 months ago

Updated to iOS 18 Beta 5 just now. First run of our app (TestFlight running SDK 9.2.2) against production servers worked! Second time, nothing. Tried latest Jitsi Meet from TF (from today), no dice. Rebooted the phone again to rule out some weird first run scenario, still not working. This feels like a timing issue somewhere.

I will try to compile the SDK next, but that may take some time. Hopefully there is some documentation that can help

saghul commented 3 months ago

This is how to build it yourself: https://jitsi.github.io/handbook/docs/dev-guide/dev-guide-ios-sdk#building-it-yourself

corby-github commented 3 months ago

Impressive. 👏 Great documentation and scripts up-to-date. Everything built perfectly the first time. Figured you want to know that. Stand by for more details.

corby-github commented 3 months ago

Hmmm. I am getting a crash when joining the meeting (as the only participant).

I built the framework from jitsi-meet_9646

Error: com.facebook.react.JavaScript (23): "-[RTCPeerConnectionFactory rtpReceiverCapabilitiesForKind:]: unrecognized selector sent to instance 0x300a9cd80"

Backtrace:

(lldb) bt
* thread #23, name = 'com.facebook.react.JavaScript', queue = 'WebRTCModule.queue', stop reason = signal SIGABRT
  * frame #0: 0x00000001f13f6254 libsystem_kernel.dylib`__pthread_kill + 8
    frame #1: 0x00000002286adef8 libsystem_pthread.dylib`pthread_kill + 268
    frame #2: 0x00000001a909fad8 libsystem_c.dylib`abort + 128
    frame #3: 0x00000002285d15b8 libc++abi.dylib`abort_message + 132
    frame #4: 0x00000002285bfbac libc++abi.dylib`demangling_terminate_handler() + 348
    frame #5: 0x000000019e6b6e04 libobjc.A.dylib`_objc_terminate() + 156
    frame #6: 0x00000002285d087c libc++abi.dylib`std::__terminate(void (*)()) + 16
    frame #7: 0x00000002285d0820 libc++abi.dylib`std::terminate() + 108
    frame #8: 0x000000019e6c3938 libobjc.A.dylib`objc_terminate + 16
    frame #9: 0x000000010549e730 libdispatch.dylib`_dispatch_client_callout + 40
    frame #10: 0x00000001054aff04 libdispatch.dylib`_dispatch_lane_barrier_sync_invoke_and_complete + 176
    frame #11: 0x000000010a2034f4 JitsiMeetSDK`-[WebRTCModule(Transceivers) receiverGetCapabilities:] + 176
    frame #12: 0x00000001a13310f4 CoreFoundation`__invoking___ + 148
    frame #13: 0x00000001a1330144 CoreFoundation`-[NSInvocation invoke] + 428
    frame #14: 0x00000001a13a6578 CoreFoundation`-[NSInvocation invokeWithTarget:] + 64
    frame #15: 0x0000000109fc3094 JitsiMeetSDK`-[RCTModuleMethod invokeWithBridge:module:arguments:] + 388
    frame #16: 0x0000000109fc51f0 JitsiMeetSDK`facebook::react::invokeInner(RCTBridge*, RCTModuleData*, unsigned int, folly::dynamic const&, int, (anonymous namespace)::SchedulingContext) + 456
    frame #17: 0x0000000109fc4ffc JitsiMeetSDK`facebook::react::RCTNativeModule::callSerializableNativeHook(unsigned int, folly::dynamic&&) + 76
    frame #18: 0x000000010a183190 JitsiMeetSDK`facebook::react::JSIExecutor::nativeCallSyncHook(facebook::jsi::Value const*, unsigned long) + 440
    frame #19: 0x000000010a179dc8 JitsiMeetSDK`std::__1::function<facebook::jsi::Value (facebook::jsi::Runtime&, facebook::jsi::Value const&, facebook::jsi::Value const*, unsigned long)>::operator()(facebook::jsi::Runtime&, facebook::jsi::Value const&, facebook::jsi::Value const*, unsigned long) const + 44
    frame #20: 0x000000010a179a60 JitsiMeetSDK`facebook::jsc::JSCRuntime::createFunctionFromHostFunction(facebook::jsi::PropNameID const&, unsigned int, std::__1::function<facebook::jsi::Value (facebook::jsi::Runtime&, facebook::jsi::Value const&, facebook::jsi::Value const*, unsigned long)>)::HostFunctionMetadata::call(OpaqueJSContext const*, OpaqueJSValue*, OpaqueJSValue*, unsigned long, OpaqueJSValue const* const*, OpaqueJSValue const**) + 392
    frame #21: 0x00000001b7f1c780 JavaScriptCore`JSC::callJSNonFinalObjectCallbackObject(JSC::JSGlobalObject*, JSC::CallFrame*) + 316
    frame #22: 0x00000001b87b8444 JavaScriptCore`JSC::handleHostCall(JSC::VM&, JSC::JSCell*, JSC::CallFrame*, JSC::JSValue, JSC::CallLinkInfo*) + 424
    frame #23: 0x00000001b881db44 JavaScriptCore`llint_default_call + 1380
    frame #24: 0x00000001b902a500 JavaScriptCore`llint_default_call_trampoline + 24
    frame #25: 0x00000001b9026d88 JavaScriptCore`js_trampoline_op_call + 8
damencho commented 3 months ago

Don't use the web tags, use the mobile tags. Take them from here: https://github.com/jitsi/jitsi-meet-release-notes/blob/master/CHANGELOG-MOBILE-SDKS.md

corby-github commented 3 months ago

Thanks @damencho. Rebuilding from mobile-sdk-10.0.0 now. I will add the framework to my app and the JitsiSDKTest app and test those to see what happens.

corby-github commented 3 months ago

Crashing on iOS 18 beta and 17.5.1 in the same place, regardless of what server I use (including meet.jit.si) Not sure what's going on ... but clearly, it's something with my build / project (since the pod doesn't do this).

Let me work out the crash issue and then I'll update this ticket with useful information

saghul commented 3 months ago

Which WebRTC version did you pick?

corby-github commented 3 months ago 
target 'JitsiSDKTest' do
  pod 'JitsiWebRTC', '118.0.0'
  pod 'Giphy'
  pod 'libwebp'
end
corby-github commented 3 months ago 
pod 'JitsiWebRTC', '124.0.0'

got me past the crash... And actually worked in the conference in our app. (on iPad 17.5.1)

Adding this from the JVB to prove (to myself) the addresses came in:

JVB 2024-08-07 16:57:36.550 INFO: [1115] [confId=15fc135558979a4a conf_name=xxx@conference.xxxx.xxxx.com 
meeting_id=8aceec45 epId=4147bf28 stats_id=Florian-bk8 
local_ufrag=1dqo91i4msch4i ufrag=1dqo91i4msch4i] Agent.triggerCheck#1727: 
Add peer CandidatePair with new reflexive address to checkList: CandidatePair (State=Frozen 
  Priority=7962116751007684095):CandidatePair 
  (State=Frozen Priority=7962116751007684095):
    LocalCandidate=candidate:7 1 udp 2113932031 129.13x.xx.xx 10000 typ host
    RemoteCandidate=candidate:10000 1 udp 1853824767 129.13x.xx.xx 54280 typ prflx

Now, let me switch to iPhone iOS 18...

corby-github commented 3 months ago

No dice on iPhone iOS 18 Beta 5 - same results - ~How do enable WebRTC logging?~ Never mind, I see it.

saghul commented 3 months ago

https://github.com/jitsi/jitsi-meet/blob/673a54adb12edc9a34b8883881851554ee0dda11/ios/sdk/src/JitsiMeet.m#L57 enable this condition.

corby-github commented 3 months ago

Changed that to #if 1, cleaned and rebuilt. I do not see anything new in Xcode or Console. Am I missing something?

I will try options.loggingSeverity = RTCLoggingSeverityVerbose; and see that gives me more.

corby-github commented 3 months ago

Messages like these are all I see, but I am pretty sure these are not new:

...
[modules/xmpp/JingleSessionPC.js] (TIME) ICE checking JVB:   3349430050.616208
[modules/xmpp/JingleSessionPC.js] JingleSessionPC[session=JVB,initiator=false,sid=6k7hkag8ii0hf] 
  setVideoCodecs: codecList=vp8,vp9,h264, screenshareCodec=undefined
rn-webrtc:pc:DEBUG 1 setRemoteDescription +410ms
rn-webrtc:pc:DEBUG 1 setRemoteDescription OK +7ms
rn-webrtc:pc:DEBUG 1 createAnswer +1ms
rn-webrtc:pc:DEBUG 1 setLocalDescription +11ms
rn-webrtc:pc:DEBUG 1 setLocalDescription OK +4ms
[modules/xmpp/JingleSessionPC.js] (TIME) ICE connected JVB:  3349431205.730541
[modules/RTC/BridgeChannel.js] datachannel channel opened
[JitsiConferenceEventManager.js] (TIME) data.channel.opened:     3349431429.349083
...
saghul commented 3 months ago

Nope that's Jitsi Meet stuff indeed, not WebRTC! So weird, I'll take a look.

corby-github commented 3 months ago

I wrote an example app using GoogleWebRTC to see what iOS 18 reported for localIceCandidates - Thankfully, the VPN addresses are not being filtered out by iOS itself - Which is great news... because we have little control over that.

The 129.x addresses on network-id 8 and network-id 9 are the ones I am looking for:

printLocalIPAddresses
------------------------
Local IP Address: 511798852 1 udp 2122260223 192.168.1.46 64525 typ host generation 0 ufrag t7vg network-id 1 network-cost 10
Local IP Address: 933931915 1 udp 2122197247 2600:1006:b118:0000:0000:000:cca6:2d77 61207 typ host generation 0 ufrag t7vg network-id 5 network-cost 900
Local IP Address: 882204813 1 udp 2122131711 2600:1006:1150:0000:0000:0000:daf:a02b 64284 typ host generation 0 ufrag t7vg network-id 6 network-cost 900
Local IP Address: 2668116925 1 udp 2122063615 100.1xx.xx.xx 52030 typ host generation 0 ufrag t7vg network-id 4 network-cost 900
Local IP Address: 3274363746 1 udp 2122003199 fdf0:0000:0000::1 49987 typ host generation 0 ufrag t7vg network-id 7 network-cost 50
Local IP Address: 2229168904 1 udp 2121932543 129.1xx.xx.xx 52547 typ host generation 0 ufrag t7vg network-id 8 network-cost 50
Local IP Address: 3967672879 1 udp 2121867007 129.2xx.xx.xx 49465 typ host generation 0 ufrag t7vg network-id 9 network-cost 50
corby-github commented 3 months ago

Went back to JitsiWebRTC, and rebuilt, seems logging is enabled now (🤷‍♂️)... I suspect my 'missing addresses' are the ones with JsepTransport doesn't exist.?:

(basic_ice_controller.cc:542): Sorting 6 available connections due to: new candidate pairs created from a new remote candidate
(basic_ice_controller.cc:546): Conn[3548d800:0:Net[en0:192.168.1.x/24:Wifi:id=1]:Wrzy2PUz:1:0:host:udp:192.168.1.x:65033->yPC1fSTD:1:2130706431:host:udp:129.2xx.xx.x:10000|C--I|-|0|0|9115038255648079871|-]
(basic_ice_controller.cc:546): Conn[3548c000:0:Net[en0:192.168.1.x/24:Wifi:id=1]:Wrzy2PUz:1:0:host:udp:192.168.1.x:65033->lHqX4gc0:1:2113932031:host:udp:129.2xx.xx.x:10000|C--I|-|0|0|9079268943356378622|-]
(basic_ice_controller.cc:546): Conn[3548e400:0:Net[en0:192.168.1.x/24:Wifi:id=1]:Wrzy2PUz:1:0:host:udp:192.168.1.x:65033->W3OqJAGp:1:2113932031:host:udp:129.1xx.xx.x:10000|C--I|-|0|0|9079268943356378622|-]
(basic_ice_controller.cc:546): Conn[3548f000:0:Net[en0:192.168.1.x/24:Wifi:id=1]:Wrzy2PUz:1:0:host:udp:192.168.1.x:65033->FEEaxa8V:1:2113932031:host:udp:129.1xx.xx.x:10000|C--I|-|0|0|9079268943356378622|-]
(basic_ice_controller.cc:546): Conn[35558000:0:Net[en0:192.168.1.x/24:Wifi:id=1]:Wrzy2PUz:1:0:host:udp:192.168.1.x:65033->RqIWlg4c:1:2113932031:host:udp:129.2xx.xx.x:10000|C--W|-|0|0|9079268943356378622|-]
(basic_ice_controller.cc:546): Conn[35558c00:0:Net[en0:192.168.1.x/24:Wifi:id=1]:Wrzy2PUz:1:0:host:udp:192.168.1.x:65033->RNfzDfku:1:2113932031:host:udp:129.1xx.xx.x:10000|C--W|-|0|0|9079268943356378622|-]
(p2p_transport_channel.cc:413): Channel[0|1|__]: Ice not completed yet for this channel as Net[en0:192.168.1.x/24:Wifi:id=1] has more than 1 connection.
(jsep_transport_controller.cc:300): Not adding candidate because the JsepTransport doesn't exist. Ignore it.
(jsep_transport_controller.cc:300): Not adding candidate because the JsepTransport doesn't exist. Ignore it.
(jsep_transport_controller.cc:300): Not adding candidate because the JsepTransport doesn't exist. Ignore it.
(jsep_transport_controller.cc:300): Not adding candidate because the JsepTransport doesn't exist. Ignore it.
(jsep_transport_controller.cc:300): Not adding candidate because the JsepTransport doesn't exist. Ignore it.
(jsep_transport_controller.cc:300): Not adding candidate because the JsepTransport doesn't exist. Ignore it.
(jsep_transport_controller.cc:300): Not adding candidate because the JsepTransport doesn't exist. Ignore it.
(jsep_transport_controller.cc:300): Not adding candidate because the JsepTransport doesn't exist. Ignore it.
(jsep_transport_controller.cc:300): Not adding candidate because the JsepTransport doesn't exist. Ignore it.
(jsep_transport_controller.cc:300): Not adding candidate because the JsepTransport doesn't exist. Ignore it.
(jsep_transport_controller.cc:300): Not adding candidate because the JsepTransport doesn't exist. Ignore it.
(jsep_transport_controller.cc:300): Not adding candidate because the JsepTransport doesn't exist. Ignore it.
(jsep_transport_controller.cc:300): Not adding candidate because the JsepTransport doesn't exist. Ignore it.
(jsep_transport_controller.cc:300): Not adding candidate because the JsepTransport doesn't exist. Ignore it.

I also noticed that the networks are changing. The first "networks" block has the right IPs:

(basic_port_allocator.cc:712): Count of networks: 12
(basic_port_allocator.cc:714): Net[utun4:fdb5:0000:0000:x:x:x:x:x/64:Unknown:id=10]
(basic_port_allocator.cc:714): Net[pdp_ip0:2600:1006:0000:x:x:x:x:x/64:Unknown:id=6]
(basic_port_allocator.cc:714): Net[pdp_ip1:2600:1006:0000:x:x:x:x:x/64:Unknown:id=8]
(basic_port_allocator.cc:714): Net[pdp_ip2:2600:1006:0000:x:x:x:x:x/64:Unknown:id=9]
(basic_port_allocator.cc:714): Net[en0:192.168.1.x/24:Unknown:id=1]
(basic_port_allocator.cc:714): Net[en2:169.2xx.0.x/16:Unknown:id=2]
(basic_port_allocator.cc:714): Net[pdp_ip0:100.1xx.xx.x/32:Unknown:id=5]
(basic_port_allocator.cc:714): Net[pdp_ip1:10.1xx.xx.x/32:Unknown:id=7]
(basic_port_allocator.cc:714): Net[utun5:129.1xx.0.x/16:Unknown:id=11] <--------
(basic_port_allocator.cc:714): Net[utun5:129.2xx.0.x/16:Unknown:id=12] <--------
(basic_port_allocator.cc:714): Net[lo0:0:0:0:x:x:x:x:x/128:Loopback:id=4]
(basic_port_allocator.cc:714): Net[lo0:127.0.0.x/8:Loopback:id=3]
(basic_port_allocator.cc:114): Filtered out ignored networks:
(basic_port_allocator.cc:116): Net[lo0:0:0:0:x:x:x:x:x/128:Loopback:id=4]
(basic_port_allocator.cc:116): Net[lo0:127.0.0.x/8:Loopback:id=3]
(basic_port_allocator.cc:913): Network manager has started
(network.cc:570): Network change was observed
(network.cc:157): Ignore link local IP:fe80:0:0:x:x:x:x:x
(network.cc:157): Ignore link local IP:fe80:0:0:x:x:x:x:x
(network.cc:157): Ignore link local IP:fe80:0:0:x:x:x:x:x
(network.cc:157): Ignore link local IP:fe80:0:0:x:x:x:x:x
(network.cc:157): Ignore link local IP:fe80:0:0:x:x:x:x:x
(network.cc:157): Ignore link local IP:fe80:0:0:x:x:x:x:x
(network.cc:157): Ignore link local IP:fe80:0:0:x:x:x:x:x
(network.cc:157): Ignore link local IP:fe80:0:0:x:x:x:x:x
(network.cc:157): Ignore link local IP:fe80:0:0:x:x:x:x:x
(network.cc:157): Ignore link local IP:fe80:0:0:x:x:x:x:x
(network.cc:157): Ignore link local IP:fe80:0:0:x:x:x:x:x
(network.cc:157): Ignore link local IP:fe80:0:0:x:x:x:x:x
(network.cc:157): Ignore link local IP:fe80:0:0:x:x:x:x:x
(network.cc:157): Ignore link local IP:fe80:0:0:x:x:x:x:x
(basic_port_allocator.cc:712): Count of networks: 3
(basic_port_allocator.cc:714): Net[en0:192.168.1.x/24:Wifi:id=1]
(basic_port_allocator.cc:714): Net[lo0:0:0:0:x:x:x:x:x/128:Loopback:id=4]
(basic_port_allocator.cc:714): Net[lo0:127.0.0.x/8:Loopback:id=3]
(basic_port_allocator.cc:114): Filtered out ignored networks:
(basic_port_allocator.cc:116): Net[lo0:0:0:0:x:x:x:x:x/128:Loopback:id=4]
(basic_port_allocator.cc:116): Net[lo0:127.0.0.x/8:Loopback:id=3]
(basic_port_allocator.cc:712): Count of networks: 3
(basic_port_allocator.cc:714): Net[en0:192.168.1.x/24:Wifi:id=1]
(basic_port_allocator.cc:714): Net[lo0:0:0:0:x:x:x:x:x/128:Loopback:id=4]
(basic_port_allocator.cc:714): Net[lo0:127.0.0.x/8:Loopback:id=3]
(basic_port_allocator.cc:114): Filtered out ignored networks:
(basic_port_allocator.cc:116): Net[lo0:0:0:0:x:x:x:x:x/128:Loopback:id=4]
(basic_port_allocator.cc:116): Net[lo0:127.0.0.x/8:Loopback:id=3]
(basic_port_allocator.cc:823): Allocate ports on en0 
(basic_port_allocator.cc:1426): Net[en0:192.168.1.x/24:Wifi:id=1]: Allocation Phase=Udp
(audio_device_buffer.cc:338): Size of playout buffer: 480
(port.cc:169): Port[4c55400::1:0:host:Net[en0:192.168.1.x/24:Wifi:id=1]]: Port created with network cost 10
(basic_port_allocator.cc:936): Adding allocated port for 0
(basic_port_allocator.cc:958): Port[4c55400:0:1:0:host:Net[en0:192.168.1.x/24:Wifi:id=1]]: Added port to allocator
(basic_port_allocator.cc:975): Port[4c55400:0:1:0:host:Net[en0:192.168.1.x/24:Wifi:id=1]]: Gathered candidate: Cand[:3738476015:1:udp:2122260223:192.168.1.x:65033:host::0:Sy65:vnJxOcjMaA8Y58KwYuCB6NHe:1:10:0]
(basic_port_allocator.cc:1008): Port[4c55400:0:1:0:host:Net[en0:192.168.1.x/24:Wifi:id=1]]: Port ready.

This seems to explain why it actually works 1 out of 20 times, based on timing.

In this run, the only candidate that "made it" was the 192.168.1.x address

I'll keep looking - Hopefully I can capture a working conference with logging

corby-github commented 3 months ago

I was able to capture the logs on a successful conference.

Here are a few findings... 1) I think the bug is somewhere in webrtc - but I am not sure. 1) The very first-time the app is started, networks are counted, and all networks are there. In my case, 10. Shortly after startup, networks are filtered out and the utun (vpn) interface addresses are removed (you can see this above) 1) The reason why it works sometimes, is due to timing, if the app can send out the ICE candidates to the JVB, it will work 1) You can see in the log attached that webrtc appears to be pruning the networks, maybe because of the network cost? 1) Even after the network is pruned, the conference continues to work (until its closed) 1) Once the network is removed, it will not come back when leaving the meeting and rejoining.

Log snippet:

(port.cc:889): Network cost changed from 50 to 10. Number of candidates created: 1. Number of connections created: 6
(basic_port_allocator.cc:712): Count of networks: 3
(basic_port_allocator.cc:714): Net[en0:192.168.1.x/24:Wifi:id=1]
(basic_port_allocator.cc:714): Net[lo0:0:0:0:x:x:x:x:x/128:Loopback:id=4]
(basic_port_allocator.cc:714): Net[lo0:127.0.0.x/8:Loopback:id=3]
(basic_port_allocator.cc:114): Filtered out ignored networks:
(basic_port_allocator.cc:116): Net[lo0:0:0:0:x:x:x:x:x/128:Loopback:id=4]
(basic_port_allocator.cc:116): Net[lo0:127.0.0.x/8:Loopback:id=3]
(basic_port_allocator.cc:898): Prune 7 ports because their networks were gone
(p2p_transport_channel.cc:2151): Removed port: Port[161c2300:0:1:0:host:Net[en2:169.254.0.x/16:Unknown:id=2]] 7 remaining
(p2p_transport_channel.cc:2151): Removed port: Port[161c2800:0:1:0:host:Net[pdp_ip0:100.1xx.xx.x/32:Unknown:id=5]] 6 remaining
(p2p_transport_channel.cc:2151): Removed port: Port[161c2d00:0:1:0:host:Net[utun7:129.1xx.0.x/16:Unknown:id=9]] 5 remaining
(p2p_transport_channel.cc:2151): Removed port: Port[161c3700:0:1:0:host:Net[utun7:129.2xx.0.x/16:Unknown:id=10]] 4 remaining
(p2p_transport_channel.cc:2151): Removed port: Port[16002800:0:1:0:host:Net[utun4:fd61:000:0000:x:x:x:x:x/64:Unknown:id=8]] 3 remaining
(p2p_transport_channel.cc:2151): Removed port: Port[16002d00:0:1:0:host:Net[pdp_ip0:2600:1006:0000:x:x:x:x:x/64:Unknown:id=6]] 2 remaining
(p2p_transport_channel.cc:2151): Removed port: Port[161f8000:0:1:0:host:Net[pdp_ip2:2600:1006:0000:x:x:x:x:x/64:Unknown:id=7]] 1 remaining
(basic_port_allocator.cc:1265): Removed 7 candidates

Full log: webrtc_debug_2.log

Note: In the log attached, I removed a lot of noise. Mainly (connection.cc:927) and (basic_ice_controller.cc:546) that was testing each candidate.

saghul commented 3 months ago

@bgrozev Any chance the problem here is that we don't have enough candidates when sending the answer to the JVB?

corby-github commented 3 months ago

@saghul - I am going to try to build webrtc myself...

I suspect my issue is here:

https://github.com/jitsi/webrtc/blob/13f5eaef79cacf4b9426166d452c139da1c1594c/sdk/objc/native/src/objc_network_monitor.mm#L75

Seems like adding a log stmt there would tell us more. Something like:

...
 auto iter = adapter_type_by_name_.find(interface_name);
  if (iter == adapter_type_by_name_.end()) {
    RTC_LOG(LS_VERBOSE) << "interface " << interface_name << " is not avilable. Will be ignored.";
    return {
        .adapter_type = rtc::ADAPTER_TYPE_UNKNOWN,
        .available = false,
    };
  }
...
corby-github commented 3 months ago

Looks like we are on the right track. The log snippet below shows interface [utun8] is not available. Will be ignored.

There seems to be some subtle difference in iOS 18 that is not properly triggering OnPathUpdate or not enumerating all networks in nw_path_enumerate_interfaces()

(basic_port_allocator.cc:712): Count of networks: 9
(basic_port_allocator.cc:714): Net[utun6:fdd4:3d3c:0000:x:x:x:x:x/64:Unknown:id=7]
(basic_port_allocator.cc:714): Net[pdp_ip0:2600:0000:0000:x:x:x:x:x/64:Unknown:id=5]
(basic_port_allocator.cc:714): Net[pdp_ip1:2600:0000:0000:x:x:x:x:x/64:Unknown:id=6]
(basic_port_allocator.cc:714): Net[en0:192.168.1.x/24:Unknown:id=1]
(basic_port_allocator.cc:714): Net[pdp_ip0:100.75.0000.x/32:Unknown:id=4]
(basic_port_allocator.cc:714): Net[utun8:129.1xx.0.x/16:Unknown:id=8]
(basic_port_allocator.cc:714): Net[utun8:129.2xx.0.x/16:Unknown:id=9]
(basic_port_allocator.cc:714): Net[lo0:0:0:0:x:x:x:x:x/128:Loopback:id=3]
(basic_port_allocator.cc:714): Net[lo0:127.0.0.x/8:Loopback:id=2]
(basic_port_allocator.cc:114): Filtered out ignored networks:
(basic_port_allocator.cc:116): Net[lo0:0:0:0:x:x:x:x:x/128:Loopback:id=3]
(basic_port_allocator.cc:116): Net[lo0:127.0.0.x/8:Loopback:id=2]
(basic_port_allocator.cc:913): Network manager has started
(network.cc:558): Network change was observed
(network.cc:155): Ignore link local IP:fe80:0:0:x:x:x:x:x
(network.cc:155): Ignore link local IP:fe80:0:0:x:x:x:x:x
(objc_network_monitor.mm:73): interface [pdp_ip1] is not available. Will be ignored.
(network.cc:167): Ignore deprecated IP:2600:0000:114b:x:x:x:x:x
(objc_network_monitor.mm:73): interface [pdp_ip1] is not available. Will be ignored.
(objc_network_monitor.mm:73): interface [pdp_ip0] is not available. Will be ignored.
(network.cc:155): Ignore link local IP:fe80:0:0:x:x:x:x:x
(objc_network_monitor.mm:73): interface [pdp_ip0] is not available. Will be ignored.
(network.cc:167): Ignore deprecated IP:2600:0000:0000:x:x:x:x:x
(objc_network_monitor.mm:73): interface [pdp_ip0] is not available. Will be ignored.
(objc_network_monitor.mm:73): interface [utun8] is not available. Will be ignored.
(objc_network_monitor.mm:73): interface [utun8] is not available. Will be ignored.
(network.cc:155): Ignore link local IP:fe80:0:0:x:x:x:x:x
(objc_network_monitor.mm:73): interface [utun6] is not available. Will be ignored.
(basic_port_allocator.cc:712): Count of networks: 3
...
corby-github commented 3 months ago

@saghul I found the issue:

In RTCNetworkMonitor.mm the nw_path_enumerate_interfaces_block_t block requires you to Return true to continue enumerating, or false to stop enumerating.

It appears that before iOS 18, this block was somehow retuning true. With iOS 18, it enumerates just one interface, and returns.

To fix this, we need to add a return true; at the end of the nw_path_enumerate_interfaces() function, after the map->insert(): https://github.com/jitsi/webrtc/blob/13f5eaef79cacf4b9426166d452c139da1c1594c/sdk/objc/components/network/RTCNetworkMonitor.mm#L91-L92

Locally, everything is working now and I now see the following log messages:

(RTCNetworkMonitor.mm:72 -[RTCNetworkMonitor initWithObserver:]_block_invoke): NW path monitor: updated.
(RTCNetworkMonitor.mm:79 -[RTCNetworkMonitor initWithObserver:]_block_invoke): NW path monitor status: satisfied.
(RTCNetworkMonitor.mm:89 -[RTCNetworkMonitor initWithObserver:]_block_invoke_2): NW path monitor available interface: en0
(RTCNetworkMonitor.mm:89 -[RTCNetworkMonitor initWithObserver:]_block_invoke_2): NW path monitor available interface: pdp_ip0
(RTCNetworkMonitor.mm:89 -[RTCNetworkMonitor initWithObserver:]_block_invoke_2): NW path monitor available interface: utun8

By the way, this will impact not just VPNs but any secondary interface on the device.

saghul commented 3 months ago

Nice!

Now of course the question is why that happens.

I'd suggest you open an issue on the WebRTC bug tracker.

I'll be happy to float a patch once there is something upstream.

corby-github commented 3 months ago

You got it! https://issues.webrtc.org/issues/359245764

saghul commented 3 months ago

@corby-github Spot on mate! I checked SDK headers and yeah we are missing a return I'd say. Since you did all the work, do you want to send a CL to upstream WebRTC? I can help you with the process.

corby-github commented 3 months ago

Happy to help!

Yeah. I am willing to submit the CL, with your guidance.

saghul commented 3 months ago

You can get started here: https://webrtc.org/support/contributing let me know if you run into any trouble!

corby-github commented 3 months ago

👍 Thanks! Looks straightforward. I'll give it a go!

corby-github commented 3 months ago

done!

saghul commented 3 months ago

Can you share a link? I guess you didn't set the Bug: tag to match https://issues.webrtc.org/issues/359245764 ?

corby-github commented 3 months ago

Hmm, I did. https://webrtc-review.googlesource.com/c/src/+/359541

saghul commented 3 months ago

👍

corby-github commented 3 months ago

Appreciate the help. Looks like it's ready to submit to CQ. If I need to do anything else, let me know. This workflow is new to me.

saghul commented 3 months ago

Will do, thanks for doing all the work!

corby-github commented 3 months ago

👏 The latest JitsiWebRTC version 124.0.1 is working perfectly now on iOS 18 Beta. I verified using both CocoaPods and Swift package manager (SPM).