AIR Runtime on macOS corrupts SecureRTMPChannel

[busitech]

The AIR Runtime corrupts SecureRTMPChannel connections on macOS, but not on Windows. Our application makes RemoteObject calls to the server over this channel, which begins to hang forever on macOS after a few transactions. The initial connection is usually successful, and sometimes even a few successful remote calls can be made before the hang occurs.

Compiler: AIR SDK 32.0.0.116 AIR Runtime: 32.0.0.116 to 50.2.1.1 Platform: macOS Ventura 13.3.1

[busitech]

Turning on disableNetworkChanges has circumvented the disconnect problem. We opened a connection on Friday, and on Monday it was still connected and usable.

Have you updated your development environment to the latest Sonoma release to make sure you have the same operating system we do for testing this issue? This should be giving you very frequent disconnects without having to cycle a VPN connection.

[ajwfrost]

That's good to hear! I will double-check, I thought our systems were up to date. But we weren't seeing that same issue with the disconnections. We're just using wifi, in case you're using an ethernet cable for this, maybe that makes a difference?

thanks

[busitech]

@ajwfrost When will a permanent fix for this issue be seen the AIR runtime? The permanent fix should be one that does not require special options to be set in our source code, or a special build of the AIR application to be released.

We are expecting that end users would simply be able to update their AIR runtime in order to access this fix for macOS, while using presently deployed versions of our AIR applications.

We do not intend to make any permanent changes to our development environment or to have to recompile our various AIR applications in order for users to obtain a stable environment for their existing AIR applications in later macOS releases.

[ajwfrost]

Hi

probably you can skip all this text and just jump to the 'update' section below.. but I'd written it and don't really want to delete it...

The issue for us is that we don't see the same behaviour as your systems are showing. You mentioned earlier:

A recent update to macOS has introduced a new architecture for communicating with other Apple devices. macOS is now constantly creating and tearing down utun (user space network tunnel) interfaces via background processes. Some of these processes are named remotepairingd and remoted. The setup and tear down of these utun interfaces cause many new "network changed" events to be sent to all applications monitoring the status of interfaces. The macOS console logs reveal copious amounts of details surrounding these events.

We don't see this though, we are not getting any spurious "network changed" events. We've checked on two different macs which have the latest software on them; the only things I am wondering that may be different are (a) are you using ethernet cables? as we are only on wifi; and (b) we are using arm64-based devices rather than intel..

But, it was also clear from the earlier testing/profiling that when your systems are sending this "network changed" event, the list of available network interfaces did not include the one that was being used for the RTMP connection, which is why there was then a manual tear-down of the socket.

I don't really like changing the existing behaviour of the runtime without having some opt-in or opt-out type mechanism, but we could look perhaps at the RTMP code ignoring these network changes and just relying on the underlying OS socket implementation to let us know if there's an issue. Based on your findings with that new AS3 flag, this is all that would be needed. The risk of course is that someone else might be relying on the early detection of a changed network in order to rapidly respond/reconnect. So it's really just a case of two options:

1. Opt-in - like we have now, you have to make a change in the code in order to have the RTMP handling ignore these spurious network adaptor changes
2. Opt-out - which is what you're after here, i.e. we make the change for everyone but provide a way for folk who needed it to change their app so they can get back to the original functionality.

Of course, it would be much nicer if the OS didn't send these notifications or didn't miss out the required network information when we queried it, and if we had a way to reproduce it here then we would be more likely to see if there's a better workaround we could do (such as perhaps delaying a few milliseconds before checking the list of network interfaces - since from what you say, and the behaviour of other apps, this is just a glitch that shouldn't actually impact a socket connection).

Update

Just doing a little more searching to see if we could find any reasons behind these messages, and we found that the same problem had been caused in Chrome. After a lot of analysis, they had essentially added an "ignore network change notifications" but in a limited/targeted way. So, we should be able to do the same... https://issues.chromium.org/issues/40939844#comment58

Seems like the issue causing these notifications is related to the wider use of Apple connected devices on the local network, hence us not reproducing it here in the dev environment.

Let me get someone to inspect the changes that the Chromium team did, and try to do the same thing as default in AIR...

thanks

[busitech]

@ajwfrost It is incorrect to assume that macOS is broken, and that no sense can be made of the events you receive. These two assumptions are leading you to the false dichotomy of ignore vs. do not ignore. The "ignore" concept leads to a serious design flaw, the one implemented by Chromium.

My belief is that after working with Apple to understand their changes, such as by opening a case with Apple Developer Support and discussing the APIs you're using to monitor the interfaces, and the events received as a result of the utun interface changes, you will be able to achieve the original functionality without ignoring events, while fixing the bug we have reported at the same time.

The bug is tearing down perfectly good connections. The trigger is a network change event for an interface you're not using.

The most important next step on your end is to do whatever it takes to reproduce the issue, and capture full details in your debugger. I would imagine that the interface being changed is part of the event.

All we have in the office for Apple products are two iPhones on WiFi and two Intel iMacs on Ethernet.

If in the end, you believe that Apple has introduced a bug in the APIs you are using, please, for the sake of all mankind, report the issue so they can fix it! : )

[busitech]

• Has the issue been recreated in your lab?
• When will a fix be available?
• Will all previous functionality be preserved?
• Will the fix be available as a runtime only?

[ajwfrost]

Hi @busitech - sorry, I should have posted an update here. Based on the findings from that Chromium team (and their later updates!) we have implemented a fix for this. We did see these events, albeit only occasionally, when we had a variety of iPads, iPhones and MacBooks on the network.

So to answer the questions:

• Yes, reproduced and we have a fix for this to avoid the spurious messages
• This will go into our next 51.x release. If we do another 50.2.x update, we'll back port it into that one too.
• Well kind of! The only change to functionality is in when (how frequently, or rather, on that basis) we send out the "network changed" update (which is what was causing the rtmps connection to drop, and looking at it, I think part of the subsequent issue was in how we detect/notify the addresses that are available..). If that earlier AS3 update worked for you to make the connection stable, then this fix should be a better way of doing that, without needing the AS3 change.
• We will put this into the shared runtime too so when we do the AIR 51.0 runtime release, this fix will be in it

Hope that helps!

thanks

[busitech]

Thank you, that is wonderful!

[busitech]

Do you know when the shared runtime will be refreshed?
The last release is from 22-Nov-2023, AIR 50.2.4.1.

[ajwfrost]

@busitech it should have been yesterday! We had some network issues though.. we should be able to get the website updated today.

[busitech]

@ajwfrost After testing 51.0.1.3 for a while, we have concluded that the issue is not resolved. Today I saw a disconnect within a few seconds of connecting, and others in our office have noticed disconnects as well. I'm sorry to share that the issue needs additional effort. Thank you.

[ajwfrost]

Hi - thanks for the feedback .. we'll get this checked out.