What are the plans to support macOS 10.11 and iOS?

[MrMage]

I am interested in using SwiftNIO's HTTP2 implementation (once available) to add an NIO-based server implementation to https://github.com/grpc/grpc-swift. Given that that project currently depends on several other fairly large libraries, it would be great to also move the client code to SwiftNIO to eliminate those dependencies.

For that to happen, though, SwiftNIO would need to support those client platforms — namely iOS (preferably 10+) and macOS 10.11. Neither currently shows up in the "Supported Platforms" list — does that mean it's technically impossible to support those platforms (e.g. because some low-level networking APIs are missing), or has the library simply not been tested on those platforms, but might already work as-is? Other than that, what level of effort would be required to add support for these two platforms, or is it already planned, and with what timeline?

[Lukasa]

Great question, and thanks for your interest!

Let me break this down slightly into two separate parts. First, macOS 10.11. I do not think there would be much technical difficulty in supporting that target, though some compatibility code may be required. That would be easy enough to do though, and should only require changing some if #available guards and fixing any compile and test failures that occur.

For iOS, the problem gets a bit thornier. NIO as it exists today uses the BSD sockets API for all its I/O, and pthreads for its thread management. This works just fine on macOS and Linux, but on iOS it's a distinctly suboptimal solution. On iOS neither of these solutions interacts well with the OS: BSD sockets don't provide information to the OS about what's needed from the radios, and pthreads don't let the kernel optimise scheduling the way Dispatch does.

Our ideal long-term solution for iOS work is to have an alternative set of event loops and channels that are built to treat iOS as a first-class citizen, using its platform native APIs to build a solution that works elegantly the rest of the operating system.

Currently the most concrete information I have about timelines for iOS is that we're looking into scheduling that work, but it's not currently in our top list of priorities. We have a bunch of other protocol work we want to do first, to really flesh out the server side of the story.

[helje5]

BSD sockets don't provide information to the OS about what's needed from the radios

I'm not entirely sure what do you mean by that, would you be so kind to elaborate? (I'm using a BSD socket server + GCD on iOS, and it is not clear to me what else you could do here)

pthreads don't let the kernel optimise scheduling

On iOS I would assume you would never run more than a single thread aka MultiThreadedEventLoopGroup(numThreads: 1). I can't see how this wouldn't be sufficient (if a handler does expensive work, he can always dispatch out to GCD).

I'm probably going to give NIO on iOS a run later this week.

[Lukasa]

I'm not entirely sure what do you mean by that, would you be so kind to elaborate?

Sure, my source here is the official networking documentation, which contains this note:

In iOS, POSIX networking is discouraged because it does not activate the cellular radio or on-demand VPN.

On iOS I would assume you would never run more than a single thread aka MultiThreadedEventLoopGroup(numThreads: 1). I can't see how this wouldn't be sufficient (if a handler does expensive work, he can always dispatch out to GCD).

Oh sure, that would definitely work. I should probably have been a bit clearer. On iOS it's advantageous to give Dispatch as much insight into the work patterns of the code as possible, to ensure that it is able to schedule work effectively on the system. This means that from our perspective our ideal model keeps the general architecture of NIO, but uses different primitives on iOS to achieve the same end result.

Of course, users would always be able to select whatever model they wish to, subject to availability on the platform. We just want good options. :grin:

[helje5]

In iOS, POSIX networking is discouraged because it does not activate the cellular radio or on-demand VPN.

OK, but this is for software connecting away from the device, it doesn't matter for the server part. (I'm not aware of any other API one could use here). In fact the very same documentation says that right after the part you quoted :-):

For daemons and services that listen on a port, or for non-TCP connections, use POSIX or Core Foundation (CFSocket) C networking APIs.

I wonder how you would hook up non-Posix stuff for outgoing traffic (that is NSURLSession). Certainly possible and probably required for the reasons mentioned.

This means that from our perspective our ideal model keeps the general architecture of NIO, but uses different primitives on iOS to achieve the same end result

OK. So I read this as you plan to use GCD on iOS to back the event loop. I currently use GCD in my own device server (in a single serial ~event loop~ sigh, dispatch queue). I don't think that running on more than 1 thread is very useful on device. The only thing I think I might wish for is a way to give the NIO thread a utility priority.

[Lukasa]

OK, but this is for software connecting away from the device, it doesn't matter for the server part.

Ah, we were talking past each other: I was talking about client usage. Yes, for server usage BSD sockets are the way to go.

So I read this as you plan to use GCD on iOS to back the event loop.

I mean, we won't remove the MultiThreadedEventLoopGroup from iOS builds, so in situations where you'd prefer to have a single OS thread, that will continue to be an option. We just want to also provide a Dispatch-based abstraction.

[helje5]

BTW: The way I read the docs, a Dispatch based solution wouldn't actually fix the issue. I don't know what CFStream does extra to do the (radio/vpn) stuff a BSD connect doesn't. But I suppose it would be easier to just add that code section for iOS targets, instead of switching the whole stack (unless that part is closed source of course).

[MrMage]

@Lukasa sorry for the delay in replying to this. Thank you for the feedback, this is very illuminating!

I noticed that MultiThreadedEventLoopGroup only works on macOS 10.12 and above — do you know if there's a specific reason for excluding 10.11 there? (I assume the team just used if #available(OSX 10.12, *) to check for macOS, with less focus on the specific version.) Also, what event loop is being used under Linux, given that MultiThreadedEventLoopGroup wouldn't work?

[Lukasa]

As discussed above, macOS 11 has been excluded only because it was not actively considered a target we needed to support. There is no profound reason that I'm aware of, and support could be added by dropping the version in those guards and fixing whatever compile errors occur.

The MultiThreadedEventLoopGroup is being used on Linux. You're misreading that guard. The asterisk means "on all platforms not explicitly mentioned, enter this block".

[normanmaurer]

Also the check is not needed anymore as we use our own Thread class and not Foundation.

See https://github.com/apple/swift-nio/pull/233

[MrMage]

The MultiThreadedEventLoopGroup is being used on Linux. You're misreading that guard. The asterisk means "on all platforms not explicitly mentioned, enter this block".

So if #available(OSX 10.12, *) means "on macOS, but only 10.12 or newer, or any other platform that is not macOS"? Thanks, I was not aware of this.

[Lukasa]

Yeah, that's what the guard means. :smile:

[Lukasa]

I believe this is addressed in the current master. Please let me know if it's not.

[normanmaurer]

+1 this should just work now

Am 29.03.2018 um 11:44 schrieb Cory Benfield notifications@github.com:

Closed #130.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[SetoKaiba]

I'd like to know whether it just does not work on iOS. Or it works, but it does not activate the cellular radio or on-demand VPN?

[Lukasa]

The second.

[SetoKaiba]

Is there any time frame for a iOS support? https://stackoverflow.com/questions/25045829/how-can-i-activate-the-cellular-radio-in-ios-since-bsd-sockets-doesnt-activate

According to this post, it seems that I can just listen for Tcp and do nothing. It will keep the cellular radio or on demand VPN activated. Does it mean that this can be a temp workaround for iOS using?

I really love to use Netty like mechanism since my server is in Java based on Netty. I can keep them look the same. Or is there any Netty like lib supporting iOS? Thank you for your reply.

[SetoKaiba]

I noticed that you mentioned the pthread management used by swift-nio. And I found the #233 says that it removes the check, because swift-nio uses its own thread impl now. Is MultiThreadedEventLoopGroup(numThreads: 1) and dispatch to GCD mentioned by @helje5 is still preferred?

[helje5]

I’m not sure what the latter question relates too. It doesn’t sound like the radio/vpn issue? But yes, on a device I can’t imagine why you’d want to have more than one NIO thread. (unless you put it in a rack an use it as a server ;-) )

You definitely don’t want to do the former. (only listen to keep the radio active) The users will hate you ;-) (btw the SO article seems to mirror my experience, that TCP seems to be fine w/o any extra tricks, but who knows. and i didn’t try VPN)

[Lukasa]

Whether NIO works for your needs is entirely dependent on what your needs are. In general, BSD sockets are less than ideal on iOS because the contract given by the operating system does not give you much feedback about the state of the system. The OS is free to do as it pleases with your BSD sockets in non-main-line scenarios (e.g. background your app could invalidate your FDs without notifying you).

Again, the best we can say is that if iOS provides a higher level API for what you’re trying to do, you are best suited using it. If it does not, NIO should work fine as a wrapper around the BSD sockets/pthreads library. GCD integration is on the roadmap, but not immediately imminent.