Adopting iOS 9 network extension points

[clowwindy]

Network extension points: Use the Packet Tunnel Provider extension point to implement the client side of a custom VPN tunneling protocol. Use the App Proxy Provider extension point to implement the client side of a custom transparent network proxy protocol. Use the Filter Data Provider and the Filter Control Provider extension points to implement dynamic, on-device network content filtering. Each of the network extension points requires special permission from Apple.

[conradev]

Each of the network extension points requires special permission from Apple :(

[clowwindy]

Now that Apple allows anyone to run the code on their own devices, we don't have to publish the app on the App Store.

No, it still requires some entitlements to run on the devices.

[conradev]

Totally, but - the API documentation is hard to piece together and there is no template in Xcode for the extension point. Gonna have to do some reverse engineering.

[clowwindy]

There's no documentation at all at the moment. The headers of NetworkExtension.framework are public, so we can figure out how to implement the proxy.

I guess we need to subclass NEAppProxyProvider to handle both NEAppProxyTCPFlow and NEAppProxyUDPFlow. And somehow activate the proxy.

Or we can subclass NEPacketTunnelProvider to create a VPN tunnel that handles NEPacketTunnelFlow.

[conradev]

Totally. We need to find the extension point identifier, too. Cisco and OpenVPN need to update their apps...

[clowwindy]

I guess it works just like an app that controls IPSec VPN settings. Before calling manager.connection.startVPNTunnelAndReturnError, we should register our own protocol with

[NETunnelProviderManager loadAllFromPreferencesWithCompletionHandler:(void (^)(NSArray<NETunnelProviderManager *> * __nullable managers, NSError * __nullable error))completionHandler]

I'll give it a try when I have time.

[conradev]

I'm going to wait for the single WWDC session before diving in

[clowwindy]

NEAppProxyProvider is actually per-app exclusive. Good news is we can use NEPacketTunnelProvider to create global VPN services.

I'm writing to Apple to see if we can get permission for the API.

[blackgear]

A ref: WWDC: What's New in Network Extension and VPN

[icodesign]

Have you made any progress on packet tunnel?

[clowwindy]

Still no reply from Apple.

[icodesign]

I'm writing to Apple to see if we can get permission for the API.

So does this mean only those who have grant permissions from Apple can develop global proxy apps?

[clowwindy]

I'm afraid yes.

[icodesign]

I'm afraid yes.

Sad but reasonable. Good luck with SS. :pray:

[muenzpraeger]

The NEAppProxyProvider API only require a MDM deployed app. That can be "simulated" as described in the video.

[xmxsuperstar]

https://developer.apple.com/library/prerelease/ios/releasenotes/General/WhatsNewIniOS/Articles/iOS9.html

[angelovAlex]

There're actually templates for Xcode. You need to install them from

/System/Library/Frameworks/NetworkExtension.framework/Versions/A/Resources/NEProviderTargetTemplates.pkg

But I have not found the way of how to activate a vpn. As there's no shared instance for NETunnelProviderManager I think we need to create a new one.

[NETunnelProviderManager loadAllFromPreferencesWithCompletionHandler:^(NSArray<NETunnelProviderManager *> * __nullable managers, NSError * __nullable error) {

        if (managers.count <= 0){
            NETunnelProviderProtocol *protocol = [[NETunnelProviderProtocol alloc] init];
            protocol.providerConfiguration = @{ @"some parameter" : @"some value" };
            protocol.providerBundleIdentifier = @"com.example.vpn.vpntunnel";

            NETunnelProviderManager *manager = [[NETunnelProviderManager alloc] init];
            [manager setProtocol:protocol];
            [manager setLocalizedDescription:@"My VPN"];
            [manager setOnDemandEnabled:NO];
            [manager setEnabled:YES];

            [manager loadFromPreferencesWithCompletionHandler:^(NSError * __nullable error) {
                NSLog(@"%@", error);
            }];
        }
    }];

On the line NETunnelProviderManager *manager = [[NETunnelProviderManager alloc] init];, the following message appears in the console app:

6/27/15 5:31:13.845 PM VPNOSX[1403]: Application does not have the required entitlements.

It doesn't say which entitlements and there's no any documentation about it. I want to try this api on MAC OS 10.11. I understand the reason why I need to ask apple for some permission to publish the app with this api to app store, but I can't believe that I have to ask them for permission to run this api on my development machine. Sorry, that's a little bit off topic, but that is the only thread that I found in the internet so far.

[clowwindy]

Yes. You need to send an email to Apple to get the entitlements. And I'm waiting for their reply.

[manjonn]

Any luck on this yet? I am looking at NEAppProxyProvider for a project for a client. I think I do understand some things, but can't be sure till I can run it on the device.

[EkkoG]

https://developer.apple.com/library/prerelease/ios/samplecode/SimpleTunnel/Introduction/Intro.html#//apple_ref/doc/uid/TP40016140

Maybe this demo is helpful to this problem?

[clowwindy]

let newManager = NETunnelProviderManager()

You'll get a warning complaining about missing entitlements when you execute this line of code.

[angelovAlex]

In README.md it says:

The NEProvider family of APIs require the following entitlement:
<key>com.apple.developer.networking.networkextension</key>
<array>
    <string>packet-tunnel-provider</string>
    <string>app-proxy-provider</string>
    <string>content-filter-provider</string>
</array>
</plist>
The SimpleTunnel.app and the provider extensions will not run if they are not code signed with this entitlement.
You can request this entitlement by sending an email to networkextension@apple.com.

If you try to compile the app with this entitlement, your app will be killed by taskgated daemon. If you like to move com.apple.taskgated.plist from /System/Library/LaunchDaemons with root permission, you will get a nice response:

sudo mv com.apple.taskgated-helper.plist ~
mv: rename com.apple.taskgated-helper.plist to /Users/alex/com.apple.taskgated-helper.plist: Operation not permitted

means that you are not admin now, you are nothing and you are in sandbox:

7/9/15 12:37:27.138 PM sandboxd[113]: ([3711]) mv(3711) System Policy: deny file-write-unlink /System/Library/LaunchDaemons/com.apple.taskgated-helper.plist

[muenzpraeger]

We just received the entitlements.

[clowwindy]

Got the entitlements, too.

[jedisct1]

[jedisct1]

Did you apply as an individual or as a company?

I didn't dare filling the form because it seemed like you had to apply as a company.

[clowwindy]

I applied as an open source organization. I explained a bit about this project in the Company name and address field.

[clowwindy]

Update:

Now I can get a virtual tun device running and route packets through UDP. While I find it a little hard to debug as I can't attach to the extension.

[clowwindy]

Now I have ShadowVPN fully working on an iPad. The next step is to add UI, etc.

https://github.com/clowwindy/ShadowVPNiOS

[clowwindy]

Jul 19 12:17:56 new-iPad ShadowVPN(NetworkExtension)[1242] : MDM must be used to create NEAppProxyProvider configurations

Looks like NEAppProxyProviderManager isn't for us. Thus to implement Shadowsocks for iOS, we need convert it to a VPN. Maybe we have to port tun2socks to iOS.

[madeye]

@linusyang has a working port of tun2socks on iOS. Maybe he can help.

[clowwindy]

@linusyang

In some old version of Shadowsocks-iOS, I had polipo's main function renamed and called from a background thread. We can do the same to tun2socks and turn it into libtun2socks. Instead of letting tun2socks create the tun device, we'll create and pass the tun fd to its main function.

Since the VPN extension is a hard place for debugging, we may debug libtun2socks on OS X where it's possible to create a tun fd in a normal process. Then the only thing left is to change the project target.

I think I'll begin working on this in mid August after I'm done with ShadowVPN and ChinaDNS (we have to do some DNS hijacking since it's not possible to listen on port 53). According to records iOS 9 will be released around Sep 18.

If you're interested in porting the library, I've created a repo for it. https://github.com/shadowsocks/tun2socks-iOS

[chrisballinger]

@clowwindy I did some work on this a year ago but I don't really remember how far I got because I eventually hit a wall.

https://github.com/chrisballinger/badvpn/tree/tun2socks-xcodeproj https://github.com/chrisballinger/badvpn/tree/darwin-utun

Not sure if any of that is useful anymore.

[conradev]

I'm happy to help in any capacity that I can. I want to get Tor working on iOS, which will require the same NEPacketTunnelProvider -> SOCKS bridge.

Correct me if I am wrong, but the parts we need are:

• Something to parse/create the IPv4/6 headers of incoming/outgoing packets.
• A SOCKS4/5 client

These pieces on their own don't sound awful to implement while looking at existing implementations – or am I misunderstanding the complexity?

[chrisballinger]

@conradev Awesome work! I have been reaching out to other people who have expressed interest in collaborating on an iOS 9 Tor implementation and I'm glad to see you've already made some progress. Sent you an email. :)

[clowwindy]

@conradev @chrisballinger Good to see we can join forces on the tun to socks bridge!

[clowwindy]

@linusyang I've added you to the team. You can push directly to the repo.

[linusyang]

@clowwindy Thanks!

[wasimshigri]

@clowwindy @linusyang @conradev @chrisballinger any one having a sample code for NetworkExtension to scan the wifi list? I guess NEHotspotHelper will be used, but i dont know how to implement

[AshokkumarDev]

Do application that uses Extension Framework can run in background or should i do something specific to run my application in background?How do I run my custom VPN application in background?

[manjonn]

I working on getting NEAppProxyProvider working. Not much luck so far. I think it the Proxy has to be configured either through the code or through an MDM config profile. Don't know which one. Is anybody working on this? Would like to compare notes.

[mikeKane]

Using the extension Framework I can get it working on the idp_ip0 but if I start with en0 it gives an error 49. However if I switch from idp_ip0 -> en0 it works. Any one else see this problem?@clowwindy

[conradev]

@clowwindy, @linusyang, would love your thoughts on this document.

Trying to actually figure out how to implement this library. I don't have the entitlements yet, so it's all I can do while I wait.

[tredds]

Hei, have you been able to run the sample code? I'm getting "Error Domain=NEConfigurationErrorDomain Code=11 "IPC failed" UserInfo={NSLocalizedDescription=IPC failed}" when trying to use NEFilterManager. I created provisioning profiles for each extension + the main app. I expect this to be a signing problem, but I just can't see it. Thanks

[clowwindy]

Update: ShadowVPN-iOS is now fully functional.

I've created UI, added CHNRoutes and ChinaDNS. You can also see how to deal with network status change (Wi-Fi/4G). You can try to compile and run it if you have the entitlements from Apple.

The bad news is, I found that UDP traffic often gets blocked very quickly (~10 minutes) in 4G network in China. After toggling Airplane mode twice to get a new 4G IP, the server received data again. But after a while it got blocked again.

So I'm coming back to working on Shadowsocks again. It seems that we can't get the file descriptor from NEPacketTunnelProvider API. As @conradev proposed, we can pass the data directly from and to PacketPassInterface and PacketRecvInterface and let tun2socks reassemble the TCP flow. For UDP, I think it's really easy to write our own forwarder rather that using tun2socks.

[angelovAlex]

@tredds are you running it on device? I was getting this error when was trying to run it in ios simulator.

[tredds]

@angelovAlex Indeed I was running on simulator. I'll give it a try on device.

[ionull]

@clowwindy on China Mobile or China Unicom 4G network?

[clowwindy]

@ionull China Unicom

[clowwindy]

@conradev @linusyang tun2socks is based on lwIP, which is a pure C, full stack TCP/IP protocol implementation.

We can just build our own Socks adapter from lwIP and GCD, without tun2socks's event loop.