Status/plans for iOS support?

[sburnicki]

Hi, I just wanted to ask what the current status or plans for iOS support are? Are the official iOS test agents already running with this code, or with the old one?

Thanks for your work!

[pmeenan]

They are still running the old code. I have plans for a new iOS agent that won't require jailbreak and can be driven by a Pi but haven't had time to build it yet. It will require building the app yourself for deployment though (only way to get remote debugging).

The short-version is:

• Build a wkWebView shell app to do the testing (now that it has the same perf as Safari)
• Have it communicate directly with the agent code over USB
• Use Record Kit for recording video of the page loading

[sburnicki]

Sounds good. Do you plan to somehow expose the complete remote debug protocol to the Pi, or do you want to implement the test logic in the iOS app?

[sburnicki]

Oh, and do you already have experience with reverse tethering iOS over cable? We still use WiFi, but I found some interesting approaches in blogs. E.g. https://9to5mac.com/2017/03/01/ios-10-2-ethernet-adapter-ui-settings-app/

[j0weiss]

Hi,

I saw that you started the iTether project and I would like to contribute to the iOS support in the upcoming time. Do you already have a precise plan how to realise the reverse tethering? Can you think of a good starting point to help? Maybe at the other end with the app containing the web view?

Thanks for your time!

[pmeenan]

I'm still experimenting with the feasibility of using Network Extensions to run a VPN over usbmuxd. I'm not sure yet if it is viable to do it system-wide (may not be able to open a listening socket from the network extension).

If that fails I will have to move the tethering into the app itself instead of having a separate tether app (usbmuxd connection to the app and have the network extension open an outbound connection to the app which forwards it over usbmuxd). In that case I'd have to check that the network extension can connect localhost to the app socket.

Basically still experimenting with the various on-device security blocks to see what will work for connecting the network extension to a socket running over usbmuxd

[pmeenan]

btw, as far as wired ethernet goes, as best as I can tell, the camera adapter kills the ability to communicate over usbmux (the lightning plug just carries charge) and the current plan is to use the ios debug proxy for safari to run the dev tools protocol to the tethered host (and it requires usbmux).

If the webview app can actually connect to the remote inspector port locally for the embedded webview then that would change things up and wired ethernet might be possible without trying to tether in software (don't think you can connect to it locally though).

[j0weiss]

Thanks for your insights. I also found the ios-webkit-debug-proxy and remotedebug-ios-webkit-adapter. And I started testing both components to control the web view on the iOS device via the dev tools protocol. Did you already have a minimal working example?

[pmeenan]

FYI, the browser side of the development is going on here. I don't have the agent side of things implemented yet but the command/control can be automated over telnet using tcprelay.py since it's just delimited text on a socket.

So far it supports loading the browser, navigating pages, providing status notifications and executing arbitrary js so it's in pretty good shape. The embedded webview is debuggable from Safari so I assume I'll be able to hook the proxy up to it and talk to the dev tools protocol for instrumentation.

Hopefully I'll have basic testing up and running later next week.

The current plan is to put devices into single app mode so they automatically re-launch on crashes, reboots, whatever, remote control the app over the socket (already working) and record the events over the dev tools protocol.

Video capture looks like it might be a little uglier than I had hoped. ReplayKit prompts users for permission to record the screen every 8 minutes which would be a non-starter (and prior to iOS 11 you couldn't access the recordings). I'll still experiment with it to see if there is a way to automate the prompt since it has the lowest recording overhead but I may have to fall back to one of the uglier "screen capture every 100ms" type hacks.

Fingers crossed I'll also be able to rotate the UI programmatically which will let the same device run portrait and landscape tests.

[pmeenan]

cc @andydavies so you can follow along if you're interested.

[andydavies]

@pmeenan On the ReplayKit prompt…

Could you swizzle the prompt method so that it always returns true and never actually shows the dialog? (in the same way the various SSLStrip implementations work on iOS)

[pmeenan]

@andydavies do you happen to have any links? My Google foo is failing me and not turning anything up.

That said, I have a couple of other options I'm looking at right now. I'm currently hooking up ASScreenRecorder to see how it performs.

Another option is the screen capture code I already have hooked up waits for the next screen update to record the screen (quickly with low overhead) and it looks like it triggers screen updates when things change, not at a constant 60fps so that might also work.

Swizzling feels a bit fragile so I'd like to stay on documented paths if possible. As it stands right now, if iOS 11 ships with resource timing it would be possible for a light version (no http headers) to be used from the play store.

I tested single app mode and it works exactly as I hoped. If the device reboots or the browser exits it automatically restarts it and it works great with the developer-installed app (and still exposes the web inspector interfaces).

Cache clearing is hooked up and working as is device rotation (so the same agent can test portrait and landscape). On the device side the only thing left is the video capture (and eventually tethering but that will be after the initial release).

I'm hoping to have some level of video capture sorted out early in the week and start plumbing the wptagent control code. Hopefully by the end of the week I'll have some testing working.

[pmeenan]

Video capture with ASScreenRecorder is working beautifully. I have it intentionally capturing at the scaled resolution instead of 1:1 which will help with CPU utilization and file sizes but it is humming along at 60fps without dropping a beat. It captures at 1:1 with no issues as well but WPT doesn't need full retina video for doing the processing and the scaled video is more than good enough.

As it stands it should support iOS 9+ (most of my testing is on iOS 10.3.3). So far all of the API's used are public and app-store worthy. The only reason it needs to be built and deployed manually is for the remote inspector data (which also works well).

I should be able to have the python side of things wired up this week and it should be deployable to see how well it works under regular testing load by the end of the week.

[andydavies]

@pmeenan I think swizzling may be on the fragile side

Did some work recently using Frida to extract client random's and master keys from app traffic so I could dcrypt TLS traffic (could only get it work for apps, not Safari as havedn't sorted extracting keys for session resumption yet)

Will install the iOS code and see what I can find in Frida for blocking the prompt in case you need it

[stefounet]

Thx for the work on this, I'm looking forward to seeing it in action!

[j0weiss]

This is very nice work! And thanks for sharing your thoughts, which helps a lot. The single app mode sounds pretty cool and helpful.

I'm excited to see the progress.

I also have a question concerning the debugging of the webview. I posted it as a issue on iWptBrowser.

[pmeenan]

Agent is shaping up nicely. wptagent can now talk to iWptBrowser and it can basically run a test, capture the screen shots, video and js-based metrics.

The web inspector interface isn't hooked up yet (so no waterfalls) and it doesn't handle page load failures yet but it is making good progress.

Here is a sample filmstrip from loading AOL.

The visual metrics and navigation timing data should all be good and all of this can be collected from a version that can be published to the app store (as well as waterfalls if resource timing ships).

Hopefully over the next week I can hook up the web inspector interface and port the processing logic from the existing agent over. At that point it should at least be as good as the existing agent and we can start hammering on it.

[pmeenan]

Dev tools is now hooked up for OSX hosts (but not decoded/processed) and events are coming in as expected. I'm optimistic that by the end of next week it should be in good enough shape to replace the existing agent and work from a tethered Raspberry Pi.

Things get complicated with the webinspector proxy if multiple devices are attached so the current plan is to only support one device per host (perfect for the Pi case). At some point in the future support for multiple devices can be added.

[pmeenan]

Testing is in pretty good shape now with full waterfalls and quite a few improvements over the existing agent:

• Improved timing visibility (dns, socket, ssl timings)
• Initiator information
• Response bodies support (which enables optimization checks)

The timeline data is being retrieved but not processed yet (necessary for the JS timings and main-thread activity) but I'm not going to hold up the migration for that. The agent also doesn't have a lot of time on it yet so I'm sure there are still some edge cases (like the content breakdown isn't currently reporting).

Next up is to do some private testing to fix any rough edges and add the bits necessary to get it working on a Linux host (x64 and arm). If all goes well I'll migrate the public agents over at the end of the week.

[pmeenan]

OK, the missing data (mime types) has been fixed as well as a few edge cases and it is looking solid. As of a few minutes ago it also works from Linux (64-bit and arm) hosts and I have a device successfully running from a raspberry Pi.

I need to see if I can automatically run usbmux if the system doesn't have it running but for right now if you have usbmuxd and libimobiledevice installed and can "ideviceinfo" to talk to the device then the wpt agent will work.

Next steps:

• Move the public iOS agents to the new agent and drive them from a Raspberry Pi
• Document the install and setup process
• Add the console log messages

Then I'd call it a stable release for general consumption. Other things I'll be working on after the initial release:

• Add support for running landscape and portrait tests on the same device (browser supports it, need to plumb it through the agent and UI)
• Add the timeline processing logic and integrate with the hosted timeline viewer
• See if I can get reverse tethering working

[stefounet]

Many thanks for the work and for regularly giving some news!

[pmeenan]

The public iPhone 6 agent is now on the new code, powered by a Raspberry Pi (and POE). I'll move the other devices over later this week when I get some more Pi's in.

Documentation and more features coming but it's ready.

[soulgalore]

Wow fantastic work @pmeenan and really great to follow your comments, thanks a lot! One thing I've seen when I tested is that data URLs are picked up in the waterfall (in our case : data:image/svg+xml,...): https://www.webpagetest.org/result/170913_NF_f6f7280e00845037c1d9a854ca3d1216/

[pmeenan]

@soulgalore Thx, Should be Fixed now. It will take up to an hour for the public agents to pick up the fix but here is a local test I ran to verify it.

At this point it looks stable and I'm calling the initial release done so any issues (or missing capabilities) that show up can be tracked as separate issues. I'll close this issue out as soon as the documentation is live (hopefully later today)

[pmeenan]

First round of docs are up and cover the device setup.

Next up will be some docs on the wptagent side for configuration there (it is basically just a --iOS command-line flag but I need to add some logic to auto-install usbmux if not already installed). After that I'll create a doc for running traffic-shaping from an external bridge.

[sburnicki]

Looks like multistep does not yet work with the agent: http://www.webpagetest.org/result/170918_PM_0325dca64b56d29869d2019f5beed54c/

[pmeenan]

Thanks, looking at it now. The plumbing should all be in place for it to work so hopefully it is just a simple bug.

[pmeenan]

Looks like the agent is working OK and running both steps but the server doesn't realize there are multiple steps. Looks like the Firefox agent has the same issue. Should be a quick fix to teach the server how to find the results for all of the steps.

[pmeenan]

All fixed (for both iOS and Firefox). You'll need to pick up this fix on the server side for counting the steps.

This week I'm adding the console log as well as timeline processing and then it should be feature complete (planning on working on reverse-tethering next).

I'm particularly interested to see how the A11 chip in the iPhone 8 does for JS compared to the Android devices (also Safari vs Chrome but I expect the chip dominates any differences). I have an iPhone 8 arriving on Friday so I hope to have the timeline processing in place well before that.

[sburnicki]

Wow, thanks for the fast fix!

I'm also quite excited for the results with the new iPhone!

[pmeenan]

Closing this out as the agents should be feature-complete for iOS at this point (timeline and console were added on Monday). I have a separate issue to track the reverse-tethering work and anything else that pops up should be tracked as separate issues.

The docs for device and agent set-up should also be in place (though remote shaping could use some better documentation and a walkthrough).