Automate results collection from Mobile Safari

[jugglinmike]

At WWDC2019, a WebKit maintainer announced, "WebDriver is coming to iOS [version 13]." We'd like to take advantage of this and configure WPT's infrastructure to publish test results for Mobile Safari to wpt.fyi (just as it does for "desktop" Safari today).

I'm not aware of any plans for Azure Pipelines to offer access to iPhones, but we should be able to use the iOS Simulator in Apple's Xcode to run Mobile Safari on the macOS systems that WPT currently uses. Test results collected from an emulated version of a browser will not necessarily match those collected from the genuine article, but it's not clear how much they will differ. We should also run a trial on a physical device so we can gauge the authenticity of the emulation.

The last hurdle (that we anticipate!) will be provisioning an environment in Azure Pipelines. All of this hinges on software that is currently available only on a "preview" basis: macOS 10.15 and Xcode 11. While we can manually opt in to these releases on locally maintained hardware, that's not an option in Azure Pipelines. To automate the process, we'll need to wait for Azure Pipelines to support this software. That's likely to occur after it is released as the current, stable version. There's no official schedule for this, but fans estimate it to occur in mid September, 2019.

[jugglinmike]

Last week, @burg published a post on WebKit's blog with some technical instructions on using WebDriver in iOS 13. I haven't been able to get it working yet, so I'm working with Apple developer relations folks to learn more.

[foolip]

We won't be pursuing this in Q4, so decreasing the priority.

[burg]

foolip wrote:

I'm not aware of any plans for Azure Pipelines to offer access to iPhones, but we should be able to use the iOS Simulator in Apple's Xcode to run Mobile Safari on the macOS systems that WPT currently uses. Test results collected from an emulated version of a browser will not necessarily match those collected from the genuine article, but it's not clear how much they will differ. We should also run a trial on a physical device so we can gauge the authenticity of the emulation.

I can't think of any reason why device and simulator results would differ for WebDriver test suite. Such an inconsistency would be concerning, and definitely worth reporting.

[foolip]

It was actually @jugglinmike who wrote that, but it's not the tests in webdriver/ that seem the most likely to differ in an emulator vs. real device. I would expect differences overall to be very small and perhaps dominated by flakiness, but there are probably some bugs that only manifest in one or the other.

@jugglinmike are radar issues files for all the things you ran into when trying to make this work previously?

[jugglinmike]

@jugglinmike are radar issues files for all the things you ran into when trying to make this work previously?

Yes; @burg identified two separate issues from the "sysdiagnose" file I shared with them and filed a report in the radar system for each.

[foolip]

I discussed this to varying degrees at https://webkit.org/meeting/ with @dcrousso and Jonathan Bedard (handle?).

I think man safaridriver has a lot more useful detail than in any of the online docs, so I've exported it here: https://gist.github.com/foolip/a0d7ac345a67a09a709ef7b28b3924ce

[foolip]

I had this slide in https://trac.webkit.org/wiki/ImprovingInteropwithWPT:

My wish immediately came true, with the clues I got and just using {"platformName": "iOS"} as a constraint I got ./wpt run churning through tests on a real iPhone: https://photos.app.goo.gl/U69KxHoDrkC8hxjn8

Turns out it won't look like my mockup, Safari under automation is orange, and there's no time to see anything before it closes :D

Some issues to work through:

• I used wpt.live, how to use the host macOS computer's running instance of wptserve? Is it possible to control the DNS in some way?
• Resizing the viewport for reftests isn't possible.
• It looks like the testharness.js setup with one tab opening and controlling another isn't working.

So it doesn't work, at all. But I don't see any obstacles to continuing to make progress on it. I'll self-assign and tinker some more next week.

[burg]

Some issues to work through:

• I used wpt.live, how to use the host macOS computer's running instance of wptserve? Is it possible to control the DNS in some way?

Don't know, but the answer would help me too.

• Resizing the viewport for reftests isn't possible.

Correct.

• It looks like the testharness.js setup with one tab opening and controlling another isn't working.

This should work. Please file a bug. Or steps to reproduce here, at least.

[foolip]

I think actually that last problem is just because I wasn’t pointing the device at the wptserve instance that is started by ./wpt run, so it boils down to the first DNS problem.

[foolip]

@burg per https://stackoverflow.com/a/3648775 it looks like for the iOS simulator it might work to just edit /etc/hosts of the macOS machine running it. Have you found that to work, or is it controlling the DNS of a real device that you've wanted to do before?

[foolip]

Unassigning since I'm not actively working on this.

[burg]

Internally, I have set up port forwarding on the device for the webserver port, so tests can just send requests to localhost. For public consumption, editing the hosts file, or being able to load files from the webserver over a local hostname rather than localhost would be better.

On Nov 14, 2019, at 3:32 AM, Philip Jägenstedt notifications@github.com wrote:

@burg https://github.com/burg per https://stackoverflow.com/a/3648775 https://stackoverflow.com/a/3648775 it looks like for the iOS simulator it might work to just edit /etc/hosts of the macOS machine running it. Have you found that to work, or is it controlling the DNS of a real device that you've wanted to do before?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/web-platform-tests/wpt/issues/17856?email_source=notifications&email_token=AAARIGE4Z3RXX7NVB4OYDKTQTUZMPA5CNFSM4IEC6GR2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEEBQZ3A#issuecomment-553848044, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAARIGD6HAZBVK42UXK2CXDQTUZMPANCNFSM4IEC6GRQ.

[burg]

Phillip wrote:

Some issues to work through:

I used wpt.live, how to use the host macOS computer's running instance of wptserve? Is it possible to control the DNS in some way? Addressed in previous comment. Resizing the viewport for reftests isn't possible. The resize endpoints should return "not supported" for MobileSafari as it doesn't support changing the window size. It looks like the testharness.js setup with one tab opening and controlling another isn't working. It is supported to create and switch between tabs in iOS WebDriver. So if this is failing via WebDriver commands, please file a bug at feedbackassistant.apple.com http://feedbackassistant.apple.com/

On Nov 2, 2019, at 3:42 PM, Philip Jägenstedt notifications@github.com wrote:

I had this slide in https://trac.webkit.org/wiki/ImprovingInteropwithWPT https://trac.webkit.org/wiki/ImprovingInteropwithWPT: https://user-images.githubusercontent.com/498917/68076526-e44f1d80-fd72-11e9-8124-e61945d206c9.jpeg My wish immediately came true, with the clues I got and just using {"platformName": "iOS"} as a constraint I got ./wpt run churning through tests on a real iPhone: https://photos.app.goo.gl/U69KxHoDrkC8hxjn8 https://photos.app.goo.gl/U69KxHoDrkC8hxjn8 Turns out it won't look like my mockup, Safari under automation is orange, and there's no time to see anything before it closes :D

Some issues to work through:

I used wpt.live, how to use the host macOS computer's running instance of wptserve? Is it possible to control the DNS in some way? Resizing the viewport for reftests isn't possible. It looks like the testharness.js setup with one tab opening and controlling another isn't working. So it doesn't work, at all. But I don't see any obstacles to continuing to make progress on it. I'll self-assign and tinker some more next week.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/web-platform-tests/wpt/issues/17856?email_source=notifications&email_token=AAARIGHJ76PCFLKVSXK3U5TQRX66LA5CNFSM4IEC6GR2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEC5GTSA#issuecomment-549087688, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAARIGG62LJZT3VAYO2PQKLQRX66LANCNFSM4IEC6GRQ.

[foolip]

Some code I experimented with is in https://github.com/web-platform-tests/wpt/pull/20097.

@burg, some follow-up questions, which might help if anyone would like to pick it up in the future.

Internally, I have set up port forwarding on the device for the webserver port, so tests can just send requests to localhost. For public consumption, editing the hosts file, or being able to load files from the webserver over a local hostname rather than localhost would be better.

Is it possible to set up such port forwarding on a real iPhone or in the simulator? For the hosts file, I guess you mean the hosts file of the mac computer controlling the iOS device? So if the hosts file names the IP of the computer, that should just work? How about certificates, is the certificate store of the controlling computer also used, or what would one need to do to make https://web-platform.test:8443 work? (In my experiment, I was using a real domain with a real certificate.)

As for the resize command and open tabs being an apparent problem, it would need some debugging to know what the cause is. Perhaps there are just invalid assumptions made in wptrunner that are true of all desktop browsers but not a mobile browser.