web-platform-tests / wpt

Test suites for Web platform specs — including WHATWG, W3C, and others
https://web-platform-tests.org/
Other
5.03k stars 3.12k forks source link

[wptrunner] Sauce Labs assets are handled unsafely #10513

Open jugglinmike opened 6 years ago

jugglinmike commented 6 years ago

System configuration scripts for Windows and macOS are uploaded to Sauce Labs' internal storage location prior to test execution. Currently, these files are sourced from the local file system, and each upload operation over-writes any previously-uploaded file with the same name. This makes it unsafe to run multiple tests in parallel using the same Sauce Labs account. If the two test executions rely on different versions of the same file, it is possible that one of them will be invalidated if the other over-writes that file.

Two potential workarounds come to mind:

Unfortunately, the Sauce Labs Connect proxy does not appear to be enabled when the "pre-run" scripts are executed. Details regarding the experimentation that supports this hypothesis are included below. I have filed a ticket with Sauce Labs technical support to verify this and to identify workarounds. For now, it appears that the first option above is a non-starter. Although this problem was initially identified in service of gh-9205, I felt a dedicated GitHub issue was warranted given the intricacy of this problem and given that it is technically orthogonal to the original issue.

Prerun tunnel experiment The following Sauce Labs test report includes script logging output which suggests that the proxy is not available at the moment of execution: https://saucelabs.com/beta/tests/abf64c20f1824e9b91b6cbf8f1a25645/logs#12 When the following commands are included in a Sauce Labs "prerun" script: powershell -Command "(new-object net.webclient).DownloadString('http://zombo.com')" 2>&1 powershell -Command "(new-object net.webclient).DownloadString('http://web-platform.test:8000')" 2>&1 The produce the following output: ZOMBO Exception calling "DownloadString" with "1" argument(s): "The remote name could not be resolved: 'web-platform.test'" At line:1 char:1 + (new-object net.webclient).DownloadString('http://web-platform.test:8 ... + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + CategoryInfo : NotSpecified: (:) [], MethodInvocationException + FullyQualifiedErrorId : WebException
jugglinmike commented 6 years ago

Sauce Labs technical support has verified my theory regarding the unavailability of the proxy. Here's the latest:

Unfortunately, by design, pre-runs are executed before Selenium starts, > meaning Sauce Connect isn't available to them yet. I can make a request to > have this changed, but I don't think it's likely to happen; This is the first > time we've had a request like this.

I've asked the technician to make that request on our behalf, but it sounds like we shouldn't get our hopes up. That makes the second option described above our best bet, unless anyone can offer a simpler solution.