web-platform-tests / results-collection

Other
41 stars 46 forks source link

Failed to automatically upgrade to latest release of STP #656

Closed jugglinmike closed 5 years ago

jugglinmike commented 5 years ago

On 2019-04-03, Apple released version 79 of Safari Technology Preview. This project is configured to use the latest release, so we expected it to automatically switch to version 79 in subsequent collections. This did not occur.

In the time since that release, we have been collecting results from STP version 78. Fortunately, we have been correctly reporting the version, so this is more inefficient than confusing.

jugglinmike commented 5 years ago

The problem was, once again, persisted state. This time, disk volumes from prior installation attempts are the culprit.

When mounting the STP package, macOS generally uses the same disk volume. Our script hard-coded that path. However, if the path is in use (as is the case if a prior installation failed uncleanly), then it dynamically generates a unique path by appending an integer.

The system continued to operate as expected because the hard-coded path described a valid package volume for STP. We didn't notice this immediately because the version of STP in that old volume happened to be the latest release.

It wasn't until the release of the new version that the error became apparent. Starting on April 3rd, the version of STP that the system downloaded no longer matched the version which it ultimately installed (via that hard-coded path).

We'd like to avoid any circumstance that would cause the system to accumulate old package volumes, of course. But in advance of that, I've updated the installation script to correctly infer the true location of the most recently-mounted volume:

https://github.com/web-platform-tests/results-collection/commit/0de1b63d0f41ae3123f546c4b1074c1cd875f146