Open joeyparrish opened 2 months ago
Logs of the last successful runs of Firefox on Mac and Edge on Mac:
Firefox 125.0 (Mac OS 10.15): Executed 2572 of 2643 (skipped 71) SUCCESS (8 mins 18.375 secs / 8 mins 10.21 secs)
Edge 124.0.0.0 (Mac OS 10.15.7): Executed 2623 of 2643 (skipped 20) SUCCESS (9 mins 36.587 secs / 9 mins 28.626 secs)
Neither Edge nor Firefox appear in /Applications in CI now. 😭
Google Chrome for Testing.app
Google Chrome.app
PowerShell.app
Python 3.11
Python 3.12
Safari.app
Utilities
Xcode.app
Xcode_14.3.1.app
Xcode_14.3.app
Xcode_15.0.1.app
Xcode_15.0.app
Xcode_15.1.0.app
Xcode_15.1.app
Xcode_15.2.0.app
Xcode_15.2.app
Xcode_15.3.0.app
Xcode_15.3.app
Xcode_15.4.0.app
Xcode_15.4.app
This could be explained by this GitHub blog post:
The macOS 12 runner image will remain latest until migration of the latest YAML workflow label to macOS 14 in Q2 FY24 (April – June 2024). While macOS 13 is now generally available under the macos-13 label, this image will not be migrated to latest. Following this announcement, macOS 11 runner image will begin deprecation immediately with retirement expected to complete by June 2024.
It is now Q2 FY24, so we may have just been auto-upgraded from macOS 12 to macOS 14 on M1, and they may not pre-install the same browsers on M1 images.
Indeed, this table now shows:
Image | YAML Label |
---|---|
macOS 14 | macos-latest-large or macos-14-large |
macOS 14 Arm64 | macos-latest, macos-14, macos-latest-xlarge or macos-14-xlarge |
macOS 13 | macos-13 or macos-13-large |
macOS 13 Arm64 | macos-13-xlarge |
So if we want x86-64, we need to use macos-13
explicitly. This may not matter much for Shaka Player, but some of the native C++ projects under shaka-project may need to explicitly choose macos-13 (x64) or macos-latest (arm64) in their build matrices.
For Shaka, we will just have to explicitly install browsers now.
Installing Firefox & Edge fixes Firefox and shows Edge now failing the same as Chrome, with a series of timeouts.
Installing Edge & Firefox explicitly solves the failure to start tests.
Switching to macOS 13 (x64) seems to resolve the timeouts on Chrome and Edge. It's not a great permanent solution, but I think it's best for now until we understand these timeouts better.
Tests are failing or failing to run on Mac, but only in CI. Everything works in the lab.
The last build-and-test job to succeed was this one for the 4.8.0 release PR: https://github.com/shaka-project/shaka-player/actions/runs/8819839697/job/24212073498
Since then, it always fails, with errors like this for Firefox and Edge:
So webdriver-installer is unable to find installed versions of Firefox and Edge, which likely means they have moved in the CI environment.
Chrome on Mac fails tests instead, with a long cascade of timeouts like:
These are very hard to explain. These tests passed in 4.8.0, but have failed since. Chrome on Mac has not been upgraded, as we can see the same browser & OS version numbers in the logs of the last successful run:
If a solution can't be found quickly, we may need to disable these in CI and rely on lab test runs to gate releases. This may result in Mac-specific regressions that are not found until release time.