Open joeyparrish opened 1 week ago
The root cause seems to be the test's call to DrmEngine.probeSupport(), which doesn't complete for more than 120 seconds on FirefoxWindows, causing the test to time out.
Furthermore, the test gets skipped on FirefoxWindows because Widevine support is not detected.
The platform support check runs in less than 2 seconds on both FirefoxLinux and FirefoxMac, and both correctly detect Widevine support.
On our Firefox Windows instance, one test for Widevine, no robustness string, no encryption scheme, through EME, takes 60 seconds.
The same through MediaCapabilities takes 120 seconds.
Running many of these in parallel with various schemes and robustness strings also takes 120 seconds, which is the only good news in my analysis of this so far. 😐
Checking ClearKey takes less than 1 second, so the issue is specific to Widevine.
The error from the EME query is "Timed out while waiting for a CDM update".
After going back through our nightly test history, this issue started after we upgraded our lab from Firefox 126 to 130. According to https://bugzilla.mozilla.org/show_bug.cgi?id=1908144, they switched to Chrome's component update service to get the Widevine CDM in Firefox 129.
I can confirm that outside of test automation and geckodriver, on startup at https://shaka-player-demo.appspot.com/support.html, Firefox seems to need to reload the page after downloading the Widevine CDM. This all happens really fast in a live session.
I tried using a fixed profile folder, so that Widevine could be pre-installed there, but it hangs without starting tests. In experiments running the Selenium node for Firefox outside of the service account, I can see Firefox start up, but it never receives a command to go to the test URL.
If I run Selenium outside of the headless service account without a custom profile folder, everything just works, including downloading, installing, and running the Widevine CDM without issue.
So the root trigger is headless mode in a service account. The profile folder, which attempts to work around that issue, causes a different, unknown issue, and is not a good workaround.
Running outside the service account, in a UI-enabled environment, but with the "-headless" argument to Firefox, the CDM does work.
NB: Running the service scripts outside the service account, as an admin, might have broken some file permissions and forced me to wipe and reinstall shaka-lab-node. Oops.
Running inside the service account, without a UI, but with the "-headless" argument to Firefox, the CDM does not work.
So the service account environment is the key to triggering this, regardless of whether Firefox is run "headless".
Have you read the FAQ and checked for duplicate open issues? Yes
If the problem is related to FairPlay, have you read the tutorial?
N/A
What version of Shaka Player are you using?
main
Can you reproduce the issue with our latest release version? Yes
Can you reproduce the issue with the latest code from
main
? YesAre you using the demo app or your own custom app? Integration tests
If custom app, can you reproduce the issue using our demo app? N/A
What browser and OS are you using? Firefox 130 on Windows 10.
For embedded devices (smart TVs, etc.), what model and firmware version are you using? N/A
What are the manifest and license server URIs?
N/A: Integration tests
What configuration are you using? What is the output of
player.getNonDefaultConfiguration()
?N/A: Integration tests
What did you do?
Run tests in the lab with
--browsers FirefoxWindows --filter 'unloads properly after DRM error'
What did you expect to happen? Tests pass
What actually happened?
The one test in the filter times out
Are you planning send a PR to fix it? Yes