Unit Tests on SauceLabs consistently failing to launch one of the Firefox instances

[Jaifroid]

Something must have changed on the SauceLabs side, because every other browser test launches and complete correctly up to that point, including Firefox 45, but on running "Firefox", we get this error every time:

08 10 2022 09:19:14.143:WARN [launcher]: firefox   on SauceLabs has not captured in 60000 ms, killing.
08 10 2022 09:19:14.145:ERROR [SaucelabsLauncher]: Could not quit the Saucelabs selenium connection. Failure message:
08 10 2022 09:19:14.146:ERROR [SaucelabsLauncher]: TypeError: Cannot read property 'deleteSession' of null

See for example https://github.com/kiwix/kiwix-js/actions/runs/3209884425/jobs/5246942344.

[Jaifroid]

Maybe the problem is here: https://github.com/kiwix/kiwix-js/blob/master/tests/karma.conf.saucelabs.js#L18 . Do we maybe need to specify a version number for Firefox? Or maybe something like "latest"?

[Jaifroid]

This issue on StackExchange seems to describe a similar situation, with suggested solution, but it is from some time ago, so I don't really trust that it's the same thing, or else our tests would have stopped working a couple of years back: https://stackoverflow.com/questions/61528952/webdriverio-test-on-saucelabs-is-not-working-for-firefox

[Jaifroid]

@kelson42 This is the ticket for the failure with Sauce Labs that I mentioned in our meeting. But actually the issues are wider and ongoing: see #737. The problem with the alternative mentioned there, Puppeteer over GitHub Actions, is that it wouldn't test older browsers, as it would be limited to what is available on GitHub Actions. BrowserStack has an Open Source programme: https://www.browserstack.com/open-source.

On this specific issue, the funny thing is that a very old Firefox (45) still works. What doesn't work any more is the latest Firefox, but we test manually on the latest Firefox with no problem. I made #895 to test different versions. As mentioned, however, even when it was working Sauce Labs would fall over regularly, and almost every PR had to be manually re-run, sometimes twice or three times, before it would complete. Now they just don't work at all.

[Jaifroid]

@Krinkle As you very kindly helped us with the Sauce Labs configuration some time ago, I wondered if you might have any insight or hints as to the cuase of this persistent failure to run the latest version of Firefox? I have tried various things in #895, but none of them have worked so far. The error corresponds to this one: https://docs.saucelabs.com/dev/error-messages/index.html#the-sauce-labs-virtual-machine-failed-to-start-the-browser-or-device, and it is suggested there that it is either an incompatible version of Selenium, or else a request for an incompatible platform/OS/browser combination.

[kelson42]

@julianharty Maybe you have an idea to help to fix this problem? Or you know someone who would know?

[Jaifroid]

@kelson42 @Popolechien I wonder if it's time to consider trying out BrowserStack as an alternative testing platform. They have an Open Source programme here: https://www.browserstack.com/open-source . We would need an institutional Kiwix account on it, ideally one I have full access to, but that is clearly identified with Kiwix Org.

I have access to emails sent to info@kiwix.org (via a login that @Popolechien gave me), so in principle I could set up an account using that email. Or you may have a better idea? I don't currently have my "own" Kiwix address (e.g. jaifroid@kiwix.org) -- I haven't felt the need for one till now.

[kelson42]

@Jaifroid No problem on my side, I'm just not really confortable to move because there is a problem we don't understand... but I have no solution to propose to better understand the problem... so....

[Jaifroid]

Well I could just disable the faulty Firefox test, because the basic tests we run do run on latest Firefox headless on GitHub actions, without needing Sauce Labs. And this might be an interim solution. I agree, it's not great to move just because there's a problem that maybe could be solved with a tweak, or with updating the way we access their API.

On the other hand, a BrowserStack account could be useful anyway to try out alternatives, assuming it really is free to an Open Source non-profit. It also (it seems) gives access to a full VM of a wide range of OS/browser combinations that can be interacted with manually, not just through automatic testing.

[Popolechien]

@Jaifroid I have no opinion on this particular service (or switch to a new service) but would entirely support your using the info (or contact@kiwix org, as the other one is a redirect) to access whatever service you need for your work. Just make sure you share the password with myself and @kelson42.

[Krinkle]

See my comment at https://github.com/karma-runner/karma-sauce-launcher/issues/236.

Can you reproduce this issue locally? To do so, you need to set the two environment variables with sauce credentials. These can be even for a (free open-source) personal account if it's just a handful of tests. Otherwise you may have to get a hold of the Kiwix ones.

[Jaifroid]

@Krinkle I have the Kiwix credentials. Thanks for the link to a related issue. I'll look into this. It's not happening locally, but we're not running anything other than Chrome and Firefox headless locally, not the full range of cross-browser tests we run remotely.

[Krinkle]

@Jaifroid Understood. The saucelabs variant can be run locally however, it does not have dependencies that we install especially in CI afaik. Running it locally might help with understanding how/when it fails, having a quicker turnaround time and able to easily make changes rather than going through GitHub.

[Jaifroid]

Ah, OK, thanks for the clarification. I'll try that.

[Jaifroid]

I tested locally using #909 (which simply enables the firefox latest environment). It fails with exactly the same errors as on GitHub. See screenshots.