jgonggrijp
commented
2 years ago @ogonkov @paulfalgout I got the Sauce Labs tests to run, mostly reliably, with the exception of Firefox 11. See the details of the above (failed) workflow run.
If you agree with the logic changes you see here, we can merge this. However, we need to make a decision about the lowest Firefox version.
The tests do pass in Firefox 11, as you can see on Sauce Labs: https://app.eu-central-1.saucelabs.com/tests/08d1010d4bea46dda25b29f2a6a9a698. However, something is going wrong in the connection with Karma. I tried this on a couple of OS versions and I had a single attempt on Windows 10 where the connection suddenly worked (run 53). So far, I haven't been able to reproduce this, but this is the reason why the karma.conf-sauce.js currently lists Firefox 11 with Windows 10.
I also tried Firefox 18, 25 and 32, but they all behaved similarly flaky, although versions 18 and 32 eventually succeeded on second attempts. See run 55.
I see three possible courses of action:
- Keep Firefox 11 and accept that it will fail on most runs.
- Drop Firefox 11 without replacement, making 40 the oldest version. This would mean that we are only testing relatively recent versions of Firefox.
- Find a version between 11 and 40 with a better tradeoff between reliability and age. This can be done with a "quadrisect" (cf
git bisect) where 18-25-32 was the first recursive step.
Please let me know what you think!
ogonkov
This is a continuation of @ogonkov's work at #4251, which I'm personally pushing directly to the main repo in order to rule out any effects from permissions. I also wanted to try a couple of things. If I end up merging this branch, #4251 will automatically be recognized as merged as well.