stephenmcgruer
commented
4 years ago @zouhir - are you still interested in pursuing using Saucelabs here?
Open zouhir opened 4 years ago
stephenmcgruer
commented
4 years ago @zouhir - are you still interested in pursuing using Saucelabs here?
zouhir
commented
4 years ago yes we are -- we'd really love to delete all these custom server & webdriver we have here: https://github.com/flackr/scroll-timeline/tree/master/test
LukeZielinski
commented
3 years ago Periodic ping - are there still plans to work on this in the next few months?
stephenmcgruer
commented
3 years ago Unfortunately at this point I doubt we will be able to action this issue from the Ecosystem-Infra side. If the interactions team (cc @flackr ) are interested in this we could probably re-prioritize in either Q1 or Q2 2021, but we've not heard from them on this at all.
We're also quite ham-stringed by not having access to saucelabs - we'd probably need to arrange at least a temporary account for testing if we wanted to get this working.
Of course, PRs are always welcome :)
fabiancook
commented
2 years ago Hi, I am interested in putting some time into a solution that works more widely, but I have had success within the app-history space running the tests individually through playwright against an independent app-history implementation
I am still testing this further to verify these results, however currently this app-history implementation shows tests passing for the current CI run of them.
The current code for this functionality can be seen here
To inject the appHistory variable itself, I needed to set it on globalThis before the test script loaded, or to define it directly in scope. Going with the direct shift and replace worked the easiest for this first pass, it allowed me to mark the test script too as type="module" and utilise const { AppHistory } = await import("/esnext/index.js");
This implementation is targeted towards more than just the web, supporting deno, & node, so it does not have any direct integration yet with location, a bunch of the tests available however relied directly. In the wrapping code for each test, there is injected code to mock location, window, and i/iframe. This renders all these tests more in the realm of unit tests for me, as I am no longer directly integrating with the browser (outside of using it to resolve the given scripts), this however is still within the scope of what I am after here.
To detect pass/fail I had mocked the usage of the async_test + test functions (+ others) utilised by the tests, this then calls the function exposed to the window through playwright. (I am looking to drop these in favour of using testharness.js, but I was looking for the most direct approach to achieve testing stability quickly)
These tests take about 3+ minutes to run at the moment, to install the git repo as a dependency takes a moment, so by default it is not included as a dependency outside of the CI job, and instead is installed each time CI runs
From this implementation's point of view I can step through each failing test one by one, and align to the expectation of the reference implementation
The What Help client-side polyfills for new Web platform features get tested via WPT and Saucelabs.
The Why Not all browsers implement a feature at the exact same time and developers tend to rely a lot on client-side shims & polyfills until a feature is available in the browser their customers are using.
While polyfills are heavily used and sometimes for an extended period of time, they are in a lot of cases not held to the same testing standards Web platform features are held onto, and developers cannot easily understand what covered and what not, and if there any caveats.
Existing Runner Capabilities & Bugs It's very promising the WPT runner had support for Saucelabs see CLI documentation, however, we ran into multiple issues that prevented us from successfully using WPT runner, and instead we wrote our own selenium-webdriver + saucelabs runner, in other words "reinvented the wheel".
I filed individual issues for each of our blockers:
Final notes & credits We'd love to collaborate with the WPT team to get our ScrollTimeline polyfill tested using WPT's harness tests, and help document our work so other teams do the same thing.
Thanks to @stephenmcgruer for guiding me through various issues I hit with WPT and his partnership to help us simplify our custom WPT webdriver / test runner.