osma
commented
1 year ago Hi @kinow , I'm belatedly replying to this as we are currently setting up the development processes for Skosmos 3 (see the skosmos-3 branch and the project board). End-to-end testing is something that we would like to implement, although we haven't made a firm decision yet on how and whether to do this.
Thanks for the Cypress recommendation. I've been mostly looking at Selenium, but at this point, any tool that gets the job done and is nice to work with is possible.
You describe in your OP how we could integrate the Cypress (or other) tests by just making some changes to the GitHub Actions workflow. Indeed it would be very useful to have E2E tests that are run automatically under the CI environment.
But I'm wondering how the E2E framework (Cypress, Selenium or other) would access the Skosmos application under the CI environment. So far we've only been running Composer and PHPUnit there, which both execute using PHP CLI commands. But there is currently no web server in that environment that could provide working URLs served by Skosmos. In my understanding, Selenium at least needs to be given URLs for loading the page(s) that then gets tested. So we need some web server that provides this. I found many tutorials on setting up Selenium with PHP, but basically they all gloss over the question of how Selenium accesses the PHP application over the web. They just assume that there is a web server running somewhere.
However, I found at least two clues on possible solutions:
- PHP CLI includes a built in web server that could be started to provide a minimum web server that could be able to serve Skosmos-generated HTML pages.
- I also found this discussion where apparently Apache has been set up to run under GitHub Actions. We could also set up a minimal Apache installation under the CI environment, just like developers typically run Apache locally on their machines.
kinow
New feature request, so removed the bug template.
At $work we use Cypress, which while similar to Selenium, is way more stable and less error-prone (i.e. you won't have to add a lot of
setTimeouts in your code to handle dynamically loaded resources, it uses Promises instead).Skosmos has several cases that are not easy to be covered by unit tests.
The #1016 bug, for example, requires a test vocabulary (which we have, with the
initFuseki.shscript starting a Fuseki instance), and requires the user to visit a vocabulary page, then click on a concept. The error can be verified by inspecting the browser console message. These last steps are all easily achieved with a tool like Cypress.The #1182 issue, another example, has a case where when you start typing a search term, it will create a URL concatenating certain parameters. When I updated Typeahead/Bloodhound, the parameters were not concatenated anymore due to changes in the JS third-party API. While @osma spotted it during the review, the code worked (slower) and it could have resulted in a performance regression. We can use a Cypress to not only drive the UI, but also act as a middleware capturing JS requests and either forwarding to real Fuseki or ending the test asserting that the URL contains certain parameters. That would prevent the regression @osma spotted.
In order to use Cypress, we could create an
e2efolder with apackage.json,cypress.json, and the tests. It wouldn't require any changes to Skosmos, only to the GitHub Action workflow.There are other tools too, like Nightwatch, cavy, etc.
Another benefit of adding a tool like this with tests now, is that it would future-proof Skosmos should it ever moves to a different UI architecture (SPA, PWA, jQuery-only without Twig, Vue, React, etc). We would be able to modify the UI and just wait for the tests to run and verify that we didn't break anything.
It would have been helpful to have a suite of e2e tests for the #1182 PR.