Open kinow opened 5 years ago
Typically such plugins take an image snapshot of the entire application under test or a specific element, and then compare the image to a previously approved baseline image.
@oliver-sanders are these the kind of tests you would like for Cylc UI?
Absolutely.
A visual diff tool should subtract one image from another leaving you with an image of the things which have moved / changed e.g:
http://diffee.me/?http://una.im?http://una.im/diffeedemo
We can gather screenshots using something Cypress, Selenium or whatever (can be done on Travis) and assemble "reports" using open source visual comparison libraries.
The big issue is storing/displaying those reports (can't be done on Travis) and GH integration. There are systems out there but they cost a fortune.
The big issue is storing/displaying those reports (can't be done on Travis) and GH integration. There are systems out there but they cost a fortune.
https://www.cypress.io/pricing
Cypress has a service with a free tier. 3 users and 500 recordings.
The open source plan https://www.cypress.io/oss-plan/ allows 5 users and unlimited recordings.
@hjoliver pointed we had a pending item from a previous meeting, about e2e tests. We have e2e set up already in the project, but no useful test for now. That's mainly due to the fact that what we had before was more of a prototype.
Soon we will have the final screens for the application UI, and then we should be ready to write good e2e functional tests.
However, during the same meeting, @oliver-sanders mentioned an issue with the progress display in the
Task
component. The issue we had was related to a CSS expression that wasn't working on Chrome. In theory, having a e2e test would cover it, but by default would not detect if that was broken.Cypress (similarly to Selenium and Nightwatch) provide only functional automated web tests. So only tests that confirm that the web application has the features from its requirements, and that they work as expected (things like the colors are right, the page loads fast enough, the gauge arrow is displayed, are normally considered "non-functional requirements").
Cypress documentation mentions "Visual Testing", see this part of their docs
@oliver-sanders are these the kind of tests you would like for Cylc UI?
I don't think we are quite ready to introduce these tests at the moment. IMO, we better wait a bit longer, until we are certain the components and views are ready, and then choose whether to use this testing technique or not, and if we decide to use, then choose what to test.
But in anyway, I'd prefer to cover just having e2e tests in Cylc now as part of #170, and then discuss visual testing later if necessary :+1: