search

cylc / cylc-ui

Web app for monitoring and controlling Cylc workflows
https://cylc.github.io
GNU General Public License v3.0
36 stars 27 forks source link

Visual testing #178

Open kinow opened 5 years ago

kinow commented 5 years ago

@hjoliver pointed we had a pending item from a previous meeting, about e2e tests. We have e2e set up already in the project, but no useful test for now. That's mainly due to the fact that what we had before was more of a prototype.

Soon we will have the final screens for the application UI, and then we should be ready to write good e2e functional tests.

However, during the same meeting, @oliver-sanders mentioned an issue with the progress display in the Task component. The issue we had was related to a CSS expression that wasn't working on Chrome. In theory, having a e2e test would cover it, but by default would not detect if that was broken.

Cypress (similarly to Selenium and Nightwatch) provide only functional automated web tests. So only tests that confirm that the web application has the features from its requirements, and that they work as expected (things like the colors are right, the page loads fast enough, the gauge arrow is displayed, are normally considered "non-functional requirements").

Cypress documentation mentions "Visual Testing", see this part of their docs

Luckily, Cypress gives a stable platform for writing plugins that can perform visual testing. ... Typically such plugins take an image snapshot of the entire application under test or a specific element, and then compare the image to a previously approved baseline image. https://docs.cypress.io/guides/tooling/visual-testing.html#Functional-vs-visual-testing

@oliver-sanders are these the kind of tests you would like for Cylc UI?

I don't think we are quite ready to introduce these tests at the moment. IMO, we better wait a bit longer, until we are certain the components and views are ready, and then choose whether to use this testing technique or not, and if we decide to use, then choose what to test.

But in anyway, I'd prefer to cover just having e2e tests in Cylc now as part of #170, and then discuss visual testing later if necessary :+1:

oliver-sanders commented 5 years ago

Typically such plugins take an image snapshot of the entire application under test or a specific element, and then compare the image to a previously approved baseline image.

@oliver-sanders are these the kind of tests you would like for Cylc UI?

Absolutely.

A visual diff tool should subtract one image from another leaving you with an image of the things which have moved / changed e.g:

http://diffee.me/?http://una.im?http://una.im/diffeedemo

We can gather screenshots using something Cypress, Selenium or whatever (can be done on Travis) and assemble "reports" using open source visual comparison libraries.

The big issue is storing/displaying those reports (can't be done on Travis) and GH integration. There are systems out there but they cost a fortune.

kinow commented 5 years ago

The big issue is storing/displaying those reports (can't be done on Travis) and GH integration. There are systems out there but they cost a fortune.

https://www.cypress.io/pricing

Cypress has a service with a free tier. 3 users and 500 recordings.

matthewrmshin commented 5 years ago

The open source plan https://www.cypress.io/oss-plan/ allows 5 users and unlimited recordings.

oliver-sanders commented 2 years ago

https://docs.cypress.io/guides/tooling/visual-testing#What-you-ll-learn