search

e-valuation / EvaP

a university course evaluation system written in Python using Django
Other
99 stars 146 forks source link

Test frontend "looks" #2046

Open niklasmohrin opened 12 months ago

niklasmohrin commented 12 months ago

If we want to pay more attention to how EvaP looks on mobile, we should come up a way to automatically check that our cool CSS keeps working in the future.

One possible approach would be screenshot tests, that is, automatically taking screenshots of rendered pages and comparing them against a previous screenshot (that is in the repository). Every PR that changes the CSS would probably also include an update of the screenshot (which reviewers could compare and see that it still looks okay). The downside is of course that PRs get bloated with updates to the screenshots and that they take up space in the git repo forever, which could lead to slow clones. We should investigate how other projects do this, one other project someone presented me had a separate branch for these screenshots and they were reviewed after the fact, but this doesn't seem optimal to me.

richardebeling commented 12 months ago

@fkoch-tgm mentioned that they have good experience with Meticulous in next.js, but we probably can't use it in django.

A way to prevent the screenshots stored in the repo (which I don't really like) would be to always run the test twice, once for main and once for the PR-base, and then compare (volatile) screenshots. We would need more CI time, but less storage in the repo. Interaction would then be based on PR comments created by some bot that show which pages changed. I don't know if there is already some library/framework for such a django-test-screenshot-github-comment workflow.

niklasmohrin commented 12 months ago

Good idea, it would be cool if the "do the screenshot" thing would be a script that I could also run locally to use with git bisect for example.

Another thought: we could probably cache the screenshot of each commit for efficiency and for being able to inspect if from the GitHub actions panel. If this works like I am currently thinking it does, it seems like a nice workflow to me :D

karyon commented 12 months ago

Regarding the repo size concern, there is git-lfs, which generally just works, except that github apparently doesn't show diffs for images in git lfs. Which is quite a letdown.

hansegucker commented 11 months ago

With the current progress of implementing live tests with Selenium in #1854, we are able to programmatically create screenshots. Together with a tool like Argos CI or Percy, visual comparisons should be possible.