Open HonkingGoose opened 2 years ago
RahulGautamSingh
commented
2 years ago Hey @HonkingGoose, Can I work on implementing automated tests using Cypress on this one? I am fairly new to front-end testing and this seems to be a good task for me to learn more.
viceice
commented
2 years ago Sure, go ahead.
RahulGautamSingh
commented
2 years ago I have added cypress's basic config in a PR and will add new tests for most basic functionality like:
@HonkingGoose Do we need to make any updates to docs regarding the cypress testing?
I think we can add this:
To write more tests using cypress, follow these steps:
HonkingGoose
commented
2 years ago I have added cypress's basic config in a PR
Cool!
I think we should install Testing Libary as a development dependency as well. It makes writing tests easier. But I don't know how to set up Testing Library so it works with Cypress. 😞
@Belco90 are you willing to help us with Testing Library? If not that's fine as well. 😉
Belco90
commented
2 years ago @Belco90 are you willing to help us with Testing Library? If not that's fine as well. 😉
Sure! It should be a matter of installing Cypress Testing Library and then extending cy commands by importing the corresponding module. You can find its setup instructions here: https://testing-library.com/docs/cypress-testing-library/intro
Feel free to reach out if you struggle to set it up!
HonkingGoose
commented
2 years ago Someone suggested adding Cypress tests to the upstream mkdocs-material project. The maintainer of mkdocs-material says: ^quote
Thanks for suggesting! I've thought on and off about adding end-to-end tests to this project, but my conclusion is that the project is so amazingly complex that E2E tests would barely help to secure the quality of this project. The reason is that we have dozens of configuration options that significantly alter the HTML and/or behavior of the resulting documentation page, which we'd need to test in all combinations. For instance, we have support for 58 languages – we would need to define a test case for each of those languages to make sure that at least search works. This is impractical from a maintenance and runtime perspective.
Furthermore, implementations between the community edition and Insiders sometimes differ, which means even more effort would be needed. As much as I want to add tests, I've decided at some point that it's currently not feasible with the time I have to work on this project. Also, when we implement tests, they would need to be maintained for which I don't have the time to do – it's already hard for me to keep up with bug fixing, feature development and community management.
Closing as (currently) not planned.
It sounds to me like adding, or even using, Cypress tests is the wrong way to go for us. Even the maintainer of the project doesn't see how it can work for them. 😄
I'll let @rarkins decide what's best, based on this information.
HonkingGoose
commented
4 months ago We now also benefit from tests built into mkdocs:
So I'm thinking of closing this issue as done enough.
What would you like to be able to do?
When I made PR #97, I forgot to check the display of our config presets page. This means we deployed a change that was not ready for production. 😞
It easy to miss bad changes like this when testing manually. It also means that updates to Material for MkDocs are risky.
I would like to have some automated tests that check for visual changes, so that unintended visual regressions are caught by the machine.
I would also like to have some basic tests that check if things like search, sidebar items, previous page/next page button, light/dark mode all work properly.
Did you already have any implementation ideas?
We could probably use Cypress ^cypress to test the functionality of our docs pages.
I'm not really familiar with any way to test for visual changes yet.