For a gem like this, it's really useful to be able to actually test the rendered output of Rails. @donv did some awesome work to develop processes that generate screenshots directly from the README.md, and compare them to expected screenshots in the repo.
Unfortunately, the expected screenshots highly depend on the platform (e.g. browser and fonts) on which they're generated. As an open-source project, we don't want to, and can't, dictate the platform used by contributors. (I use Ubuntu. Most devs these days have a Mac. Some devs, and most users of web sites, use Windows. And the real users of our apps are using their phones.)
The suggested solution is to generate the expected results via a GitHub action, so anyone can generate a PR that updates the expected screenshots in a consistent way, regardless of their O/S and browser of choice. The goal of this issue is to create or update GitHub actions triggered on creating or updating a PR, that will run the system tests, and if the screenshots change, add the changed screenshot to the PR.
It will then be the responsibility of the reviewers of the PR to ensure that the changes are acceptable. Changes to screenshots would be acceptable if the PR is actually fixing a bug, where the previous screenshot was actually unacceptable. Changes to screenshots should usually be taken as breaking changes, and we should question ourselves before accepting them. However, in the short term, we can accept more changes as we learn about the testing approach and discover that we need to allow changes to the expected screenshots.
Out of scope for this issue is the question of cross-browser testing. In theory, that's one of the things that Bootstrap should be handling for us. And if there are issues, they're probably at the level of Bootstrap, not our gem. But it's something to think about.
For a gem like this, it's really useful to be able to actually test the rendered output of Rails. @donv did some awesome work to develop processes that generate screenshots directly from the
README.md, and compare them to expected screenshots in the repo.Unfortunately, the expected screenshots highly depend on the platform (e.g. browser and fonts) on which they're generated. As an open-source project, we don't want to, and can't, dictate the platform used by contributors. (I use Ubuntu. Most devs these days have a Mac. Some devs, and most users of web sites, use Windows. And the real users of our apps are using their phones.)
The suggested solution is to generate the expected results via a GitHub action, so anyone can generate a PR that updates the expected screenshots in a consistent way, regardless of their O/S and browser of choice. The goal of this issue is to create or update GitHub actions triggered on creating or updating a PR, that will run the system tests, and if the screenshots change, add the changed screenshot to the PR.
It will then be the responsibility of the reviewers of the PR to ensure that the changes are acceptable. Changes to screenshots would be acceptable if the PR is actually fixing a bug, where the previous screenshot was actually unacceptable. Changes to screenshots should usually be taken as breaking changes, and we should question ourselves before accepting them. However, in the short term, we can accept more changes as we learn about the testing approach and discover that we need to allow changes to the expected screenshots.
Out of scope for this issue is the question of cross-browser testing. In theory, that's one of the things that Bootstrap should be handling for us. And if there are issues, they're probably at the level of Bootstrap, not our gem. But it's something to think about.