GitHub Actions Cypress formatter for better output

[Splines]

What would you like?

GitHub Actions are widely adopted. They offer so-called workflow commands (a special log syntax) to show warning/error messages directly in the GitHub UI. This is awesome as you will see the warnings/errors in the summary of your last run and directly in the Files changed tab in pull requests. As an example, consider this image:

The preferred way to specify this formatter would be:

cypress run --format github

Why is this needed?

This feature would be great to ease seeing which Cypress tests failed in a GitHub Actions pipeline. It would be really nice to have the failing tests marked as "failing" directly in the Files changed tab (you may also mark files there that weren't changed in a pull request). But even more advantageous, the Summary tab in your GitHub Action runs will give you a summary of which tests failed, without you having to drill down to the specific GitHub Action task that executed cypress run.

Other

Note that other libraries have a GitHub formatter available:

• Rubocop (a Ruby linter): rubocop --format github
• Pylint (a Python linter): pylint --output-format=github
• rspec-github - a library that adds a GitHub Actions formatter to RSpec tests: rspec --format RSpec::Github::Formatter
• ESLint (a JS linter) uses stylish as default formatter that GitHub Actions are smart enough to understand (even though they don't follow the GitHub Actions workflow syntax). Usage: npx eslint -f stylish (default)

[MikeMcC399]

@Splines

Reading your enhancement request, I was uncertain whether you were familiar with the existing JavaScript Action from Cypress.io. The Cypress solution for GitHub Actions is described in https://docs.cypress.io/guides/continuous-integration/github-actions with repo https://github.com/cypress-io/github-action.

[Splines]

Thanks for pointing me to that documentation. I've already seen the action, however, we wanted to use Docker Buildx with GitHub container registry as seen in our workflow file. The cypress run command is finally issued in the entrypoint here. We do make use of your pre-built Docker images though (FROM cypress/included:13.11.0).

Having seen that your GitHub action even provides nice job summaries, we might rethink our Dockerfile and try to switch to your GitHub action. From the documentation, it's not clear to me whether you generate annotations for failed jobs that are shown on the same level as the job summary and also in the Files changed tab right next to the line that made a Cypress test fail.

[MikeMcC399]

@Splines

cypress-io/github-action does not provide any annotations apart from the statistical summary of job outcome.

Cypress Cloud with Test Replay is the tool of choice for tracking down and resolving individual test failures. See also Using Cypress Cloud with GitHub Actions.