Option to generate a report without failing the tests

[hugohaggmark]

We're using pa11y-ci as part of our CI. We use different threshold approaches depending on which branch the CI is building.

• When a user opens a PR, we use pa11y-ci with thresholds to prevent new PRs from adding new errors. Works great!! Thank you!
• When we merge a PR to our main branch, we use pa11y-ci without thresholds because we want to generate a report and push those statistics so we can see the trends of errors over time. This works ok, the report is generated perfectly but pa11y-ci fails, well because it found errors. So in this case, we need to ignore errors from that specific CI step, which is to be honest, not great.

Is there a way to accomplish the latter scenario today in pa11y-ci?

If not, would you consider introducing a flag like dontFailOnErrors to make this flow possible?

Thank you 🙏

[greggles]

It seems like this is kind of a duplicate of #54.

I'm most familiar with Jenkins and can think of 2 solutions to this problem in Jenkins that might work in your context. You can use command || true so that when the command fails the exit code is hidden from the CI system. You can also use:

set +e
command-you-know-will-fail
set -e

This lets Jenkins keep running.

[danyalaytekin]

Hi both, we'll track this in #54 as @greggles has suggested. I imagine the suggested flag dontFailOnErrors will be inverted but this is good food for thought. Thanks!

[danyalaytekin]

Duplicate of #54