search

davelosert / vitest-coverage-report-action

A GitHub Action to report vitest test coverage results
MIT License
157 stars 20 forks source link

Feature Request: negative trend check #395

Open modosc opened 3 months ago

modosc commented 3 months ago

right now it's possible to enforce coverage via the test.coverage.thresholds values. if these are set and the pr causes coverage to drop below these values, the action fails.

however, this makes it hard to introduce coverage/tests in an existing codebase where there are none.

i'd like to add a check which looks at the same numbers you use for calculating the trends and fails when a pr causes coverage to go down. i'm not sure if having thresholds would be useful here - for me, i'd want any negative trend to fail.

does this sound like something useful? if so i'd be happy to open a pr.

davelosert commented 3 months ago

@modosc : Thank you very much for this feature request.

if these are set and the pr causes coverage to drop below these values, the action fails.

One correction: So far, this action does NOT fail on unmet thresholds, as I believe that should be the job of the test-step itself, that is whatever step executes vitest and also fails when tests are failing and where the thresholds are also validated.

(One note on the side: I am thinking about including the test command into this action as well, but that will still take a while).

Nonetheless, I could imagine a feature like you described to have this action fail on certain coverage drops.

My only question is over what reports this drop (or increase) of coverage is calculated:

  1. Just compared to the one coverage report given through json-summary-compare-path (as in Coverage drop within this single PR), it should be quite straight forward.
  2. If you wanted to compare it against multiple numbers of PRs (like Coverage drop of X Percent over the last Y Pull Requests), it will be harder, as we would have to figure out a way to store the coverage data in a recoverable format.

I am open for 1), but I believe 2) is a tad to complicated and already goes into the direction of full featured products like CodeCov.

I am happy for a PR for this, but please note that I am in the middle of a major refactoring to this action which will take me at least another 2 weeks. So it might make sense to wait until I am done to start in order to not run into any merge conflicts.