JuliaRegistries / CompatHelper.jl

Automatically update the [compat] entries for your Julia package's dependencies
https://JuliaRegistries.github.io/CompatHelper.jl/dev/
MIT License
138 stars 40 forks source link

add codecov threshold #480

Closed ericphanson closed 11 months ago

ericphanson commented 11 months ago

to help with https://github.com/JuliaRegistries/CompatHelper.jl/pull/477

docs: https://docs.codecov.com/docs/common-recipe-list#set-status-checks-to-block-coverage

codecov[bot] commented 11 months ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (9b7249f) 99.24% compared to head (85b2189) 99.24%. Report is 2 commits behind head on dpa/merge-queue.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## dpa/merge-queue #480 +/- ## ================================================ Coverage 99.24% 99.24% ================================================ Files 12 12 Lines 395 395 ================================================ Hits 392 392 Misses 3 3 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

ericphanson commented 11 months ago

I wonder if this configuration would work for that?

DilumAluthge commented 11 months ago

Hmmm, so in the current state of this PR, it would allow the merge_group job to have code coverage strictly less than 100%, right?

ericphanson commented 11 months ago

I don't really understand if this is about the branch we are merging from (/ running on), or the branch we are merging into. Either way I think this is incomplete though. If it is the branch we are running on, then we want the merge-queue branch to have the strict checks, not just main itself. If it is the branch we are merging into, then this won't help for PRs to main.

ericphanson commented 11 months ago

I think maybe it does support regex actually. The docs are very incomplete but I found some old ones that suggested it does (and some older ones that suggested it didn't then)

ericphanson commented 11 months ago

Ok, I had to fix the syntax a bit, but this now validates:

curl -X POST --data-binary @codecov.yml https://codecov.io/validate
Valid!

{
  "coverage": {
    "status": {
      "project": {
        "main": {
          "branches": [
            "^main$",
            "^gh-readonly-queue/main/.*"
          ],
          "target": 100.0
        },
        "default": {
          "target": 98.0
        }
      }
    }
  }
}
DilumAluthge commented 11 months ago

How would we go about testing this to make sure it works the way we want?

ericphanson commented 11 months ago

I think the easiest way is just by merging things. We can test the happy path by merging this and your PR down. Then we can test the failure path by deleting the integration tests and seeing if that merges down. If so we can revert back.