search

keras-team / keras-core

A multi-backend implementation of the Keras API, with support for TensorFlow, JAX, and PyTorch.
Apache License 2.0
1.27k stars 115 forks source link

[GHA] fix several codecov issues #912

Closed kiukchung closed 10 months ago

kiukchung commented 10 months ago

This PR does a few things:

  1. Excludes keras_core/legacy from coverage metrics since this module there for BC reasons
  2. Adds the test backend (jax, tensorflow, torch, numpy) to flags to allow tracking coverage metrics per backend. Note that each backend's coverage will be lower than the aggregate (e.g. keras_core will have higher coverage than keras_core-jax), so this flag should be used to track trends rather than impose thresholds.
  3. Moves coverage and pytest configuration from setup.cfg to pyproject.toml to keep all project configs in toml vs cfg (unfortuantely flake8 configs need to stay in setup.cfg since flake8 doesn't support pyproject.toml yet: https://github.com/PyCQA/flake8/issues/234)
  4. Adds fail_ci_if_error: true to codecov upload steps to fail the CI (and force the author to retry the workflow) on transient codecov upload failures of the form 
     Unable to locate build via Github Actions API. Please upload with the Codecov repository upload token to resolve issue.', code='not_found

    This happens when we hit codecov API rate limits since we don't use CODECOV_TOKEN (see https://github.com/codecov/feedback/issues/126). We could circumvent the issue by adding the token as a secret but GH secrets are not visible from forks, implying that contributors will not be able to get coverage diffs for their PRs. To allow this, the only option seems to be to hardcode the token (which seems ok since the token can be setup with limited permissions) but hardcoding tokens is just sketchy...

kiukchung commented 10 months ago

will graduate from draft commit after I validate GHA and check codecov reports codecov metrics + flags look good

codecov[bot] commented 10 months ago

Codecov Report

Patch coverage has no change and project coverage change: +6.79% :tada:

Comparison is base (a465816) 76.82% compared to head (742e71c) 83.62%. Report is 3 commits behind head on main.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main codecov/codecov-action#912 +/- ## ========================================== + Coverage 76.82% 83.62% +6.79% ========================================== Files 329 318 -11 Lines 31427 28390 -3037 Branches 6112 5409 -703 ========================================== - Hits 24144 23740 -404 + Misses 5719 3150 -2569 + Partials 1564 1500 -64 ``` | [Flag](https://app.codecov.io/gh/keras-team/keras-core/pull/912/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=keras-team) | Coverage Δ | | |---|---|---| | [keras_core](https://app.codecov.io/gh/keras-team/keras-core/pull/912/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=keras-team) | `83.51% <ø> (+6.78%)` | :arrow_up: | | [keras_core-jax](https://app.codecov.io/gh/keras-team/keras-core/pull/912/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=keras-team) | `67.58% <ø> (?)` | | | [keras_core-numpy](https://app.codecov.io/gh/keras-team/keras-core/pull/912/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=keras-team) | `60.73% <ø> (?)` | | | [keras_core-tensorflow](https://app.codecov.io/gh/keras-team/keras-core/pull/912/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=keras-team) | `67.07% <ø> (?)` | | | [keras_core-torch](https://app.codecov.io/gh/keras-team/keras-core/pull/912/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=keras-team) | `69.42% <ø> (?)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=keras-team#carryforward-flags-in-the-pull-request-comment) to find out more. [see 13 files with indirect coverage changes](https://app.codecov.io/gh/keras-team/keras-core/pull/912/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=keras-team)

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.