CI: Fall back to codecov@v3 when @v4 has no token, add coveralls

[bernhardkaindl]

The merge of #5734 on master (merge commit on master: https://github.com/xapi-project/xen-api/commit/5bd9b86b30c9c02d3b702b45194fd5943e8ca10a) uses OpenID Connect (OIDC) to have a token-based coverage upload to Codecov.

But because OpenID Connect is only available for builds from users that use OIDC:

• The build of the merged commit on master (https://github.com/xapi-project/xen-api/commit/5bd9b86b30c9c02d3b702b45194fd5943e8ca10a) failed to upload because it has no context of the user pushing the merged PR: OIDC is not available during that build and the upload fails:

  Run codecov/codecov-action@v4 evenName: push Error: Codecov: Failed to get OIDC token with url: https://codecov.io./ Error message: Unable to get ACTIONS_ID_TOKEN_REQUEST_URL env variable

Docs on the Codecov action and using OIDC: https://github.com/marketplace/actions/codecov#using-oidc

The most reliable way to upload to codecov.io is codecov@v4 with the CODECOV_TOKEN secret.

• It needs secrets.CODECOV_TOKEN, generated in codecov.io by the repo or organisation owner. These are the possible ways to set it up:
  • Only for one repo / for each repo individually:
  • Generated by the owner xen-api repo owner (@robhoes) and added as repo secret to xen-api, or
  • For all repos in the organisation:
  • Generated by the org owner of xapi-project and added as organisation secret to the xapi-project organisation.
  • Precondition: The respective owner needs to have confidence (trust) that Codecov will not hurt the project.
  • When generating the token, github will show a very scary message saying that Codecov can impersonate the user who generates the secret token (using the given account permission).
  • The secret token is guarded by GitHub action of course.
  • As long as the repo or organiation owner use a secure environment to copy and paste the generated CODECOV_TOKEN from codecov into the repo or respective organisation secret on GitHub.
    • this is of course key: It must not be stored by any other means besides as repo/organisation secret for the GitHub action workflows.
    • The other trusted party is codecov.io, which must be trusted to not abuse the trust.

Without that hurdle jumped, we only have the option of fallbacks:

• Fall back to OIDC when the OIDC is available from the user
  • It will be used automatically on PR workflows, but not always on push workflows.
• If that fails too, we can only fall back to v3, with the known unreliable uploads due to GitHub's API rate-limiting.
  • This fallback is the last resort for the coverage uploads on branches/pushes OIDC is not accessible.
• Upload to coveralls.io, which was already used by the xen-api repository in the past.
  • Can always be used, no issue there,
  • Does not need a login to view coverage.
  • Very easy to administer, no complicated setup like for codecov.io.
  • The repo is already at https://coveralls.io/github/xapi-project/xen-api
  • Already works, this PR is uploaded at: https://coveralls.io/jobs/145432327

This PR implements all these fallbacks. These are useful for the xapi-project/xen-api and for working on and with forks.

If xapi-project/xen-api adopts the setup of the CODECOV_TOKEN, also forks of it would use it. GitHub is trusted to protect the tokens like it it is trusted with all other GitHub action secrets.

[psafont]

I'd rather revert the change that introduced v4, or use coveralls rather than adding this much complexity. Is coveralls more stable than codecov?

[lindig]

I agree with Pau; complexity in these nice-to-have features will cause breakage down the line that will be expensive to fix.

[psafont]

coveralls works well to upload coverage for both python versions: https://coveralls.io/builds/68333955