Feedback: communication around GitHub Action removing support for token-less uploads

[JoshuaKGoldberg]

Coming over from https://github.com/codecov/feedback/issues/112 -> https://github.com/codecov/feedback/issues/112#issuecomment-1964319140, https://github.com/codecov/codecov-action/issues/1293, and https://github.com/codecov/codecov-action/issues/1348: I'd like to leave some friendly feedback that the removal of token-less uploads in the GitHub Action was surprising and mildly disruptive. For context, I manage a few dozen open source repositories on a shared template that sets up Codecov. Each of their code coverage tracking was silently broken by the v3->v4 upgrade because v4 passes builds despite erroring uploading coverage (https://github.com/codecov/codecov-action/issues/1348).

Echoing https://github.com/codecov/feedback/issues/112#issuecomment-1964319140: in general, if you have a need to "break" users the way Codecov needed, there are several proactive steps you can take ahead of time to minimize customer pain or even avoid the need for breakage:

• Letting the community know our token-less uploading was causing service issues
• Adding the token as a strongly recommended -but not required- option first
• Adding Codecov rate limiting so particularly active repositories don't hog the requests
• Changing the action to fail the build if it fails on token-less upload (instead of just warn and then pass)

Was this change communicated out to people ahead of time? The only reason I learned of this is that my repositories were no longer updating Codecov numbers (despite passing codecov-action builds).

It's of course understandable if the Codecov team doesn't have the bandwidth to work on those changes first. But letting us know early on could have saved some community pain, I think.

Codecov is a fantastic service and I sincerely appreciate the services and generous free tier you provide to open source maintainers such as myself. I'm hopeful that in the future, we can keep using & promoting your product without disruptions like this one. I'd be happy to provide more details and/or chat directly if you'd find that helpful. ❤️

[rohan-at-sentry]

Hi @JoshuaKGoldberg

Thanks for leaving this feedback here! I wanted to make sure I addressed them sufficiently and transparently.

Why does v4 need tokens v4 of the Codecov Action moves away from wrapping the Codecov Uploader to the new Codecov CLI . We built out a requirement for tokens primarily to ensure the service was reliably available. Previously we'd have instances (almost daily) where repos would not receive coverage reports from Codecov as we'd reach the GitHub API rate limit (we'd not be able to query Github which would power the ability to post comments). We'd silently fail in this scenario as well, with the philosophy of not blocking a build.

Prior Communication I think we have an opportunity (as Codecov) to do better here. We did release version v4-beta in September 2023 to iron out issues ahead of release because we knew it was going to be challenging.

We also released a blog post when we were ready to GA v4 (see here https://about.codecov.io/blog/january-product-update-updating-the-codecov-ci-uploaders-to-the-codecov-cli/).

In retrospect, I feel like it wasn't enough

• There was a long enough time between September and Jan that warranted maybe a broader blast and gentler rollout and
• There was likely a need to have something like an FAQ to help make that transition smoother

While this doesn't make the past experience magically better, here's what we are doing to make sure changes like this are better managed.

• Clearly identifying upfront if a change breaks the community and building a comms plan around it ahead of release
• Trying to be more responsive to issues raised (we've tried to address as and when they've come up, but we're continuing to improve how quickly we respond, provide insight / timelines on fixes etc)