Lately we've been experiencing some pushes to codecov failing, particularly if the PR comes from branches in forks. This failures are random and not related to a particular reason. We might need to research more on this. In the meantime we can make some changes to improve the contributor experience.
~Currently every runner pushes to codecov. This might generate a lot of hits to codecov that could limit the number of pushes. Since codecov only tracks test coverage, we don't need one push per runner: only having one runner using the latest (or "optional") dependencies should be enough.~ Actually, in some scenarios max coverage is achieved by combining coverage from different runners.
Having the codecov push inside the test jobs creates some confusions: if the codecov push, the test job is marked as failed, even if the test passed. We could move the push to codecov to its own job to improve this.
Having the coverage report as an artifact we can download could be helpful. We can use that report with local tools (like https://github.com/andythigpen/nvim-coverage) without the need to rerun all tests. Also, we can use this report to check which lines aren't being covered in CI.
In summary, I think we should apply these steps to our test workflows:
Split the push to codecov to its own job. It should require the tests to pass.
The new job should only make one single push to codecov. ~Maybe using the report generated by ubuntu-latest using the latest dependencies. If optional dependencies are also defined, it would be better to use those in case we have tests that rely on optional dependencies.~ Download all artifacts containing coverage reports and upload them all at the same time to Codecov (on a single push).
Store the coverage report as an artifact ~on ubuntu-latest (with latest/ optional dependencies)~ for each runner, so it can be reused by the codecov job and also downloadable.
Update 2024-05-14
After playing with the initial ideas in fatiando/harmonica#502, I've updated the text of the tasks that should be carried out. In summary: make all runners update the coverage.xml file as an artifact, and make the new codecov-upload job to download them all and push them all together to codecov in a single run.
Start by opening Pull Requests on each repository listed above.
Optionally, we can open Issues on each repository if further discussion specific to that repository is needed.
Mention this Issue on every Issue or Pull Request opened on each opened: Related to fatiando/community#151
Check-off the repository on the list above once the Pull Request is merged.
Close this issue when all items are checked-off.
We want your help!
We know that maintenance tasks are very demanding, so we don't expect a single person to tackle this issue by themselves. Any help is very welcomed, so please comment below that you want to take care of the changes on any repository and we will assign it to you.
Description:
Lately we've been experiencing some pushes to codecov failing, particularly if the PR comes from branches in forks. This failures are random and not related to a particular reason. We might need to research more on this. In the meantime we can make some changes to improve the contributor experience.
In summary, I think we should apply these steps to our test workflows:
ubuntu-latestusing thelatestdependencies. Ifoptionaldependencies are also defined, it would be better to use those in case we have tests that rely on optional dependencies.~ Download all artifacts containing coverage reports and upload them all at the same time to Codecov (on a single push).ubuntu-latest(withlatest/optionaldependencies)~ for each runner, so it can be reused by the codecov job and also downloadable.Update 2024-05-14
After playing with the initial ideas in fatiando/harmonica#502, I've updated the text of the tasks that should be carried out. In summary: make all runners update the
coverage.xmlfile as an artifact, and make the newcodecov-uploadjob to download them all and push them all together to codecov in a single run.Apply to:
Further instructions:
Related to fatiando/community#151We want your help!
We know that maintenance tasks are very demanding, so we don't expect a single person to tackle this issue by themselves. Any help is very welcomed, so please comment below that you want to take care of the changes on any repository and we will assign it to you.