Migrate CI from CircleCI to GitHub Actions

galargh commented 6 months ago

User Story

GitHub Actions provides Developer Experience better suited for Lotus development team going forward.

Acceptance Criteria

[x] The p50 and p95 success rate of the new GitHub Actions workflows is the same or higher than their CircleCI counterparts
[x] The p50 and p95 execution time of the new GitHub Actions workflows is the same or smaller than their CircleCI counterparts (*p90 is worse for the Test workflow but it is due to an outlier caused by an issue with self-hosted runners that is being addressed)

### Deliverables (GitHub Actions workflows)
- [x] docs-check
- [x] gen-check
- [x] gofmt
- [x] lint-all
- [x] mod-tidy-check
- [x] test
- [x] test-conformance
- [x] test-itest-*
- [x] test-unit-*
- [x] docker-build
- [x] docker-push
- [x] release

Technical Breakdown

- [x] Create GitHub Actions jobs performing checks and generation (docs-check, gen-check, gofmt, lint-all, mod-tidy-check)
- [x] Provision self-hosted runners (large) for use in lotus (to accommodate expected high parallelism demands)
- [x] Create GitHub Actions jobs executing tests (test, test-conformance)
- [x] Create GitHub Actions jobs executing tests that require high parallelism (test-itest-*, test-unit-*)
- [x] Create GitHub Actions jobs executing docker builds (docker build, docker push)
- [x] Create GitHub Actions jobs performing release (release (dry), release)
- [x] Gather and compare CI stats for CircleCI job runs and their GitHub Actions counterparts*
- [ ] Decommission CircleCI jobs after 1-2 weeks of stable execution of GitHub Actions counterparts*
- [ ] Update lotus-priv clone

### Testing
- [x] Run the new GitHub Actions workflows in shadow mode for 1-2 weeks
- [x] Gather success rate and execution time metrics for both the CircleCI and GitHub Actions workflows
- [x] Compare and prepare a report

Stebalien commented 5 months ago

Any idea what caused the apt locking issue in https://github.com/filecoin-project/lotus/actions/runs/8529890853/job/23366624530?pr=11808?

galargh commented 5 months ago

Any idea what caused the apt locking issue in https://github.com/filecoin-project/lotus/actions/runs/8529890853/job/23366624530?pr=11808?

I have seen that before caused by ubuntu auto-updates but we disabled them since then. I propose we add extra logging to see what process is holding up the lock in case it happens in lotus again - https://github.com/filecoin-project/lotus/pull/11815 Maybe bet would be that there is another scheduled process that is causing this.

Stebalien commented 5 months ago

Any idea why test runs like https://github.com/filecoin-project/lotus/commit/a40cced0d92a7c30afbd74c300695342e0102796 are showing up as failed? I assume they're just getting cancelled because we're merging something else to master, unless something else is cancelling them.

galargh commented 5 months ago

Any idea why test runs like a40cced are showing up as failed? I assume they're just getting cancelled because we're merging something else to master, unless something else is cancelling them.

That's exactly it. We can disable this with - https://github.com/filecoin-project/lotus/pull/11842. Cancelling in progress runs is quite useful in the PRs, but we can do without it in other cases.

filecoin-project / lotus