[WIP] Lotus Release v1.28.0

[rjan90]

🚢 Estimated shipping date

• RC1:

  • ~2024-06-27 (last confirmed 2024-06-24T18:44:46Z)~
  • 2024-07-01 (updated on 2024-06-28T03:58:09Z - slack thread)
    • Published https://github.com/filecoin-project/lotus/releases/tag/v1.28.0-rc1

• RC2:

  • 2024-07-04: https://github.com/filecoin-project/lotus/releases/tag/v1.28.0-rc2

• RC3:

  • 2024-07-04: https://github.com/filecoin-project/lotus/releases/tag/v1.28.0-rc3

• RC4:

  • 2024-07-05: https://github.com/filecoin-project/lotus/releases/tag/v1.28.0-rc4

• RC5:

  • 2024-07-11: https://github.com/filecoin-project/lotus/releases/tag/v1.28.0-rc5

• v1.28.0 stable estimated: ~2024-07-23~ 2024-07-24

Dependencies for releases

This is the set of changes that need to make it a given RC. This is effectively the set of changes to cherry-pick from master.

RC1

• [x] https://github.com/filecoin-project/lotus/pull/12147
• [x] https://github.com/filecoin-project/lotus/pull/12158
• F3 items
  • [x] https://github.com/filecoin-project/lotus/pull/12119
  • [x] https://github.com/filecoin-project/go-f3/issues/373
• [x] Cherry pick https://github.com/filecoin-project/lotus/pull/12154 - backported here

RC2 (current "calibration final-candidate")

• [x] https://github.com/filecoin-project/lotus/issues/12165 (CONFIRMED TO WORK ON BUTTERFLY)
• [x] https://github.com/filecoin-project/lotus/issues/12160
• [x] https://github.com/filecoin-project/lotus/pull/12135
• [x] Set upgrade epoch on Calibration
• [x] Update upgrade warning section to warn about: This upgrade includes an additional migration to the events database. Node operators running Lotus with events turned on (off by default) may experience some delay in initial start-up as a minor database migration takes place. See [filecoin-project/lotus#12080](https://github.com/filecoin-project/lotus/pull/12080) for full details.
• [x] Backport: https://github.com/filecoin-project/lotus/pull/12177
• [x] Backport: https://github.com/filecoin-project/lotus/pull/12175
• [x] Backport: https://github.com/filecoin-project/lotus/pull/12176
• [x] Backport: https://github.com/filecoin-project/lotus/pull/12181

Mainnet Upgrade Release

• [x] Backport everything with the backport label (and then remove the backport labe)
• [x] ~ https://github.com/libp2p/go-libp2p/issues/2858 ~ (We have bumped go-libp2p. Will close issue once we have a bit more confidence after people have upgraded)
• [x] https://github.com/filecoin-project/go-f3/issues/386
• [x] Dependency updates
  • [x] https://github.com/filecoin-project/builtin-actors/issues/1569
  • [x] https://github.com/filecoin-project/filecoin-ffi/issues/471
  • [x] https://github.com/filecoin-project/go-state-types/issues/292

✅ Release Checklist

• [x] Wait for the v1.27.1 release to complete: https://github.com/filecoin-project/lotus/issues/12073
• [x] Fork a new branch (release/v1.28.0) from master.
  • Any further release related changes need to be made in this branch or cherry-picked to it.
  • If any "non-trivial" changes get added to the release, uncheck all the checkboxes and return to this stage.
• [x] (After the RC gets cut) Bump the version in build/version.go in the master branch to vX.Y.(Z+1)-dev (bump from feature release) or vX.(Y+1).0-dev (bump from mandatory release). Run make gen and make docsgen-cli before committing changes https://github.com/filecoin-project/lotus/pull/12074

Prepping an RC

RC prep steps for network upgrade for Calibration

• [x] Update Lotus dependencies following https://github.com/filecoin-project/lotus/pull/12107
• [x] Set the upgrade epoch for the Calibration network

RC1 Steps

• [x] version string in build/version.go needs to be updated to end with '-rcX' (in the release/vX.Y.Z branch)
• [x] run make gen && make docsgen-cli
• [x] Changelog prep
  • [x] Generate changelog using the script at scripts/mkreleaselog
  • [x] Add contents of generated text to lotus/CHANGELOG.md
  • [x] Editorializing (FIPs, actor bundles, new feature callouts - example)
• [x] Commit using PR: https://github.com/filecoin-project/lotus/pull/12169
• [x] tag commit with vX.Y.Z-rcN
• [x] cut a pre-release here

We've cut the first RC using these steps at https://github.com/filecoin-project/lotus/releases/tag/v1.28.0-rc1.

RC2 Steps

• [x] version string in build/version.go needs to be updated to end with '-rcX' (in the release/vX.Y.Z branch)
• [x] run make gen && make docsgen-cli
• [x] Changelog prep
  • [x] Generate changelog using the script at scripts/mkreleaselog
  • [x] Add contents of generated text to lotus/CHANGELOG.md
  • [x] Editorializing (FIPs, actor bundles, new feature callouts - example)
• [x] Commit using PR
• [x] tag commit with vX.Y.Z-rcN
• [x] cut a pre-release here

Stable Release

• [x] Final preparation
  • [x] Verify that version string in version.go has been updated.
  • [x] Verify that codegen is up to date (make gen && make docsgen-cli)
  • [x] Ensure that CHANGELOG.md is up to date
  • [x] Merge release-vX.Y.Z into the releases branch.
  • [x] Tag this merge commit (on the releases branch) with vX.Y.Z
  • [x] Cut the release here.

Post-Release

• [x] Merge the releases branch back into master, ignoring the changes to version.go (keep the -dev version from master). Do NOT delete the releases branch when doing so!
  • [x] Double check that everything in releases and release/v1.28.0 are in master.
• [x] Update RELEASE_ISSUE_TEMPLATE.md with any improvements determined from this latest release iteration.
• [ ] Create an issue using RELEASE_ISSUE_TEMPLATE.md for the next release.

[rjan90]

@aarshkshah1992 I have added some steps for pulling in all the needed dependencies for a network upgrade Lotus-version here: https://github.com/filecoin-project/lotus/pull/12107

A couple of notes:

• The Lotus v1.28.0-rc1 should be based on the last stable Lotus version out.
• The upgrade epoch for the Calibration network has been alligned on here
  • You need to update this in the build/params_calibnet.go file

[BigLep]

(expanding comment) Tracker of things that were done that weren't in the release issue template or other observations and lessons learned:

• Step for updating Lotus dependencies per https://github.com/filecoin-project/lotus/pull/12107
• Step for setting the upgrade epoch for the Calibration network
• Adjusted the template to make clear that we cut release branch from master.
• Added section to issue template to make clear the issues/PRs we need to wait on for a given PR
• Broke out the changelog steps more extensively and gave an example.
• Made clear that we update master's build/version.go once RC1 is released

[BigLep]

It's currently TBD whether this release will be cut from master or from releases (tracking 1.27.1).

We will have to cherry pick some changes from master regardless, so it would be logistically simpler to cut the branch from master. @aarshkshah1992 is going to do due-diligence on 2024-06-25 looking at https://github.com/filecoin-project/lotus/compare/master...release/v1.27.1 to see if there's anything like events db migrations or anything else calling out for a mandatory release version.

We’ll then update the text in issue description based on the decision.

Note that until the release/v1.28.0 branch is cut, we should only merge changes relevant for nv23.

(related #fil-lotus-dev thread)

[BigLep]

For anyone watching, we have confirmed we are still targeting RC1 for Thursday, 2024-06-27.

[jennijuju]

fwiw i did a quick pass on https://github.com/filecoin-project/lotus/compare/master...jen/v1271prep and i think we can cut off base on master commit 71418479278ab854bdc59bfafb93f6abad4a2cdf + other nv23 must have PRs

[BigLep]

@aarshkshah1992 @jennijuju : I think we should use this github issue to track the code specifics for Lotus v1.28.0.

As a result, I added tracking to the issue for things we're waiting on for RC1 and RC2.

[BigLep]

RC1 is now scheduled for 2024-07-01. Here is the relevant slack thread: https://filecoinproject.slack.com/archives/CP50PPW2X/p171952803942240 . (Issue description was updated too.)

[masih]

Re https://github.com/filecoin-project/lotus/pull/12119 review comments addressed.

[BigLep]

For anyone watching this issue about the 1.28.0 final release, it will not be happening today (2024-07-23). It's expected to come 2024-07-24. The PR for the release is https://github.com/filecoin-project/lotus/pull/12271, which lists the remaining work item (0.0.5 release of go-f3).

[rjan90]

• [ ] Merge release-vX.Y.Z into the releases branch.
• [ ] Tag this merge commit (on the releases branch) with vX.Y.Z

I was unable to resolve conflicts, and failing test when trying to merge the release/v1.28.0 branch into the releases branch (Attempt 2 from Aarsh: https://github.com/filecoin-project/lotus/pull/12291) in such as way that we would be able to tag the v1.28.0 release from the releases branch. This is partly because some of the automation code has landed in the releases branch, when we tested shipping the automatic releases with the v1.27.2 release.

I will go ahead and tag the v1.28.0 release, from the release/v1.280 branch, and in the interim get the releases branch to point to the v1.28.0 tag, so that people which depend on git checkout releases as part of their script/upgrade process, will get the correct version.

[BigLep]

For visibility, there have been active verbal discussions about how to handle the fact that the 1.28.0 tag is from the release/v1.28.0 branch rather than releases. I'll post the notes within an hour.

[BigLep]

For context, my understanding is that the historical release process has been to tag from releases and to cut patch releases (e.g., 1.28.1) from the releases branch. I don't know all the reasons why the releases branch was created, but I assume it is being used by some users or automation as the defacto branch to pull the latest non-RC released code to build from source.

@rjan90 understandably ran into some issues when attempting to merge the release/v1.28.0 into the releases branch. As @rjan90 shared, this is in part due to some release automation that we wanted to try out with a real release (1.27.2 in this case).

I think we have two paths forward:

1. Maintain the previous commitment of releases having the latest final release. To do this, given there are commits in releases from 1.27.2 that we don't want as part of 1.28.x (i.e., release automation changeups), I think the cleanest thing is to revert releases to the pre-1.27.2 state (i.e., make the head of releases be 1.27.1). Looking at https://github.com/filecoin-project/lotus/commits/releases, this would mean reverting the commits to get us back to the 2024-06-24 (commit 36d9634). We then merge in releases/v1.28.0 into releases.
2. Introduce the deprecation/abandonment of the releases today. This is effectively fast-tracking something that maintainers were planning to do post-nv23 anyways as part of https://github.com/filecoin-project/lotus/issues/12072 . This would mean 1.28.x patches (e.g., 1.28.1) would be made off the current release/v1.28.0 branch. It also means anyone relying on releases to follow final releases would need to update to instead track the release/v1.28.0 branch (and its decedents).

Given we're a couple of weeks away from network upgrade, I don't think we should introduce more process changes, and I would vote to go with option 1 to maintain the status quo.

Regardless of what path is taken, there is still the exercise of confirming that there are no changes in the releases branch that should make their way into release/v1.28.0. @aarshkshah1992, @jennijuju, and I took an hour or so to comb through the differences. This was done in a couple of ways:

1. Manually looking at the commits in https://github.com/filecoin-project/lotus/commits/releases and https://github.com/filecoin-project/lotus/commits/release/v1.28.0
2. Diffing the commit title differences the two branches with a command like colordiff <(git log --pretty=format:"%s" release/v1.28.0..releases | sort) <(git log --pretty=format:"%s" releases..release/v1.28.0 | sort)

We couldn't find any instance of a commit that was in releases that should have been in release/v1.28.0. The cases where there were commits that were in releases that weren't in release/v1.28.0 boiled down to:

1. Release automation items (discussed above)
2. Dependency updates that were eclipsed by even higher dependency upgrades

(And yes, I'm sure there is a better "git fu" way of answering the questions of "are there any meaningful commits in branch X that aren't in branch Y - feedback/pointers welcome.)

---

Diff comparing releases commits to releave/v1.28.0 commits using the `colordiff` command above

```bash 1,23c1,6 < Add Lotus-Miner / Curio related changes < Bump NodeBuildVersion to v1.27.1-rc1 < Bump pubsub-dep < Bump version - make gen/make docsgen < Make gen / make docsgen-cli < Merge branch 'release/v1.27.1' into jen/v1271prep < Merge branch 'releases' into resolve-releasev1270-conflicts < Merge pull request #11972 from filecoin-project/prep-v1270-rc2 < Merge pull request #12003 from filecoin-project/backports-rc3 < Merge pull request #12039 from filecoin-project/pre-final-v1270 < Merge pull request #12058 from filecoin-project/resolve-releasev1270-conflicts < Merge pull request #12077 from filecoin-project/prep-v1271-rc1 < Merge pull request #12115 from filecoin-project/jen/changlog < Merge pull request #12117 from filecoin-project/phi/v1271-rc2 < Merge pull request #12125 from filecoin-project/jen/v1273 < Merge pull request #12137 from filecoin-project/jen/v1271final < Merge pull request #12138 from filecoin-project/jen/v1271prep < Merge pull request #12140 from filecoin-project/rvagg/changelog-note-events-db < Merge pull request #12250 from filecoin-project/release/v1.27.2 < Prep v1.27.0-rc2 < Typo fixes, and more changelog updates < Update CHANGELOG.md < Update CHANGELOG.md --- > Apply suggestions from code review > Apply suggestions from code review > Merge pull request #12147 from filecoin-project/chore/nv23-v1.28.0-backport > Merge pull request #12149 from filecoin-project/jen/butterflyinit > Merge pull request #12169 from filecoin-project/aarshkshah1992/update-lotus-version-v128 > Rough draft (#12139) 25,37c8,17 < Update changelog < Update changelog < Update changelog < Update changelog < Update changelog < Update changelog / make docsgen < Update changelog and version < Update changelog to RC2 < Update date and upgrade warnings < add ETH addrs API to Gateway (#11979) < address review - resolve one more conflicts - revert 2 new line added < build: release: v1.27.0-rc1 (#11947) < build: release: v1.27.1-rc2 (#12101) --- > Update codename for NV23 (#12104) > add backport PR > build release > build: release: v1.28.0 (#12271) > build: release: v1.28.0-rc2 (#12179) > build: release: v1.28.0-rc3 (#12186) > build: release: v1.28.0-rc4 (#12190) > changelog > chore: actors: pull in the latest v14 bundles > chore: actors: update to builtin-actors@v14.0.0-rc.1 39,40c19 < chore: backport #12054 to release/v1.27.0 branch (#12056) < chore: ci: allow using pat for tag creation during release --- > chore: bump version in master (#12074) 41a21,22 > chore: ci: use self-hosted runners in the original repo or any fork inside the filecoin-project org (#12134) > chore: curio: remove forgotten parts of curio config (#12087) 42a24 > chore: deps: prep for upstream removing ConstructNilRouting() 44,49c26,31 < chore: deps: upgrade to libp2p@v0.35.3 from v0.34.1 < chore: libp2p: update to v0.34.1 (#12027) < chore: lint: address feedback from reviews < chore: lint: fix lint errors with new linting config < chore: lint: update golangci lint config < chore: remove unmaintained bootstrappers (#11983) --- > chore: deps: update ffi and go-state-types for NV23 (#12156) > chore: deps: upgrade to go-state-types@v0.14.0-rc2 > chore: docs: add changelog upgrade warning for events db migration (#12145) > chore: docs: remove unused badge from the README (#12097) > chore: merge releases back to master (v1.27.1) (#12141) > chore: test: refactor, fix and improve unmanaged miner (#12143) 51,58c33,35 < chore: update drand (#12021) < chore: update go-data-transfer and go-graphsync < ci: deprecate circle ci in favour of github actions (#11786) < doc: eth: restore comment lost in linter cleanup < doc: events: note events db migration impact < docs: lotus v1.27.2 prep < docs: lotus v1.27.2-rc1 prep < docs: lotus v1.27.2-rc2 prep --- > docs: changelog: fix link syntax (#12085) > docs:Building_a_network_skeleton.md cleanups (#12079) > feat!: ni-porep for lotus-bench 60,61c37 < feat: ci: automate the new release process < feat: ci: remove gorelease, create drafts and automate release notes --- > feat: ec: Integrate Fast Finality for Filecoin (#12119) 63,64c39,46 < feat: libp2p: Lotus stream cleanup (#11993) < fix go releaser and test with rc3 --- > feat: itest: minor UnmanagedMiner refactor for reusability and simplicity > feat: lotus-bench: add script to benchmark standard sector steps > feat: niporep: introduce SealProofVariant > feat: test: add LOTUS_RUN_VERY_EXPENSIVE_TESTS and include niporep real-proofs > feat:ec: integrate F3 dynamic manifest (#12173) (#12185) > final changes > finish changelog > fix typos 65a48 > fix: ETH getLogs: fix slowness at head and ignore null blocks (#12207) 66a50,51 > fix: build: pdate UpgradePhoenixHeight to -26 in inteop (#12106) > fix: ci: bump manual onboarding test to 4xlarge 68c53,54 < fix: copy Flags field from SectorOnChainInfo --- > fix: ci: post sorted pr checks comments under PRs from forks > fix: ci: update concurrency group settings of the sorted pr checks workflow 70a57,60 > fix: fevm: update tests for latest actors bundle (#12144) > fix: genesis preseal: get it working > fix: itest: fix wdpost proving epoch calculation > fix: niporep: unflake provecommit failure for niporep 72,76c62,72 < go mod tidy < lotus v1.27.1 prep < release: v1.26.3 (#11908) (#11915) < revert go version change (#12050) < small fix in changelog --- > fix: state: rename Actor.Address and only use it for f4 addresses (#12155) > nv23 deps (#12129) > only soft laungh > run CI > soft launch > test: niporep: initial niporep e2e test > update CL > update genesis network version for dev testnets > update go state types > update go-state-types (#12154) > update version string ```

[BigLep]

Per verbal, we are going to go with option 1 above. @rvagg is going to work on this PR. (I also updated the text to better clarify what is meant by "option 1").

[rvagg]

https://github.com/filecoin-project/lotus/pull/12299

[rjan90]

Closing this now that https://github.com/filecoin-project/lotus/releases/tag/v1.28.1 has been published, and also the final step of merging releases branch back to master has been merged https://github.com/filecoin-project/lotus/pull/12317