Postmortem of trinity v0.1.0-alpha.20 release

[carver]

What is wrong?

Releaser: @carver

There was a bug in the code that only affected packaged code (the source project was unaffected). In the process of fixing it and re-releasing, a lot of other problems occurred.

• The first problem was mostly inconsequential, but set the mood: the first release crashed mid-flow with

  twine upload dist/*
  ContextualVersionConflict: idna 2.7

• That didn't cause any public problems, because the upload failed. It was fixed by requiring a more recent version of requests in: efce4981ff095b0d9d6249974093fb63a421c39e and re-running the twine step of the release
• Right after the release, I ran a local smoke-test of the package (trying to sync ropsten). It failed with an error (like):

  FileNotFoundError: [Errno 2] No such file or directory: .../trinity/assets/eip1085/ropsten.json

• :rage1: anyone who upgrades trinity to the alpha 18 (or installs it for the first time) will be greeted with a hard crash! Time to identify/patch/release as quickly as possible...
  • Relatively quick identification, luckily: trinity had added genesis file support, and used that new support when launching Ropsten and Mainnet. The feature relies on genesis file assets being included with the release (narrator: They weren't.)
  • Patch
  • Unfortunately, I am not familiar with the manifest format, so it took more than looking at this manifest file to understand why it failed: https://github.com/ethereum/py-evm/blob/a52e5446d85461e751ac0a83b3def77a95132c2b/MANIFEST.in#L9
  • After inspecting the log output of the local python setup.py sdist bdist_wheel run, and inspecting the trinity/assets folder, it became clear that the files were included, but not the subfolders.
  • A quick stackoverflow search later, and I found the different include command to use
  • Ok, so patch made, with a quick re-release of trinity in sight: https://github.com/ethereum/py-evm/commit/8d902b4f3b56b903a788a82548a77a4537ab46f2
  • (for those who enjoy dramatic irony: find the new bug, which I have not yet detected)
• I was a bit more cautious with the release this time: I wanted to catch problems before the release so I manually ran through each of the commands in the Makefile manually, stopping just before the twine upload: https://github.com/ethereum/py-evm/blob/a251ffbd773a6d062497341ae41789be8b76dcd3/Makefile#L64-L69
• (more dramatic irony, find the two new bugs that have accumulated -- total undetected bugs: 3)
• Good news: I found an issue during this "cautious" approach, before uploading another busted release with twine:
  • The python setup_trinity.py line lists out all files added via manifest, and the assets subfolders still weren't listed...
  • :rage: I didn't notice that the recursive-include command uses a different format for the argument, which space-separates the folder from its contents, instead of slash-separating
  • Found "New Bug 0" -- total undetected bugs: 2
  • fixed locally with: https://github.com/ethereum/py-evm/commit/30779883444ff508c7975c5dbd5e52221f7876ec
  • Re-run the local makefile steps again. This time the files show as included. Fixed, hooray! :tada: (narrator: It wasn't.)
  • This is a tiny fix of a fix within a few minutes, which hasn't been released yet. Squash it, update the readme with the busted release (v18), and force push to master.
  • Oh, py-evm doesn't allow force-pushing to master. I guess that's reasonable for a high-traffic and sensitive project, but it was news to me. Undo the squash by looking at git-reflog history and push it up.
  • Ok, finally, time to do the next trinity release: v19
• twine upload trinity v19. Smoke test. Same failure. :rage2: :rage2:
• Oooh, despite the assets being in the trinity module, they are actually bundled (and searched for) in the py-evm package... Great, time to re-bump py-evm and trinity, and release again :man_facepalming: .
  • Found "New Bug 1" -- total undetected bugs: 1
• Bump py-evm to a new version, run the local build steps to confirm the manifest includes the right files.
• This time I'll be ultra cautious, and look up how to test the final package before uploading via twine
• Oh, I can pip install the wheel in the dist/ directory, that's neat. Why weren't we doing that all along? No time for that now...
• pip install the wheel into a fresh virtualenv, after looking up how to install a wheel with extras, ie~ pip install 'py-evm/dist/py_evm-0.2.0a37-py3-none-any.whl[p2p,trinity]'
• It works, hooray! Time to do the release, with total confidence that the release would go completely smoothly. (narrator: It didn't.)
• twine upload/* the py-evm build, watching the logs scroll by. See a trinity wheel getting uploaded...
• At this point, I'm on tilt, and Ctrl-C because I don't want an unexpected package to be uploaded.
• I immediately regretted the keypress, thinking that whatever intermediate state I just caused was probably worse than another bad (but otherwise normal) release.
• WTF, why was there a trinity package being uploaded? :rage3: :rage3: :rage3:
• Oooh, when I was manually running the Makefile, I wasn't manually running the clean step which is tucked away on the release: line... https://github.com/ethereum/py-evm/blob/a251ffbd773a6d062497341ae41789be8b76dcd3/Makefile#L63
• Found "New Bug 2" (new undetected bugs: 0)
• Now it's time to go see how badly I borked the py-evm release by hitting Ctrl-C in the middle.
• Make a fresh virtual env, install the latest pip install -U py-evm[p2p,trinity], and get some terrible crash that I don't recognize/remember :rage4: :rage4: :rage4: :rage4: (did you spot my mistake?)
• I don't have time to diagnose how/why the Ctrl-C broke it, intermediate states can fail in all sorts of weird, hard-to-diagnose ways. People could be installing trinity right now! Just bump py-evm one more time and release: 68026a4524108f8840aa3602f6205355055a9e4a
• Local smoke test. It works! :tada: What a :roller_coaster:
• Take some notes for future post-mortem. Although I made several mistakes, one or two human mistakes shouldn't cause this much of an "outage" (although we're in alpha, better to treat it like production, so we are ready when it is stable).
• End to end, it was ~45 minutes of the latest release being broken.

Follow-ups:

• Remember when things looked super broken after the Ctrl-C? -- It turns out it was because I forgot to --pre in my pip install. Weirdly there is currently a stable v0.1.0 release on pip (which is super old & busted). So (AFAICT) the last py-evm version bump was unnecessary.
• Finally, I "broke the build" (red :x: in CI) on master when I hot-patched the live release. It looks ugly, but turned out to simply be a warning in the docs. It is bad practice in general to have a red master, but IMO, quickly fixing a broken release outweighs a doc warning on master.

How can it be fixed

• [x] split the Makefile targets in two:
  • [x] one to produce a local release wheel for testing
  • [x] another to push to git (esp since master is not force-push-able) and upload to pypi via twine
• [ ] update the command line smoke tests in CI to generate a local build using the first of those two new Makefile targets ^, and run the test from the wheel (instead of running from source). eg~ pip install 'py-evm/path/build.whl[p2p,trinity]'
• [x] delete the v0.1.0 stable build of py-evm
• [ ] (already planned) move trinity/p2p out of py-evm repo, so the MANIFEST.in isn't reused in both setup.py builds.
  • This would have prevented the issue about fixing the manifest and releasing the wrong package
  • This also will speed up the process of creating the trinity version bumps/tags
• [ ] Add a recommendation to roll back a stable release that gets borked, here: https://github.com/ethereum/snake-charmers-tactical-manual/blob/master/cutting-a-release.md (tag an old commit with a new version number, build, & release)
• [x] Prefer longer underlines on doc headers than needed, so that adding a few characters doesn't break the doc build: ac7d86537b791540ce4c5bc27398eab06fb2f63e

[carver]

• delete the v0.1.0 stable build of py-evm

I wanted to highlight this. Anything I'm missing about why it might be a bad idea?

[carver]

Oh, and if this had been for a stable release, the best course of action would probably have been to immediately issue a rollback version that tags an old commit with a new version number. Then anyone doing pip install -U or installing fresh will get a working version. It adds more churn to the versioning, but that's better than trying to rush into finding a solution.

A note about that probably belongs here:

• [ ] https://github.com/ethereum/snake-charmers-tactical-manual/blob/master/cutting-a-release.md

---

That seems like the closest thing to the "best practice" from the web app deployment world of always having a "revert deploy" button easily accessible. It doesn't translate perfectly to pip, since

[pipermerriam]

I wanted to highlight this. Anything I'm missing about why it might be a bad idea?

I think it's fine. I'm pretty sure that's a leftover of me initially not finding the UX for deleting repositories in the pypi interface.