search

freedomofpress / securedrop-builder

Packaging logic for building SecureDrop-related Debian packages
GNU General Public License v3.0
7 stars 11 forks source link

Use poetry for bootstrap dependency mgmt #468

Closed eloquence closed 10 months ago

eloquence commented 1 year ago

Once #467 lands, we can migrate the bootstrap dependency management from pip-compile to poetry.

eloquence commented 11 months ago

Took a first stab at this in e39d74065c299a37463a0e7372cbc7ee5329da99. Our reqs are pretty outdated so I had to version-pin a couple more to avoid incompatibilities -- however nothing should be older than in the current bootstrap. I think a holistic refresh (#464) would be a good follow-up task.

Currently this is failing CI on Bookworm; as far as I can tell, it's attempting a source instead of wheel install for cython, and then rightly complaining that the sha256 for the wheel doesn't match the source tarball. Not sure why it's doing that, see https://app.circleci.com/jobs/github/freedomofpress/securedrop-builder/30477 for CI output. Will try to repro with Python 3.11 and if necessary with a Bookworm VM.

eloquence commented 11 months ago

In hindsight it of course makes sense that Python 3.11 wants a different version of Cython. I've built one and added it in f7465d074613d84bd87d70b55bfa27eeac24afb1, which make the bookworm jobs happy. Open Q: how much diff review do we want to apply to these core Python ecosystem dependencies? Will raise that in standup tomorrow.

Otherwise next up: making reprotest happy.

eloquence commented 11 months ago

To recap, as part of migrating the bootstrap to Poetry, I tried to have Poetry update any bootstrap dependency that wouldn't cause immediate further breakage.

The reason reprotest-wheels is failing on f7465d0 is that the new bootstrap produces new wheels with different checksums. I suspect this is due to the Cython bump.

Additionally, the Cython bump won't work with securedrop-proxy currently due to https://github.com/freedomofpress/securedrop-client/issues/1681.

I'm not sure anymore if it's a good idea to include any wheel updates in the initial Poetry migration.

I would suggest we migrate to Poetry for the bootstrap now by effectively pinning all versions in pyproject.toml to the versions used in the current bootstrap. This should enable us to finish this part of the Poetry migration, handling bootstrap updates separately.

Let me know if you have any concerns with that approach, if not I'll proceed along those lines on Monday.