Support for Poetry

[mixxorz]

Poetry is a python package manager, similar to Pipenv.

At the moment, heroku-buildpack-python supports both vanilla pip (via requirements.txt) and Pipenv (via Pipfile, Pipfile.lock). Unfortunately, there's been a lot of backlash with regards to Pipenv in the Python community, and on the back of that, Poetry was born to serve as a better Pipenv.

That being said, it would be good if this buildpack can detect and install dependencies using Poetry. The package manager uses two files: pyproject.toml (which has been provisionally accepted in PEP 518) and poetry.lock.

[kennethreitz]

Perhaps a bin/install would be appropate (in the app repo), which would replace the entire pip install step with whatever logic you like!

[CaseyFaist]

Hey there! It's on the product roadmap to add experimental Poetry support - there's a user submitted PR #801 that I might be able to iterate on. I'll circle back in a few weeks with a progress update, but keep an eye on that pr for more details for now 👍

[bsdz]

I think one wouldn't need explicit poetry support but simply the latest version of pip (>19.). See https://github.com/sdispater/poetry/issues/321. Would be nice if one could easily configure which pip version is used from a buildpack runtime.txt or similar.

[rschuetzler]

It looks like #801 was closed by the user. Any update on the status of Poetry support?

[bsdz]

It looks like #801 was closed by the user. Any update on the status of Poetry support?

Not sure if helpful but another PR (by me) also adds Poetry Support

[b0g3r]

Looks like progress about feature is frozen :(

[cjolowicz]

Here is a series of PRs which implement support for Poetry and other PEP-517 compliant build backends:

• 833

• 834

• 835

[RobRoseKnows]

Curious as to the status of this. There seems to be numerous community PR to do this, but none of them are merged.

[CaseyFaist]

Thanks for checking in on the status y'all - the first of @cjolowicz 's prs (pip update to 20) has been merged and released, and fully plan on working through the last two of that trio to provide poetry support. I'll leave this issue open until resolved, but will close other Poetry feature requests so we can track progress and questions in one place.

[CaseyFaist]

Good news everyone! Prepare yourselves for a rambly update...

So, while I've been pushing out smaller improvements, this issue has been hanging around in the back of my mind. I think one of the lessons from the way pipenv support was added is that a tight coupling between pre-python install (path updates, necessary system packages, flags, more sysadmin-type setup) and post-python install (pip installs, package installs, any sort of venv setups) leads to a lot of user and maintenance friction. Especially as the package changes, grows and adapts.

We do some magic to reach into the pipenv config files and do the right thing with them in the sysadmin step, because it as a tool doesn't cause deterministic builds at the python install level, it just documents what the determined python version for the build should be. The proposed solution in #835 is good work, but follows the same convention - and, I'm concerned the convention sets us up for that same friction.

From Poetry's documentation, the solution to that is to rely on pyenv for pre-python setup, and then poetry for post-python setup, with a clearer divide between the two. This is the path forward here.

Poetry has a lot of promise and I'm very excited about it, but I don't believe Heroku should be restrictive - it should be possible to use the tooling of your choice, and it should be plug and play. So I want to be able to easily plug and play new systems as they gain ground and maturity (and may no one wait this long again 😂). That's the benefit of a strict pre/post python divide using pyenv (or at first, just a .python-version file) -- it's a simple and straightforward base to build on, whether it's for Poetry, standard pip workflow flow or other tools. The downside is that it's yet another file to be specified on Heroku, albeit a community-standard one. So while I hate to ask for duplicate specifiers or files, I think we need both.

With this decision formed, though, expect to see progress towards this speed up a bit in the next several weeks. Especially after I'm able to figure out how to record my PyCon talks from my home 😬

[alfondotnet]

Hi @CaseyFaist !!,

has there been any update regarding this?. Not trying to be pushy, just curious :-)

[edmorley]

@alfonsoperez Hi! I've just taken over from Casey as Python owner. Exploring support for alternative package managers is definitely on my list of things to do, though the overall list is pretty long at the moment, so not sure as to timeframe :-)

[nextmat]

Hi @edmorley, great to see all the things you are cleaning up!

Just wanted to check back and see if you have any sense of a likely timeline to support or if this is still a ways off?

[edmorley]

@nextmat I don't have a timeline at the moment, but this is definitely bubbling up closer to the top of the list, now some of the more urgent issues have been addressed :-)

[ipmb]

It'd be great if the buidpack could leverage pyproject.toml to make this determination. PEP-518 already provides a standard way to define a build system, but Heroku could leverage its own tool table for Heroku-specific configuration as well.

[nextmat]

In case this helps anyone else, this buildpack works as an interim solution if run first.

It exports a requirements.txt file from Poetry to allow the official buildpack to work afterwards.

[anentropic]

I agree with @ipmb ... this should be less about specific support for Poetry or Pipenv and more that any modern Python buildpack should detect a pyproject.toml file and support PEP-518 (this can replace both requirements.txt and runtime.txt)

But yes, I would like Poetry support

[apollo13]

@anentropic How can detection of the buildsystem replace something like runtime.txt? Especially given that you can specify something along the lines of python = ">=3.6,<3.9" in pyproject.toml for poetry. So which python would heroku choose and how would it know where to look in the first place? Similar issues imo exist for requirements.txt

[anentropic]

@apollo13 it should do the same thing other tools do

if a range is given they try to use the latest in the range

but if I'm deploying my app to heroku I'm going to specify exactly the python version I want in that field, and if I don't it's on me if I get a surprise

[apollo13]

@anentropic Fair enough, but not every app is ment to be deployed to just heroku, in that sense I think runtime.txt makes sense to override the range. That said, how do you think heroku would parse that pyproject.toml? parse the buildsystem to see if it is poetry and then custom code to parse whatever the tool uses to specify the python version? I still think that generically determining that without support for the individual tools will not be possible. In that sense, explicit poetry support is still needed.

[ipmb]

If the existing info is too vague, a [tool.heroku] table could be used in pyproject.toml to define Heroku-specific values https://www.python.org/dev/peps/pep-0518/#tool-table

[michaelhays]

Hey @edmorley, any update on an estimated timeline for this?

No worries if not, I'm mainly curious because I've been putting off switching from Pipenv to Poetry for when Heroku has official Poetry support. But if that's still likely to be several months off, then I'll just go ahead and use the community Poetry buildpack.

[edmorley]

@michaelhays Hi! Non-Python projects consumed most of my time the last few months (eg Heroku-20 beta/GA/default stack, Cedar-14 EOL + sunset, Heroku-16 deprecation, amongst others), but I'm now able to return to the Python buildpack for the next few months. Top of my list is fixing the somewhat dismal test coverage (and non-production-representative tests), after which it will be much safer to make further changes. I would use the community buildpack for now and I'll update this issue when I start working on native support :-)

[zyv]

Hi @edmorley ,

Author of the Poetry buildpack here :) I hope it won't come across as an unwelcome intrusion - my intention is not to advertise our stuff or interfere with your plans... I understand that you're up to investing quite some work into the official Python buildpack, so maybe it would be a good thing to have a discussion as to which direction should we take?

Back when we migrated to poetry at my shop, I was hoping that it will be officially supported by the Python buildpack soon, but we ended up writing our own buildpack because we needed to solve our problem fast. Composing our buildpack with the official buildpack ended up working amazingly well for us so far for almost two years I think, and now I'm wondering whether the "official" support makes any sense at all and whether it's good for the users or not...

In as far as our buildpack is concerned, I think it's pretty feature-complete by now (there is a number of environment variables you can set, and it can optionally take care of generating a runtime.txt for you), and I have also recently implemented a rather comprehensive test suite. Feel free to have a look and steal whatever you like - it's all MIT licensed.

Back when we were using pipenv, which was officially supported by Heroku Python buildpack, we were actually quite unhappy about this support - pipenv versions kept changing under our feet breaking the deploys all the time, you couldn't influence pipenv enough through environment variables. Getting stuff fixed by filing the pull requests in the official buildpack repository wasn't really working either, and so we ended up generating requirements.txt from our CI anyways.

Then we switched to poetry and our own pre-buildpack as described above and have been quite happy until now. So, if you decide to implement the "official" support, then I hope that you would take this past experience of ours into account...

On the other hand, there are obvious advantages to the "official" support - (1) one only needs one buildpack and (2) one doesn't need to rely on third-parties that are not affiliated with Heroku and can in theory stop taking their pills anytime without warning ;-)

One thing that I think fits nicely into the official buildpack is the support for pinning Python versions or picking the correct runtime - this would be rather a pyproject.toml support topic than poetry support though.

Anyways, I would be quite interested in hearing your thoughts. Do you think build tool support belongs to the buildpack at all, or maybe there should be a Python buildpack that does requirements.txt perfectly with all the caching and reinstalling magic, and pipenv, poetry and other package manager preprocessors? If you think that composition is actually a good thing, then would you / Heroku want to co-maintain or take over our buildpack?

Cheers!

[sidmitra]

https://github.com/moneymeets/python-poetry-buildpack is the buildpack @zyv means. I had to go find it, since i wasn't aware it existed.

[bolinocroustibat]

@sidmitra Thanks! I think the poetry buildpack should be promoted more. Actually using pyproject.toml instead of requirements.txt has nothing to do with Poetry, I believe a Heroku buildpack using pyproject.toml could actually be package-manager-agnostic and doesn't even need to name Poetry.

[frafra]

Actually using pyproject.toml instead of requirements.txt has nothing to do with Poetry

Different tools store dependencies in different sections of pyproject.toml.

poetry is not PEP 631 compliant (https://github.com/python-poetry/poetry/issues/3332): https://github.com/python-poetry/poetry/blob/4ec09d4e6b8ca007e67bb873c96277b54986fbdb/pyproject.toml#L31-L52 pdm is PEP 631-compliant and stores dependencies under [project]: https://github.com/pdm-project/pdm/blob/952bde688338ac4d8bf810712acc5a7148284ed1/pyproject.toml#L11-L25

[AstraLuma]

I will note that PEP 621 was accepted less late last year, and PDM is younger than Covid-19.

[edmorley]

@zyv Hi! Sorry for not replying sooner :-)

Glad to hear that the current approach with the supplementary Poetry buildpack is working well for now. I do believe there is still value in first-party support for Poetry for what it's worth.

However we also want to make it easier to extend buildpacks too, which is one of the benefits we'll get once we migrate to Cloud Native Buildpacks (https://buildpacks.io) due to its support of buildpack composition. For example having a "python-engine" buildpack that's consumed by a pip buildpack, but could equally be consumed by a poetry/pipenv/... buildpack.

We're in the process of writing new CNBs for each language (initially for the Salesforce Functions product, but eventually they'll replace the existing Heroku buildpacks) - Node.js and Java have a native CNB, Python is next. The Python repository is currently empty, but development will be occurring here: https://github.com/heroku/buildpacks-python

[ddahan]

@edmorley Thanks for the answer, glad to hear that Heroku is still working on innovative things (including in Python land). (And off-topic, but this article made me doubt of it).

[zyv]

@zyv Hi! Sorry for not replying sooner :-)

No problem at all, and thanks for the update! So, if I understood it correctly, the plan is to leave the classic buildpack as is for now, and provide official Poetry support for the CNB buildpack later, because it would compose well? This certainly feels like a sound idea to me.

We'll continue to maintain our buildpack as long as we are still using Heroku, but quite honestly with all the outages that happened in the last year and a half, and annoying changes like 2FA + reduced login session lifetime, Salesforce banner in the dashboard, difficult GDPR story and stuff like this are pushing us to move directly to AWS, even if it's quite a lot of work :-/ So at some point we might have to put it up for adoption...

[edmorley]

Hi :-)

So, if I understood it correctly, the plan is to leave the classic buildpack as is for now, and provide official Poetry support for the CNB buildpack later, because it would compose well?

Yeah that's correct. Since the CNBs will eventually replace the classic buildpacks, we don't want to invest too much time in the latter - though once the CNBs exist there may be some amount of backporting to the classic buildpacks, to make the transition easier.

[Garrett-R]

Node.js and Java have a native CNB, Python is next.

I'm curious if there's any ETA on the Python native CNB. Sounds pretty cool BTW!

[edmorley]

Hi! We've been working on libcnb.rs, which will power our CNBs moving forwards. That library is now in a good shape (as of the last few weeks), so work on the Python CNB will be starting imminently.

[ulgens]

@edmorley Thanks for the update! Is there any resource-channnel we can track the progress of Python CNB implementation?

[edmorley]

I would repo watch:

• https://github.com/Malax/libcnb.rs
• https://github.com/heroku/buildpacks-python (currently empty, but once past design/research/local prototyping will start to have content added)

[edmorley]

Also, if you have opinions on ways to specify Python version, feel free to comment on: https://github.com/heroku/heroku-buildpack-python/issues/913#issuecomment-1031468659

[edmorley]

Thank you for everyone's patience here! I've finally had a chance to return to this buildpack after working on the Python CNB (the next-generation Cloud Native Buildpack, that's set to replace this one; see the Python CNB repo and the CNB preview announcement).

As of #1682 this buildpack now supports Poetry :-)

If you are using the moneymeets/python-poetry-buildpack buildpack you will need to remove that buildpack in order to use the native Poetry support, since otherwise the requirements.txt exported by that buildpack will take precedence (intentionally done this way for now for backwards compatibility).

You will also need to add a .python-version file to specify the Python version, since we don't support reading the version from poetry.lock (see the #1682 PR description for more details). Our newly added .python-version file support also supports specifying just the major Python version (eg 3.13 rather than 3.13.0) , meaning you can get automatic security updates without having to remember to bump the version every time an upstream CPython release occurs.

For all other details on the Poetry usage/behaviour, see the #1682 PR description.