Micropipenv does not work with Poetry 1.5.0+

[abompard]

Since version 1.5.0, Poetry no longer writes the "category" key into the poetry.lock file (poetry#7637). This breaks micropipenv:

+ micropipenv install --deploy
Warning: Currently, Micropipenv is not able to parse complex Python version specifications used by Poetry. Desired version: ^3.10, current version: 3.10.
Failed to parse poetry.lock and pyproject.toml: 'category'
error: build error: error building at STEP "RUN /tmp/scripts/assemble": error while running runtime: exit status 3

A comment in the ticket suggest that we could just do a .get(("category", "main") for a simple fix. I tried that locally and it works around the crash.

A better fix, I think, would be to get the information whether a package is in the "dev" group from the right place, since the category key is apparently no longer relevant. It does not seem to be in poetry.lock anymore though, maybe we'll have to read it from pyproject.toml.

[frenzymadness]

Thanks for the detailed report. I won't have time to take a deeper look at it sooner than the end of next week so PRs are more than welcome.

[goern]

/kind feature

[frenzymadness]

I think that we have a bigger issue here. With kinda rapidly evolving poetry, the current approach "convert poetry.lock into pipfile and then install whatever is in that pipfile" doesn't work anymore. Poetry supports groups so default/dev groups in micropipenv are not enough anymore. And if we add groups to micropipenv, the intermediate file would be different from Pipfile.lock so we might need to rewrite that part and handle both files differently. That would also require rewriting the CLI to allow (de)selecting groups.

I'm not sure about the best path forward now.

[frenzymadness]

As I'm thinking about it, I see two ways forward. First, let me summarize how I understand how micropipenv works now:

• micropipenv supports Pipfile.lock files directly and it can install from them as they are. Pipenv and Pipfiles are pretty stable so there is no much maintenance burden.
• The poetry support is done by converting info from pyproject.toml and poetry.lock files into Pipfile.lock file and then using the functionality mentioned above.

If we want to keep the support of poetry (which I think is important because poetry is much more popular than Pipenv), we can either implement full support or make some compromises.

• full support means support for groups. pyproject.toml might contain custom groups of packages and to respect that we need to:
  • offer the possibility to enable/disable them (via CLI), and
  • diverge the format we convert the poetry.lock into from Pipfile.lock, or
  • create a new internal format capable of handling both methods.
• compromise might mean that we find a way how to convert complex custom groups of packages from poetry.lock/pyproject.toml into Pipfile.lock so we can keep most of the current functions. That might mean having tool.poetry.dependencies as the main/default ones and all other groups (test, docs, …) combined together into develop.

IMO we can start with the latter and see how many users will need the full support. @abompard how important is it for you to handle specific groups of dependencies in your projects?

[frenzymadness]

Another possibility for micropipenv is to become the leader in creating a new lockfile standard as discussed here and in #249 . That would mean we can include all important information from pyproject.toml and all other lockfiles into the new standard format used internally only in micropipenv. And if it works and covers all known use cases, we can propose it upstream as a PEP. Not sure how doable this is and how much time other maintainers can invest in this. Cc @fridex @VannTen

[fridex]

+1 on rewriting the current poetry handling, IMHO this is the best long-term solution. Keeping the internal conversion from poetry.lock to Pipenv.lock can just cause issues long term. I could imagine some internal intermediate representation of what needs to be installed, but not in the form of a Pipfile.lock as it might cause troubles.

... we can propose it upstream as a PEP.

The scenario with the lockfile would be great - I'm not sure the current state of the lockfile but note the rejected PEP-665.

[goern]

@fridex thanks for the input/opinion

@frenzymadness @VannTen do you think you guys find some time this quarter to work on it?

[VannTen]

Unlikely. I'm leaving Red Hat and while I'm probably gonna do some Ansible/Python stuff, I surely won't be involved in micropipenv for quite some time.

[goern]

Unlikely. I'm leaving Red Hat

🙄 sad to hear that! I hope it was at least a little good to stay with us?!

[VannTen]

I hope it was at least a little good to stay with us?!

Sure it was !

[marksie1988]

Does anyone know of a workaround for this issue? is there a way to tell micropipenv to use a version of poetry that does work?

[abompard]

@marksie1988 I'm personally rebuilding the lockfile with an older version of poetry in .s2i/bin/assemble: https://github.com/fedora-infra/fmn/commit/04152981

[frenzymadness]

Thanks, @abompard, for the quick reply. Micropipenv does not use poetry directly, so the workaround for now is to generate lockfile with an older version of poetry.

[marksie1988]

@marksie1988 I'm personally rebuilding the lockfile with an older version of poetry in .s2i/bin/assemble: fedora-infra/fmn@04152981

Thanks this worked for me 👍

[frenzymadness]

Fix ready for review in #284