poetry>=1.2 package with group dependencies cannot be installed with default build-system

lindycoder commented 2 years ago

[x] I am on the latest Poetry version.
[x] I have searched the issues of this repo and believe that this is not a duplicate.
[x] If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

OS version and name: macOS Monterey 12.0.1
Poetry version: 1.2.0a2
pyproject.toml: default from poetry init -n

Issue

Hello! Thanks a lot for this tool.

When creating a project with poetry>=1.2 it adds the default build-system:

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

If you add a --group dependency in this project, it becomes impossible to install because the latest stable release of poetry-core does not support group and produces

Traceback (most recent call last):
  File "/tmp/tmpbqlvtksr_in_process.py", line 280, in <module>
    main()
  File "/tmp/tmpbqlvtksr_in_process.py", line 263, in main
    json_out['return_val'] = hook(**hook_input['kwargs'])
  File "/tmp/tmpbqlvtksr_in_process.py", line 133, in prepare_metadata_for_build_wheel
    return hook(metadata_directory, config_settings)
  File "/tmp/pip-build-env-l3emamb6/overlay/lib/python3.10/site-packages/poetry/core/masonry/api.py", line 43, in prepare_metadata_for_build_wheel
    poetry = Factory().create_poetry(Path(".").resolve(), with_dev=False)
  File "/tmp/pip-build-env-l3emamb6/overlay/lib/python3.10/site-packages/poetry/core/factory.py", line 43, in create_poetry
    raise RuntimeError("The Poetry configuration is invalid:\n" + message)
RuntimeError: The Poetry configuration is invalid:
  - Additional properties are not allowed ('group' was unexpected)

Reproduce

This Dockerfile shows the issue

FROM python

RUN pip install "poetry==1.2.0a2"
# Create a package with a dev dependency
RUN mkdir -p /mypkg/mypkg \
    && cd /mypkg \
    && touch mypkg/__init__.py \
    && touch README.md \ 
    && poetry init -n \
    && poetry add --group dev pytest

 # Create a project that will import the package
 RUN mkdir -p /importer \
    && cd /importer \
    && poetry init -n

 # Trigger the error
 RUN cd /importer && poetry add /mypkg

Mitigation

Changing the build dependency to requires = ["poetry-core>=1.1.0a6"] will currently fix the issue

Resolution

If poetry 1.1 was forward compatible (as raised here #4628) it could "maybe work" if the only groups are dev dependencies not meant to be installed with the package.
Maybe poetry 1.2 could set a minimum build dependency, which would be poetry-core>=1.1 but i did not find a way to tell the build system to install with --pre to allow for pre-releases, so it would need to be explicitly something like poetry-core>=1.1.0a6.

Maybe this is not an issue, once poetry==1.2 and poetry-core==1.1 are officially release this might all go away.

Thank you very much!

kakulukia commented 2 years ago

I was just stumbling upon this while trying to build pendulum. Is there a way around this?

lindycoder commented 2 years ago

@kakulukia From the look of pendulum's pyproject.toml they use poetry >=1.2. Do you have this version installed? Needs to be installed using the preview flag, see installation here : https://python-poetry.org/docs/master/

kakulukia commented 2 years ago

I was not aware that its using a preview version of poetry .. thx, its working now.

ljnsn commented 2 years ago

I have the same issue on version 1.2.0b1.

AkechiShiro commented 2 years ago

Hey, why is pendulum requiring a preview version of poetry for installation was this necessary ? I had to build from source poetry as the one I use from my classical repos is too outdated, but I find this weird, for one package to be installed, we have to update the whole package manager...

Secrus commented 2 years ago

Hey, why is pendulum requiring a preview version of poetry for installation was this necessary ? I had to build from source poetry as the one I use from my classical repos is too outdated, but I find this weird, for one package to be installed, we have to update the whole package manager...

I don't think this is an issue of Poetry, but a decision of Pendulum maintainers. You should raise the question in the Pendulum repository.

AkechiShiro commented 2 years ago

Hey @Secrus, just for reference, it seems it isn't a Pendulum issue but a pip issue : https://github.com/pypa/pip/issues/6264

TBBle commented 2 years ago

To be clear, the original issue this ticket was raised for still exists: A Poetry 1.2-managed project which uses groups (per the original example) still declares an unversioned build-system requirement on poetry-core, and so will fail if this is not manually adjusted to depend on poetry-core>=1.1.0a6 or similar. I just checked this hasn't changed in Poetry master (from earlier this week, anyway).

Pendulum already has this versioned build-system requirement, so it's not suffering from the issue in this ticket.

Poetry itself (in master) depends on poetry-core ^1.1.0b3, so it might make sense for Poetry projects created with poetry 1.2 to depend on that version of poetry-core for their build-system from the start; for existing projects, that requirement could be adjusted when some poetry-core-1.0-incompatible change , e.g. groups, or the newer console scripts support, is made via poetry CLI.

It might also make sense for the documentation to mark those pyproject.toml features that depend on poetry-core 1.1 in the documentation so that people editing their pyproject.toml manually with reference to the documentation, e.g., as happened in #4628, will know they need to ensure their build-system dependency is appropriately updated.

And as a further molly-guard, it might make sense to have Poetry notice when pyproject.toml contains features needing poetry-core 1.1 but does not declare this in its build-system block, similar to how it notices when your poetry.lock is out-of-date compared to pyproject.toml.

All this depends on having a clear record and documentation of the poetry-core 1.1 added and changed features, and the ability to recognise them in pyproject.toml. And parse the build-system requirement line to determine which version of poetry-core is being depended upon..

Separately, it might make sense for Poetry 1.2, when given add --group dev, to generate the poetry-core 1.0 format if the build-system requirement is not versioned for poetry-core 1.1 already (as opposed to bumping the poetry-core dependency as I suggested above). However, I assume that option would have been discussed in the initial implementation of generalised group support and not seen as valuable at the time.

finswimmer commented 1 year ago

The original issue is solved, because poetry-core 1.1 is out.

wiseaidev commented 1 year ago

I recently stumbled across this issue while building brave-chat/brave-chat-server after adding python 3.8.10 to circleci. However, there is no issue with python 3.9.10. So, it seems tied to the python version being used.

neersighted commented 1 year ago

Something is wrong in your environment and you are building with an old version of Poetry installed using get-poetry.py, as evidenced by the path ~/.poetry/lib/poetry/_vendor/py3.8/poetry/core/factory.py which indicates both the old installer and a 1.1.x build using the old distribution method.

Please reach out on Discord or through Discussions instead of commenting on old issues; usually any relation in a solved issue is surface level, like this one.

wiseaidev commented 1 year ago

Oh, thanks for the reply. I didn't notice it is actually a closed issue. I totally missed that, ma bad. However, in response to the method I am currently using to install poetry is not the get-poetry.py one, but rather with the following command as mentioned in the docs to install version 1.2.2:

curl -sSL https://install.python-poetry.org | python3 - --version 1.2.2

neersighted commented 1 year ago

I can't tell you how you are getting an old copy of Poetry on your system, but you are, and it is taking priority in your PATH. The stack trace and paths clearly indicate you are on 1.1.x (likely shadowing the 1.2.2 install you perform).

Please take support questions to a different medium -- we'd be happy to help you there, where we aren't generating noise and notifications for a solved issue.

AXer-neil commented 1 year ago

I tried above given solutions all, but didn't work in poetry==1.2.0\~1.2.2, poetry-core=1.1\~1.2

neersighted commented 1 year ago

@AXer-neil You have not provided any details, so it is impossible to diagnose whatever you are experiencing. This issue is for build-system.requires = ['poetry-core'] and is definitively solved/does not track any other manifestation. Please reach out for support via Discord or Discussions, with a detailed explanation of what you have tried, your project files contents, what commands you have run, and full output from those commands in verbose mode (-vvv for any Poetry command).

github-actions[bot] commented 6 months ago

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

python-poetry / poetry