Closed cjolowicz closed 2 years ago
Linking @sobolevn's comment about this problem on another Poetry-related issue: https://github.com/dependabot/dependabot-core/issues/1556#issuecomment-604382270
...
- Every dependency update has merge conflicts. Because of the
[metadata].content-hash
field: https://github.com/wemake-services/wemake-python-styleguide/pull/1287/files#diff-41fe8bebc1a2a52eb5321b759e40b3a8R1627 Now all merge must be done like: merge first -> rebase second -> merge second. I guess it is a problem withpoetry
. Here's the upstream issue: auto-resolve all/most merge conflicts python-poetry/poetry#496
While I agree that Poetry could be better at avoiding merge conflicts, the issue here is that Dependabot triggers the conflicts even when there is no need for it, because the version constraints don't need to be changed.
It should be noted that the examples in the issue description (cookiecutter, reorder-python-imports) are development dependencies.
The case is different for core dependencies. AFAIU Dependabot attempts to preserve constraints for core dependencies, so merge conflicts would only happen with those on major version bumps, when the constraint is widened.
Also worth noting that the problem does not exist with indirect dependencies, because by definition those have no version constraints in pyproject.toml.
@cjolowicz not sure if I understand entirely, but it seems like you might want to configure dependabot to only update the lockfile, or use the increase-if-necessary
versioning strategy?
Hi @jurre,
not sure if I understand entirely, but it seems like you might want to configure dependabot to only update the lockfile, or use the
increase-if-necessary
versioning strategy?
The drawback of versioning-strategy: lockfile-only
is that we no longer receive automated updates for major releases. That pretty much makes this option not viable, except for projects where we're comfortable to remove upper version bounds from all dependencies.
The increase-if-necessary
strategy would be perfect, but I understand the documentation as saying it's not supported for Python?
The drawback of versioning-strategy: lockfile-only is that we no longer receive automated updates for major releases.
Yeah, that's fair.
it's not supported for Python?
Ah apologies, you're right. I think for Poetry it could be supported but we'll need to make some changes to dependabot-core for it to work.
I think Dependabot is behaving as expected here, but we should add support for increase-if-necessary
for Poetry. I've changed the labels to reflect this, and will try to see if I can get it on the teams backlog. We have a lot of thing going on at the minute so I can't really speak to when we'd be able to get to it (or even guide a contribution that adds support for it the right way).
I think Dependabot is behaving as expected here, but we should add support for
increase-if-necessary
for Poetry. I've changed the labels to reflect this, and will try to see if I can get it on the teams backlog. We have a lot of thing going on at the minute so I can't really speak to when we'd be able to get to it (or even guide a contribution that adds support for it the right way).
Totally understand, and appreciate the quick response. 👍
FWIW this feature should translate into a considerable cut in GA's electricity bill. Maybe that helps with getting this scheduled 😉
To reiterate: The default settings for Poetry projects result in cascading rebases of Dependabot PRs, triggering 1+2+...+n CI runs for N dependency updates.
Support for increase-if-necessary
versioning strategy in Python will be added by #5605, so we'll be able to get rid of this problem 😄.
Thanks for your patience!
Dependabot updates version constraints in Poetry projects even when they already cover the new version (see example below).
This behavior causes merge conflicts with every other Dependabot PR updating the
pyproject.toml
file. The merge conflicts happen because Poetry computes a hash over the version constraints in pyproject.toml (and some other things), and embeds the resulting hash in its lockfile (metadata.content-hash
in thepoetry.lock
file). This means that changes to version constraints always conflict with each other.This update strategy results in considerable developer churn and energy consumption, as Dependabot PRs are rebased many times before merging, triggering CI runs. For example, given a batch of 5 Dependabot PRs, CI will be triggered up to 15 times (1+2+3+4+5).
Package ecosystem
Package manager version
Language version
Manifest location and content prior to update
dependabot.yml content
Updated dependency
What you expected to see, versus what you actually saw
^1.7.2
to^1.7.3
Native package manager behavior
Diff after running `poetry update cookiecutter` on the base revision:
```diff diff --git a/poetry.lock b/poetry.lock index 6b91fc2..998c7f3 100644 --- a/poetry.lock +++ b/poetry.lock @@ -221,7 +221,7 @@ test = ["flake8 (==3.7.8)", "hypothesis (==3.55.3)"] [[package]] name = "cookiecutter" -version = "1.7.2" +version = "1.7.3" description = "A command-line utility that creates projects from project templates, e.g. creating a Python package project from a Python package project template." category = "dev" optional = false @@ -230,9 +230,8 @@ python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*" [package.dependencies] binaryornot = ">=0.4.4" click = ">=7.0" -Jinja2 = "<3.0.0" +Jinja2 = ">=2.7,<4.0.0" jinja2-time = ">=0.2.0" -MarkupSafe = "<2.0.0" poyo = ">=0.5.0" python-slugify = ">=4.0.0" requests = ">=2.23.0" @@ -1378,8 +1377,8 @@ commonmark = [ {file = "commonmark-0.9.1.tar.gz", hash = "sha256:452f9dc859be7f06631ddcb328b6919c67984aca654e5fefb3914d54691aed60"}, ] cookiecutter = [ - {file = "cookiecutter-1.7.2-py2.py3-none-any.whl", hash = "sha256:430eb882d028afb6102c084bab6cf41f6559a77ce9b18dc6802e3bc0cc5f4a30"}, - {file = "cookiecutter-1.7.2.tar.gz", hash = "sha256:efb6b2d4780feda8908a873e38f0e61778c23f6a2ea58215723bcceb5b515dac"}, + {file = "cookiecutter-1.7.3-py2.py3-none-any.whl", hash = "sha256:f8671531fa96ab14339d0c59b4f662a4f12a2ecacd94a0f70a3500843da588e2"}, + {file = "cookiecutter-1.7.3.tar.gz", hash = "sha256:6b9a4d72882e243be077a7397d0f1f76fe66cf3df91f3115dbb5330e214fa457"}, ] coverage = [ {file = "coverage-6.1.2-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:675adb3b3380967806b3cbb9c5b00ceb29b1c472692100a338730c1d3e59c8b9"}, ```Images of the diff or a link to the PR, issue or logs
Log for updating cookiecutter: not available
Log for reorder-python-imports (excerpt):
Full log for Dependabot update
``` proxy | time="2021-11-21T07:51:25Z" level=info msg="proxy starting" commit=d5f262668736016da1a91e42cb4fba36a081bddf proxy | 2021/11/21 07:51:25 Listening (:1080) updater | 2021-11-21T07:51:26.134892342 [anonymous-instance:main:WARN:src/firecracker/src/main.rs:370] You are using a deprecated parameter: --seccomp-level 2, that will be removed in a future version. updater | 2021-11-21T07:51:26.154285753 [235098818:main:WARN:src/devices/src/legacy/serial.rs:432] Detached the serial input due to peer close/error. updater | time="2021-11-21T07:51:27Z" level=info msg="guest starting" commit=aca5609d0a3f160d4065a3c391d9a0e3c36b620d updater | time="2021-11-21T07:51:27Z" level=info msg="starting job..." fetcher_timeout=5m0s job_id=235098818 updater_timeout=45m0s updater_version=0.167.0-e7ac8e25ca27aeb1f1ca9b86f06b2771984f7451 updater | I, [2021-11-21T07:51:29.012284 #8] INFO -- sentry: ** [Raven] Raven 3.1.2 ready to catch errors updater | warning: parser/current is loading parser/ruby27, which recognizes updater | warning: 2.7.4-compliant syntax, but you are running 2.7.1. updater | warning: please see https://github.com/whitequark/parser#compatibility-with-ruby-mri. updater | INFO🕹 Bonus points: Smallest manifest that reproduces the issue