Unexpected inability to resolve dependencies depending on state of requirements.txt file

[davidhao3300]

Given the following file setup:

requirement.in:
httpx==0.24.1,>=0.23.3
pytest-httpx>=0.22.0

pip-compile generates

requirement.txt:
[others omitted]
httpx==0.24.1
pytest-httpx==0.24.0

But if there is an existing, possibly-stale requirements.txt file, like:

httpx==0.25.0
pytest-httpx==0.26.0

pip-compile will throw an error:

  ERROR: Cannot install -r requirements.in (line 2) and httpx==0.24.1 and >=0.23.3 because these package versions have conflicting dependencies.
...
pip._vendor.resolvelib.resolvers.ResolutionImpossible: [RequirementInformation(requirement=SpecifierRequirement('httpx==0.24.1,>=0.23.
3'), parent=None), RequirementInformation(requirement=SpecifierRequirement('httpx==0.25.*'), parent=LinkCandidate('https://files.pytho/
nhosted.org/packages/32/6d/f0a2334a5a0620e9eda69cb7a05754f51e0ab0ff678c87ecdece6eefcfab/pytest_httpx-0.26.0-py3-none-any.whl (from htt
ps://pypi.org/simple/pytest-httpx/) (requires-python:>=3.9)'))]

Removing the pytest-httpx line in the requirements.txt file allows pip-compile to complete successfully. Ideally manual patching of the existing requirements.txt is not needed.

[webknjaz]

Try the upgrade mode?

[davidhao3300]

I will give this a try on Tuesday, I suspect it will work, but it will probably upgrade a bunch of dependencies that we'd rather not upgrade if possible.

Will let you know!

[webknjaz]

You can pass a specific package to that arg.

[davidhao3300]

I'll do some more playing around given these suggestions, thanks. I suspect that for our usecase, working with --upgrade will be unnatural.

When generating a new requirements.txt, we generate an ephemeral requirements.in and compile alongside the the previous version of the requirements.txt. We'd like to make a "minimal" set of version changes across packages (although minimal is a nebulous quality), even if it involves downgrading packages.

We could work around this by doing a diff of the packages being added/removed/modified in the requirements.in file, but I suspect it'll be errorprone/not-correct, as there could be intermediate dependencies that hit this issue as well

I guess I left out a detail that makes the existing behavior interesting: If you have the following set:

requirements.in
requirement.in:
httpx==0.24.1,>=0.23.3
pytest-httpx>=0.22.0

requirements.txt
httpx==0.25.0

pip-compile also happily generates

requirement.txt:
[others omitted]
httpx==0.24.1
pytest-httpx==0.24.0

It's happy to downgrade httpx from 0.25.0 to 0.24.1 in this case

[tboddyspargo]

I would agree that specifying --upgrade seems unnatural and I would expect it to be unnecessary.

I also wanted to point out some potential similarities with #1818. I believe both cases involve the resolver not discarding the existing lockfile version even when it should have enough information to know that it is now invalid. The

Should we expect pip-compile to error rather than discard existing lockfile versions if they are incompatible with the input constraints? IMO, it should discard and resolve, rather than error, but there may be good reasons to error which I'm not considering.

If the exception is preferrable, then I would expect the exception to indicate that the impossible constraints were httpx==0.24.1 and httpx==0.25.0 (and probably also indicate the origin of httpx==025.0).