zanieb
commented
9 months ago Interesting, thanks for the details.
echo "apache-airflow[all]" | uv pip compile - --only-binary apache-beam resolves
Closed notatallshaw closed 7 months ago
zanieb
commented
9 months ago Interesting, thanks for the details.
echo "apache-airflow[all]" | uv pip compile - --only-binary apache-beam resolves
zanieb
commented
9 months ago Related discussion of resolver heuristics in https://github.com/astral-sh/uv/issues/1398
astrojuanlu
commented
9 months ago When uv can resolve apache-airflow[all] you can declare 1.0 already 😄
notatallshaw
commented
9 months ago Related discussion of resolver heuristics in #1398
I was distinguishing the two based on that was just "slow" where as this ends in a failure.
Although in both cases the high level issue is uv backtracks too far on a transitive dependency, the symptom, and possibly solution, are different.
Any guidance on how to file resolution issues in the future that would best fit the uv teams workflow would be appreciated. I still have a lot more scenarios to try out from real world issues reported to Pip to see if uv can handle them.
zanieb
commented
9 months ago Feel free to open a new issue for each case. I'm cross-linking them for my own sake, but we also have a label.
The absolute best thing one can do is write scenarios reproducing the problem so we can a clear statement of the problem and test coverage for the solution.
notatallshaw
commented
9 months ago The absolute best thing one can do is write scenarios reproducing the problem so we can a clear statement of the problem and test coverage for the solution.
Yeah, I am starting to take a look at packse as I would like to add it to the Pip test suite (https://github.com/pypa/pip/issues/12526).
However it seems like one has to manually build scenarios? Compared to say pip-resolver-benchmarks which can build a scenario automatically (though the JSONs are huge). But I'll take that up on the packse repo.
notatallshaw
commented
9 months ago
echo "apache-airflow[all]" | uv pip compile - --only-binary apache-beamresolves
I just assumed this was true, but I tried it right now and it didn't resolve for me:
$ echo "apache-airflow[all]" | uv pip compile - --only-binary apache-beam
error: Failed to download and build: apache-beam==2.2.0
Caused by: Building source distributions is disabledIs this a seperate issue? This resolves fine:
$ echo -e "apache-airflow[all]\napache-beam>=2.47" | uv pip compile -
potiuk
commented
9 months ago Hey - Airlfow maintainer here and the "The CI/dev tool Airflow guy",
Apache Beam is notroriusly complex to get right. This is by far biggest problem contibutor to airflow (including the fact that it's the only provider we have to disable for upcoming Python 3.12 support.
The problem you likely have there is - I guess - that the old version of apache-beam has not enough metadata and you want to install it to get the metadata and this really really old version which does not have a binary wheel at all https://pypi.org/project/apache-beam/2.2.0/#files and fails the compilation - both cases should likely simply lead to skipping that version entirely in your resolution process.
BTW. It's a little tangential @zanieb and other maintainers, But speaking of airflow - you might not be aware - but I just merged https://github.com/apache/airflow/pull/37692 where we use uv to cut down the time for our CI image building and updating by 50% - 60% thanks to the speed of uv. Thanks for such a great tool (we adopted uv for that even faster than ruff, and looking so much forward to using more of the features there (resolution --lowest is particularly interesting as we almost completed our own poor version of it and found out that uv can do it for us).
We are not yet switching for production - kind of usage to uv, not recommend it to our users (yet) - but I am quite impressed so far with the speed improvements especially and definitely will keep an aye and adopt (and maybe even some day contribute) to packaging tooling out there - finally it seems that the packaging tooling is getting close to provide a lot of the things we've been doing in custom ways in Airflow (because of our size and setup and complexity we had to do a lot on our own) - and we can slowly replace and improve pieces of our CI / development tooling with more standard solution (I love when I can do it with confidence).
You can see mailing discushttps://lists.apache.org/thread/8gyzqc1d1lxs5z7g9h2lh2lcoksy2xf9
BTW. I will be at Pycon US in Pittsbuirgh and how to meet a lot of the packaging team and astral team there! Signed up to packaging summit and hope to see you and talk to you there.
charliermarsh
commented
9 months ago @potiuk - Thanks so much for chiming in and for the kind words -- that PR is so so cool! I'll be at PyCon US too and also signed up for the packaging summit.
notatallshaw
commented
9 months ago I just assumed this was true, but I tried it right now and it didn't resolve for me:
$ echo "apache-airflow[all]" | uv pip compile - --only-binary apache-beam error: Failed to download and build: apache-beam==2.2.0 Caused by: Building source distributions is disabled
FYI I created a seperate issue for this: https://github.com/astral-sh/uv/issues/2003
potiuk
commented
9 months ago FYI. My non-scientific comparision (after > 24 hrs switching to uv) is that Airflow's workflow is getting a HUGE boost.
We use our CI image also to make sure our breeze development environment for reproducing CI tests and running them locally. And we get a 70% boost for a complete rebuild from the scratch with uv - at least on multi-core machines with fast network.
In my case I got the worst case full rebuilt, with upgrade to latest dependencies and disabling docker cache altogether down from 12 minutes with pip to 4 minutes with uv. It's literally a game-changer for our developmetnt environment. Maybe even (if I get some more datapoints) I might be able to make things simpler and remove some of the optimizations I've implemented to specifically target the slow resolution and installation from pip.
https://lists.apache.org/thread/sq70ch6lllryv4cr5q0xjt6b9z5n0vd8
Thanks again for this one. I hope my "First time contributor's workshop" for Pycon will get accepted in the hatchery and this will make lives of those "first-time contributors to be" hell a lot easier.
notatallshaw
commented
9 months ago I've implemented to specifically target the slow resolution and installation from
pip.
Btw, do you know if any of those optimizations are still required for pip? Or if you've had any specific resolution issues in last ~6 months?
I know uv is a lot faster, but I only have a handful of examples where it resolves "better" (i.e. visits less packages to resolve), and I have a PR on pip side which fixes all of those. So any more examples would appreciated .
And of course this issue is an example with pip chooses a better resolution path than uv.
potiuk
commented
9 months ago Btw, do you know if any of those optimizations are still required for pip? Or if you've had any specific resolution issues in last ~6 months?
It's mostly for the design of caching by Docker layers and addressing several cases - when users in CI upgrade only dependencies, or when they upgrade to conflicitng dependencie - not the resolution itself.
What is left from the "resolution helper" is the list of dependencies i add to address the --eager upgrade https://github.com/apache/airflow/blob/main/Dockerfile.ci#L1289 - currently
# Those are additional constraints that are needed for some extras but we do not want to
# force them on the main Airflow package. Currently we need no extra limits as PIP 23.1+ has much better
# dependency resolution and we do not need to limit the versions of the dependencies
#
# boto3 is limited to <1.34 because of aiobotocore that only works with 1.33 and we want to help
# `pip` to limit the versions it checks and limit backtracking, by explicitly specifying these limits
# when performing eager upgrade of dependencies - this way it won't even consider 1.34 versions of boto
# We should update it every time a new version of aiobotocore is released supporting 1.34
#
ARG EAGER_UPGRADE_ADDITIONAL_REQUIREMENTS="boto3>=1.33,<1.34"That one however was based on pure guesses - whenever pip went into a long backtracking I tried to guess (by looking into what has been released since last successful resolution) and figured a way to add extra expectations to --eager-upgrade to limit a bit space of solutions to help with the imperfect backtacking algorithm. This list here changed over time and it's largely based on guesses and bi-secting rather than some science. I am going to remove it now and see where it will get us.
potiuk
commented
9 months ago PR running here to remove it https://github.com/apache/airflow/pull/37745
notatallshaw
commented
9 months ago What is left from the "resolution helper" is the list of dependencies i add to address the
--eager upgradehttps://github.com/apache/airflow/blob/main/Dockerfile.ci#L1289 - currently
Well uv can certainly suffer from the same issues: https://github.com/astral-sh/uv/issues/1398
It will be interesting to see if uv is performant enough for you when it also goes backtracking in the wild.
potiuk
commented
9 months ago It will be interesting to see if uv is performant enough for you when it also goes backtracking in the wild.
Hard to say - those extra requirements tend to solve itself over time. - I periodically removed them and added new ones when we got into bactracking issue, also what (obviously) helps is regular bumping of lower limits in some dependencies.
This change is fine (I had to handle edge case where extra requirements are empty) - the changes generated by removal of those extra requirements works fine:
< boto3==1.33.13
< botocore==1.33.13
---
> boto3==1.34.34
> botocore==1.34.34
498c498
< s3transfer==0.8.2
---
> s3transfer==0.10.0
505c505
< sentry-sdk==1.40.5
---
> sentry-sdk==1.40.6No excessive backtracking (the images were built in 3 minutes).
konstin
commented
9 months ago For context, boto3/botocore are notoriously hard due to their large number of releases, where we have to backtrack through every single one: https://pypi.org/project/boto3/#history, https://pypi.org/project/botocore/#history. We're planning on improving the situation around boto specifically.
potiuk
commented
9 months ago For context, boto3/botocore are notoriously hard due to their large number of releases, where we have to backtrack through every single one: https://pypi.org/project/boto3/#history, https://pypi.org/project/botocore/#history. We're planning on improving the situation around boto specifically.
Oh absolutely - approach of boto3/botocore is particularly difficult for package managers and resolution. When backtracking happens the first thing I do is trying botocore/boto limitation.
notatallshaw
commented
7 months ago FYI, the error output is different now, but the issue of uv backtracking too far back on apache-beam for Python 3.11 with latest version of airflow still exists as of today:
$ uv pip install --dry-run "apache-airflow[all]"
error: Failed to download and build: apache-beam==2.2.0
Caused by: Failed to build: apache-beam==2.2.0
Caused by: Build backend failed to determine extra requires with `build_wheel()` with exit status: 1
--- stdout:
--- stderr:
<string>:24: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
Traceback (most recent call last):
File "<string>", line 14, in <module>
File "/home/dshaw/.cache/uv/.tmpvHTOBL/.venv/lib/python3.11/site-packages/setuptools/build_meta.py", line 325, in get_requires_for_build_wheel
return self._get_build_requires(config_settings, requirements=['wheel'])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/dshaw/.cache/uv/.tmpvHTOBL/.venv/lib/python3.11/site-packages/setuptools/build_meta.py", line 295, in _get_build_requires
self.run_setup()
File "/home/dshaw/.cache/uv/.tmpvHTOBL/.venv/lib/python3.11/site-packages/setuptools/build_meta.py", line 487, in run_setup
super().run_setup(setup_script=setup_script)
File "/home/dshaw/.cache/uv/.tmpvHTOBL/.venv/lib/python3.11/site-packages/setuptools/build_meta.py", line 311, in run_setup
exec(code, locals())
File "<string>", line 61, in <module>
File "/home/dshaw/.cache/uv/.tmpvHTOBL/.venv/lib/python3.11/site-packages/pkg_resources/__init__.py", line 497, in get_distribution
dist = get_provider(dist)
^^^^^^^^^^^^^^^^^^
File "/home/dshaw/.cache/uv/.tmpvHTOBL/.venv/lib/python3.11/site-packages/pkg_resources/__init__.py", line 384, in get_provider
return working_set.find(moduleOrReq) or require(str(moduleOrReq))[0]
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/dshaw/.cache/uv/.tmpvHTOBL/.venv/lib/python3.11/site-packages/pkg_resources/__init__.py", line 937, in require
needed = self.resolve(parse_requirements(requirements))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/dshaw/.cache/uv/.tmpvHTOBL/.venv/lib/python3.11/site-packages/pkg_resources/__init__.py", line 798, in resolve
dist = self._resolve_dist(
^^^^^^^^^^^^^^^^^^^
File "/home/dshaw/.cache/uv/.tmpvHTOBL/.venv/lib/python3.11/site-packages/pkg_resources/__init__.py", line 839, in _resolve_dist
raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'pip' distribution was not found and is required by the application
---I don't know if pubgrub-rs is felxible enough, but I still strongly suggest that when backtracking and choosing between two packages, try as much as possible to avoid choosing one that involves having to compile sdists. I recently further discussed (https://github.com/pypa/pip/issues/12035) this idea on the pip side and plan to eventually create a PR for pip.
potiuk
commented
7 months ago This is somewhat related to what I reported today in https://github.com/astral-sh/uv/issues/2821 - similarly like in the other issue - backtracking is not only considering some really old versions of dependencies (apache-beam 2.2.0 has been released in 2017 (!) but also installation fails in case those considered candidates have some serious issues with metadata.
I think while the "failing on non-installable candidate" has an easy solution, indeed some extra heuristics for candidate selection should speed the resolution even further. Hard to say what heuristics though.
I also think there is something wrong with the current candidate selection. Currently (and likely for quite some time) - we limit apache-beam in apache-airflow-provider-apche-beam (this is the packge that brings in apache-beam as dependency) to apache-beam>=2.53.0 - and the "no-limit" beam comes likely from some older versions of the provider ( I just checked that originally apache-airflow-providers-apache-beam==1.0.0 had no lower binding.
We could potentially - of course - in the future versions of airflow to say for example apache-airflow-providers-apache-beam>3.0.0 - we could come up with some policies of adding lower-bounds for providers (we do not do it for now), however naturally - I think cutting of some "older" versions of dependencies when newer versions exists could be a way to go for such heuristics.
Of course I know it's an NP-complete problem and choosing some heuristics will cause problems for some other cases - and I do not know the details of how uv resolution is done, so maybe I am stating the obvious, but possibly a good solution could be to figure out a few such "likely" heuristics that will limit big part of the tree resolution and gradually remove such "stronger" heuristics when resolution could not be found with them - with the speed of UV resolution, even if you try several times, with weaker and weaker heuristics it could be fast enough (and might help in 9X% of cases that will work super-fast and reliable with the stronger heuristics.
Just a "lame" proposal - without knowing the internals - it might be - again - stating the obvious thing that already happens (or maybe I am missing somethign that I am not aware of), so apologies if that's the case :)
konstin
commented
7 months ago Our pacakage selection heuristic is currently just going through packages in order we first see them in the requirements of another package - There's definitely room for improvement.
We could potentially - of course - in the future versions of airflow to say for example apache-airflow-providers-apache-beam>3.0.0 - we could come up with some policies of adding lower-bounds for providers (we do not do it for now), however naturally - I think cutting of some "older" versions of dependencies when newer versions exists could be a way to go for such heuristics.
Lower bounds would be very helpful! My heuristic is that the lower bound should be the lowest version that still works (passes tests with --resolution lowest or --resolution lowest-direct) or that you would still support with airflow. Otherwise there's always the risk that we start backtracking for the wrong package and end up with an ancient version that's technically "compatible" because it doesn't use any deps that conflict with what is used now or that we run into crashing builds as we do now; there's no good way to guarantee we don't end up in the bad path otherwise. It also helps resolver performance because we cut a large part of the search space.
The conflicts we encounter are often of a specific shape: Say we have two dependencies a and b. We see that for the latest versions, a==5 and b==5, a==5 wants c==2 and d==3 while b==5 wants c==3 and d==2. We can start iterating either over a's or over b's past releases. With a lower bound on a, say a>=4, we try few versions until we determine that no version of a works and we have to reject b==5 instead.
I've confirmed that we resolve apache-airflow to 2.9.0 (ubuntu, python 3.11) with:
apache-airflow[all]
apache-airflow-providers-apache-beam>3.0.0Our pacakage selection heuristic is currently just going through packages in order we first see them in the requirements of another package - There's definitely room for improvement.
Is there a reason you haven't taken pip's approach and prioritize certain heuristics?
It's probably why pip can resolve this requirement and uv can not, the relevant heauristics here would be direct, pinned, and inferred depth.
My heuristic is that the lower bound should be the lowest version that still works
The problem is that a library must support multiple versions of Python, let's say the lower bound for dependency foo is 10 because the library supports Python 3.8 to 3.12, but foo has C extensions and version 10 was releases when 3.10 and higher didn't exist, foo agressively drops old versions of Python and version 11 dropped Python 3.8 support. My library can now only put a lower bound on foo of 10 because I do support Python 3.8, even though Pythons 3.10 to 3.12 are not compatible with this old version of foo.
Your heuristic is fine, but for library authors supporting wide ranges of Python it doesn't actually help that much.
I've confirmed that we resolve apache-airflow to 2.9.0 (ubuntu, python 3.11) with:
Yes, or you can do this https://github.com/astral-sh/uv/issues/1560#issuecomment-1949626187. That's why I titled this a performance issue, not a bug with uv, uv does not do as good of a job limiting the number of candidates it is checking against compared to pip.
konstin
commented
7 months ago I think the problem and it's solution are sufficiently understood to close this issue
@potiuk You're enforcing a lower version bound on apache-airflow-providers-apache-beam in https://github.com/apache/airflow/blob/5fa80b6aea60f93cdada66f160e2b54f723865ca/airflow/providers/apache/beam/__init__.py#L37-L42, if you move that to the package metadata you should be able to drop the check there. I unfortunately do understand enough of your build system to change that.
@notatallshaw Closing in favor of #1398.
Please feel free to open a new issue issue if other performance (or any other kind of ) problems with apache airflow should arise.
notatallshaw
commented
7 months ago @konstin sorry I don't understand why you've closed this issue:
Unless I'm missing something, such as some root cause analysis, can you please reopen this ticket.
charliermarsh
commented
7 months ago I don't know if pubgrub-rs is felxible enough, but I still strongly suggest that when backtracking and choosing between two packages, try as much as possible to avoid choosing one that involves having to compile sdists.
Haven't been paying close attention to this issue but this heuristic has some downsides. For one, the output resolution for a given set of inputs could change despite no changes in the available versions on PyPI, just by way of wheels being uploaded for existing versions.
notatallshaw
commented
7 months ago Haven't been paying close attention to this issue but this heuristic has some downsides. For one, the output resolution for a given set of inputs could change despite no changes in the available versions on PyPI, just by way of wheels being uploaded for existing versions.
I don't think so, except in the case where it is a transitive dependency that is optional, in the sense a solution can be found without it. In which case, how significantly negative is this?
But I think uv should first try pip's priorities (https://github.com/astral-sh/uv/issues/1560#issuecomment-2052673990), or similar, they seem to be strong enough here to find a solution while avoiding old versions of apache-beam, at least for pip which can install this requirement without issue.
charliermarsh
commented
7 months ago I mean, it's definitely true that changing the order in which you visit packages will change the output resolution in some cases. So whenever you change the prioritization, you will change the output resolution in some cases. And my point here is that you're now changing the resolution based on subtle properties that don't map to how users think about their requirements.
Regardless, we should try out some of pip's priorities, would be happy to see a separate issue for that.
notatallshaw
commented
7 months ago I mean, it's definitely true that changing the order in which you visit packages will change the output resolution in some cases.
That's true, but I think the word "some" is doing a lot of heavy lifting. I think you would be hard pressed to find a real world example (though I'm sure one could easily artificially construct an example) where it would impact resolution solution (beyond allowing it to actually find a solution in cases like this). But I understand if you consider the chance of this weird behavior appearing in unusual edge cases to be an extremely negative property of a resolution algorithm.
Regardless, we should try out some of pip's priorities, would be happy to see a separate issue for that.
I am a long way off sufficient Rust knowledge to contribute to these projects, probably for a year or two, otherwise I would have been happy to make a PR to try this out.
Regardless though of the solution, it remains my original issue as posted is still somewhere that uv fails to resolve and pip does not. #1398 was a wall clock performance issue, that didn't cause any failures, and that was solved. This issue is a performance of uv's resolution in the sense it visits too old packages (for some sense of old that means won't compile) which causes real world failures, and is not solved.
notatallshaw
commented
7 months ago There seems to be some communication issue here:
As such I've opened this as a new issue that is much more focused on the problem: https://github.com/astral-sh/uv/issues/3078
charliermarsh
commented
7 months ago I don't know why @konstin closed the issue. I'll just re-open until we can resolve this case. My prior comment was only meant to signal that I thought the idea of using different rules for prioritization was sensible.
charliermarsh
commented
7 months ago Sorry, now I feel like I'm doing the wrong thing by re-opening given the new issue, so I'll re-close and we'll continue from #3078.
notatallshaw
commented
7 months ago I don't mind which issue is kept open, I just don't want this to drop off as a known issue. This issue was the original but the other is way more focused and so doesn't have the baggage of this thread ¯\(ツ)/¯.
konstin
commented
7 months ago Thanks, #3078 is much more actionable
This is for Linux Python 3.11.6 with uv 0.1.3:
The exception from building apache-beam==2.2.0 is fine, the issue is that uv is backtracking all the way to apache-beam 2.2.0, which is too old to compile in my Python 3.11 environment
Pip does not suffer this issue and would install apache-beam-2.54.0:
Interestingly
riphas a very similiar issue: https://github.com/prefix-dev/rip/issues/174.I have speculated there that a good heuristic when resolving is to first resolve requirements that have wheels over sdists, or at least when a requirement has newer package versions with wheels and older package versions with sdists to prefer other requirements once the all wheels (metadata) have been collected by the resolution process and further collection (of metadata) requires building sdists.
I have suggested the same thing for Pip (and recently developed a loose idea of how to actually implement it), but I haven't tried doing it yet: https://github.com/pypa/pip/issues/12035 (it's at the bottom of my list of things to try and improve about performance resolution in Pip).
You of course may have very different ideas on how to solve this issue! I would be interested to follow any improvements you make here.