1.2 performance regression

ipmb commented 2 years ago

[x] I have searched the issues of this repo and believe that this is not a duplicate.
[x] I have searched the documentation and believe that my question is not covered.

Issue

I mentioned this in Discord, but didn't want it to get lost in the shuffle. I put together a Python package manager benchmark. I got a suggestion to add the current prerelease to the test suite.

Preliminary results showed the same operations taking ~2x the time vs. the non prerelease version. You can see results here: https://github.com/lincolnloop/python-package-manager-shootout/actions/runs/2705531257

Compare elapsed time between poetry and poetry-pre:

dimbleby commented 2 years ago

I suspect you are mostly measuring variability in download speed.

I tweaked your Makefile so that poetry-import does poetry add --dry-run so that it measures only the locking time. (Installing seems to require source builds of some dependencies I don't have and don't care about).

with nothing in cache, so that poetry has to download lots of stuff during locking, this takes about 2 minutes for me on both 1.1.14 and current master
but with the poetry cache fully populated the locking takes about 5 seconds, for both versions

ie what is being measured here is absolutely dominated by time spent downloading from pypi. Also notable is that I see approximately the same timings for both versions.

du -sh ~/.cache/pypoetry says that both versions put about 21M of stuff in the cache, so it doesn't seem as though the amount of downloading has changed.

I suspect you are simply seeing variation in download speed.

ipmb commented 2 years ago

The results are repeatable, so I don't think this is a one-off network blip. Here is another run I just did:

The operations being run between these two are identical with the exception of one being installed with --pre.

I've run a bunch of these benchmarks and while there is some variability between them, it is nowhere near what I'm seeing here. You can see the typical variability in the min/max values here across 4 runs:

dimbleby commented 2 years ago

do you agree that what you are mostly measuring is network download speed? eg do you, as I do, see the poetry lock time go to <5s if you have a populated cache?

if yes then it's hard to see how to interpret your timings of 80s / 175s as anything other than measuring download speed. Perhaps you have flipped a coin twice and hit heads twice; perhaps you are consistently being throttled at the same point in your run, who knows.

ipmb commented 2 years ago

I'm not trying to interpret the results, just sharing the data.

poetry 1.2.0b3 is ~2x slower than poetry 1.1.14 consistently (I've seen this more than twice) for the exact same operations.

I'll merge the PR to test this version into the regularly scheduled tests and you can see if the results improve over time.

dimbleby commented 2 years ago

per my earlier comment, that doesn't match my attempts to reproduce - in which the two versions consistently perform about the same.

ipmb commented 2 years ago

I saw 1.2 was released and still contains the performance regression (perhaps due to #6060). The 1.1 release will drop off our testing matrix, but you can still see the historical comparison here: https://htmlpreview.github.io/?https://github.com/lincolnloop/python-package-manager-shootout/blob/e1c48f257c0e4104a9c5e3ed1b219ba5a028d0b2/index.html

These are the results for installation:

install performance

michaeldimchuk commented 2 years ago

I'm also seeing this after upgrading to 1.2, running poetry in Gitlab pipelines. The exact same dependencies are taking over twice as long to install when comparing to 1.1.4 consistently.

vlcinsky commented 2 years ago

I made repeated tests of installing my project using poetry v1.1.15 and v1.2 The version 1.2 is almost twice as slow

action	v1.1.5	v1.2
update dependencies	0.3s	0.8s
clean install	13.5s	29s

OS: Debian 11 python: 3.10.0

Both tests were done repeatedly and only latest times are shown.

My pyproject.toml:

[tool.poetry]
name = "orgdioformat"
version = "0.1.0"
description = ""
authors = ["Jan Vlcinsky <jan.vlcinsky@tamtamresearch.com>"]

[tool.poetry.dependencies]
python = "^3.10"
fastapi = "^0.79.0"
uvicorn = "^0.18.2"
invoke = "^1.7.1"
requests = "^2.28.1"

[tool.poetry.dev-dependencies]
pytest = "^5.2"

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

Note: New documentation describes how can one install multiple versions of poetry using pipx concurrently. This makes testing much simpler.

rzuckerm commented 2 years ago

I have also noticed that poetry 1.2.0 is significantly slower that 1.1.15 and have downgraded poetry in our renovate docker image that we use to update dependencies in our projects. The integration tests for this dependency system went from about 10 minutes to 30. When we downgraded poetry to 1.1.15, the integration tests went back to 10 minutes.

radoering commented 2 years ago

@ipmb @vlcinsky Can you try if the current 1.2 branch performs better? You can install via

pipx install --suffix 12 git+https://github.com/python-poetry/poetry.git#1.2

vlcinsky commented 2 years ago

@radoering with the current poetry branch the total installation times got on par with the poetry v1.1.15:

dependency resolution stays at 0.8s
remaining installation time improved significantly or is even a bit faster than poetry v1.1.15

action	v1.1.5	v1.2	branch12
update dependencies	0.3s	0.8s	0.8s
clean install	13.5s	29s	13.s

Anyway, when I run the test with the poetry from branch 1.2, the second attempt got stuck and I had to kill it after 2 minutes. This happened only once, later attempts to repeat the installation did not fail.

Note: by "clean install" I mean

clean the virtualenv and poetry.lock file first $ rm -rf .venv *.lock
and then run the $ poetry install

rushilsrivastava commented 2 years ago

I have also noticed that poetry 1.2.0 is significantly slower that 1.1.15 and have downgraded poetry in our renovate docker image that we use to update dependencies in our projects. The integration tests for this dependency system went from about 10 minutes to 30. When we downgraded poetry to 1.1.15, the integration tests went back to 10 minutes.

We have had a similar experience, it seems like the best course of action is to use the latest branch build as per @vlcinsky comment.

mkniewallner commented 2 years ago

I have also noticed that poetry 1.2.0 is significantly slower that 1.1.15 and have downgraded poetry in our renovate docker image that we use to update dependencies in our projects. The integration tests for this dependency system went from about 10 minutes to 30. When we downgraded poetry to 1.1.15, the integration tests went back to 10 minutes.

We have had a similar experience, it seems like the best course of action is to use the latest branch build as per @vlcinsky comment.

Poetry 1.2.1 has been released, so you can use it instead of 1.2 branch now.

neersighted commented 2 years ago

Closing as 1.2.1 is released.

vlcinsky commented 2 years ago

Yes. I can confirm that with the poetry v1.2.1 the clean installation time is now on par with poetry v1.1.5

Well done (@radoering?). Thanks.

github-actions[bot] commented 8 months ago

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

python-poetry / poetry

1.2 performance regression #6047

Issue