Does `pip-compile` take a long time to run for you?

[davidleejy]

Hello fellow pip-tool users,

I don't know if it's just me or does pip-compile take a long time to complete? I'm new to pip-tools, love the concept, but am worried that I may be running afoul of best practices.

At the moment what i'm doing repeatedly is running pip-compile -U --resolver=backtracking after editing requirements.in.

A pretty short & basic requirements.in like the following takes pip-compile 3-5 minutes:

# A pretty short & basic requirements.in
bert_score
blobfile
nltk
numpy<1.24.0
packaging
psutil
PyYAML
setuptools
spacy
torch
torchmetrics
tqdm
transformers
wandb
datasets

My intuition tells me that there's a 50% chance that i'm doing something wrong here and I would be grateful for advice 😄

[atugushev]

How much time does pip-compile take with legacy resolver? Also how fast it is without -U flag?

[IzakMaraisTAL]

pip-compile has started taking extremely long for me (killed after more than an hour).

python version: 3.7.10 pip version: 23.0 pip-compile version: 6.10.0

requirements-dev.in:

tfx[kfp]==1.12.0
tensorflow_text
tensorflow_recommenders
scann
tqdm
ujson
google-api-python-client
google-auth
google-auth-httplib2
pytest
pytest-cov
pytest-mock
requests-mock
Pillow
pre-commit

After a bit of debugging, it looks like the -U flag is to blame.

The following completes fast (just over a minute) (output_no_U.txt).

time pip-compile -v --resolver=backtracking requirements-dev.in --output-file requirements-dev.txt | ts -i "{{%H:%M:%.S}}" > output_no_U.txt

Whereas, this does not:

time pip-compile -v -U --resolver=backtracking requirements-dev.in --output-file requirements-dev.txt | ts -i "{{%H:%M:%.S}}" > output_with_U.txt

I usually kill after many minutes (sometimes more than an hour). I have tried multiple times and 9/10 times it would be extremely slow, but one time it was relatively fast at just over 3 minutes. Looking at the attached output_with_U.txt might give some insight into which steps are slow. The incremental timestamps show these steps taking > 40s each:

{{00:00:43.895765}}     Using cached google_cloud_aiplatform-1.16.1-py2.py3-none-any.whl (2.2 MB)
{{00:00:45.071425}}     Using cached google_cloud_aiplatform-1.16.0-py2.py3-none-any.whl (2.2 MB)
{{00:00:49.434197}}     Using cached google_cloud_aiplatform-1.15.1-py2.py3-none-any.whl (2.1 MB)
{{00:00:45.209635}}     Using cached google_cloud_aiplatform-1.15.0-py2.py3-none-any.whl (2.1 MB)
{{00:00:53.008086}}     Using cached google_cloud_aiplatform-1.14.0-py2.py3-none-any.whl (1.9 MB)
{{00:00:52.136853}}     Using cached google_cloud_aiplatform-1.13.1-py2.py3-none-any.whl (1.8 MB)

and some steps taking minutes:

{{00:01:51.449972}}   INFO: pip is looking at multiple versions of absl-py to determine which version is compatible with other requirements. This could take a while.
{{00:00:44.400301}}   INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. See https://pip.pypa.io/warnings/backtracking for guidance. If you want to abort this run, press Ctrl + C.

[androiddrew]

This is more of a general question about the difference in the resolvers. I noticed pip-compile --resolver=backtracing is downloading the packages while the legacy resolved is not. I assume that is a "Feature" of the backtracing resolver. I assumed it would be caching what it downloaded but then I see some of the packages being redownloaded during pip-sync.

Can someone explain to me why this is happening? Obviously it is contributing to the time I experience when using the backtracing resolver.

[Shookit]

I just wanted to echo @IzakMaraisTAL -- this is slow enough where it doesn't ever actually resolve on some of my bigger requirements files. I originally thought it might be a pip-tools 6 vs 7 issue, but I confirmed that this is happening on both versions if you explicitly specify --resolver=backtracking.

[SD-13]

I am using pip-tools 7.4.1 with python 3.11. I am running the below command, it is taking more than 30 mins and still running pip-compile --no-emit-index-url --generate-hashes requirements.in --output-file requirements.txt This is my requirements.in file. Am I missing something?

apache-beam[gcp]>2.54.0
beautifulsoup4
bleach
certifi
deepdiff
defusedxml
elasticsearch
firebase-admin
future
google-cloud-storage>2.1.0
google-api-core[grpc]>1.34.1
google-auth>=1.35.1
google-cloud-bigquery
google-cloud-dataflow-client
google-cloud-logging
google-cloud-ndb>2.2.1
google-cloud-secret-manager
google-cloud-tasks
google-cloud-translate
grpcio-status>1.48.2
gunicorn
html5lib
mailchimp3
mutagen
numpy>=1.26.4
packaging
Pillow
protobuf>3.20.3
pyarrow>=12.0.0
pylatexenc
pytest
PyYAML>=6.0.1
redis>=5.0.0
requests
requests-mock
requests-toolbelt
result
rsa
simplejson
six
soupsieve
typing-extensions
urllib3
webapp2
webencodings

[IzakMaraisTAL]

I don't mean to disparage the pip-tools project, but for people struggling with this specific problem, you might try uv. I tested with uv pip-compile and it resolved the same dependencies in reasonable time.