Open davidleejy opened 1 year ago
How much time does pip-compile
take with legacy resolver? Also how fast it is without -U
flag?
pip-compile has started taking extremely long for me (killed after more than an hour).
python version: 3.7.10 pip version: 23.0 pip-compile version: 6.10.0
requirements-dev.in:
tfx[kfp]==1.12.0
tensorflow_text
tensorflow_recommenders
scann
tqdm
ujson
google-api-python-client
google-auth
google-auth-httplib2
pytest
pytest-cov
pytest-mock
requests-mock
Pillow
pre-commit
After a bit of debugging, it looks like the -U
flag is to blame.
The following completes fast (just over a minute) (output_no_U.txt).
time pip-compile -v --resolver=backtracking requirements-dev.in --output-file requirements-dev.txt | ts -i "{{%H:%M:%.S}}" > output_no_U.txt
Whereas, this does not:
time pip-compile -v -U --resolver=backtracking requirements-dev.in --output-file requirements-dev.txt | ts -i "{{%H:%M:%.S}}" > output_with_U.txt
I usually kill after many minutes (sometimes more than an hour). I have tried multiple times and 9/10 times it would be extremely slow, but one time it was relatively fast at just over 3 minutes. Looking at the attached output_with_U.txt might give some insight into which steps are slow. The incremental timestamps show these steps taking > 40s each:
{{00:00:43.895765}} Using cached google_cloud_aiplatform-1.16.1-py2.py3-none-any.whl (2.2 MB)
{{00:00:45.071425}} Using cached google_cloud_aiplatform-1.16.0-py2.py3-none-any.whl (2.2 MB)
{{00:00:49.434197}} Using cached google_cloud_aiplatform-1.15.1-py2.py3-none-any.whl (2.1 MB)
{{00:00:45.209635}} Using cached google_cloud_aiplatform-1.15.0-py2.py3-none-any.whl (2.1 MB)
{{00:00:53.008086}} Using cached google_cloud_aiplatform-1.14.0-py2.py3-none-any.whl (1.9 MB)
{{00:00:52.136853}} Using cached google_cloud_aiplatform-1.13.1-py2.py3-none-any.whl (1.8 MB)
and some steps taking minutes:
{{00:01:51.449972}} INFO: pip is looking at multiple versions of absl-py to determine which version is compatible with other requirements. This could take a while.
{{00:00:44.400301}} INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. See https://pip.pypa.io/warnings/backtracking for guidance. If you want to abort this run, press Ctrl + C.
This is more of a general question about the difference in the resolvers. I noticed pip-compile --resolver=backtracing
is downloading the packages while the legacy resolved is not. I assume that is a "Feature" of the backtracing resolver. I assumed it would be caching what it downloaded but then I see some of the packages being redownloaded during pip-sync.
Can someone explain to me why this is happening? Obviously it is contributing to the time I experience when using the backtracing resolver.
I just wanted to echo @IzakMaraisTAL -- this is slow enough where it doesn't ever actually resolve on some of my bigger requirements files. I originally thought it might be a pip-tools 6 vs 7 issue, but I confirmed that this is happening on both versions if you explicitly specify --resolver=backtracking
.
I am using pip-tools 7.4.1
with python 3.11
. I am running the below command, it is taking more than 30 mins and still running
pip-compile --no-emit-index-url --generate-hashes requirements.in --output-file requirements.txt
This is my requirements.in
file. Am I missing something?
apache-beam[gcp]>2.54.0
beautifulsoup4
bleach
certifi
deepdiff
defusedxml
elasticsearch
firebase-admin
future
google-cloud-storage>2.1.0
google-api-core[grpc]>1.34.1
google-auth>=1.35.1
google-cloud-bigquery
google-cloud-dataflow-client
google-cloud-logging
google-cloud-ndb>2.2.1
google-cloud-secret-manager
google-cloud-tasks
google-cloud-translate
grpcio-status>1.48.2
gunicorn
html5lib
mailchimp3
mutagen
numpy>=1.26.4
packaging
Pillow
protobuf>3.20.3
pyarrow>=12.0.0
pylatexenc
pytest
PyYAML>=6.0.1
redis>=5.0.0
requests
requests-mock
requests-toolbelt
result
rsa
simplejson
six
soupsieve
typing-extensions
urllib3
webapp2
webencodings
I don't mean to disparage the pip-tools project, but for people struggling with this specific problem, you might try uv
. I tested with uv pip-compile
and it resolved the same dependencies in reasonable time.
Hello fellow pip-tool users,
I don't know if it's just me or does pip-compile take a long time to complete? I'm new to
pip-tools
, love the concept, but am worried that I may be running afoul of best practices.At the moment what i'm doing repeatedly is running
pip-compile -U --resolver=backtracking
after editingrequirements.in
.A pretty short & basic
requirements.in
like the following takespip-compile
3-5 minutes:My intuition tells me that there's a 50% chance that i'm doing something wrong here and I would be grateful for advice 😄