search

pantsbuild / pants

The Pants Build System
https://www.pantsbuild.org
Apache License 2.0
3.27k stars 626 forks source link

Slow lockfile generation with PEX-native lockfile #14998

Closed stuhood closed 2 years ago

stuhood commented 2 years ago

The following PEX lockfile generation (with [python].lockfile_generator=pex) invoke took more than 30 minutes (before being canceled):

14:34:22.55 [DEBUG] spawned local process as Some(2737762) for Process { argv: ["$HOME.pyenv/versions/3.10.2/bin/python", "./pex", "--tmpdir", ".tmp", "lock", "create", "--python-path", "$HOME.pyenv/versions/2.7.18/bin:$HOME.pyenv/versions/3.10.2/bin:$HOME.pyenv/versions/3.6.15/bin:$HOME.pyenv/versions/3.7.11/bin:$HOME.pyenv/versions/3.8.12/bin:$HOME.pyenv/versions/3.9.9/bin", "--output=lock.json", "--no-emit-warnings", "--style=universal", "--resolver-version", "pip-2020-resolver", "--indent=2", "--no-pypi", "--index=https://pypi.org/simple/", "--manylinux", "manylinux2014", "--interpreter-constraint", "CPython==3.9.*", "APScheduler==3.8.1", "Babel==2.9.1", "Cython==0.29.24", "Deprecated==1.2.13", "Faker==13.3.4", "Flask-Admin==1.5.8", "Flask-SQLAlchemy-Session==1.1", "Flask-SQLAlchemy==2.5.1", "Flask-WTF==0.15.1", "Flask==2.0.1", "Jinja2==3.0.1", "Mako==1.2.0", "MarkupSafe==2.0.1", "Pillow==8.4.0", "PyGithub==1.55", "PyJWT==2.3.0", "PyNaCl==1.4.0", "PySocks==1.7.1", "PyYAML==5.4.1", "Pygments==2.11.2", "Quandl==3.7.0", "Quart==0.15.1", "SQLAlchemy-Paginator==0.2", "SQLAlchemy-Utils==0.38.2", "SQLAlchemy==1.3.24", "Send2Trash==1.8.0", "Sphinx==4.2.0", "UnleashClient==4.4.1", "WTForms-SQLAlchemy==0.2", "WTForms==2.3.3", "Werkzeug==2.0.3", "aiofiles==0.8.0", "aiohttp==3.7.4.post0", "aioredis==1.3.1", "alabaster==0.7.12", "alembic-verify==0.1.4", "alembic==1.7.3", "amazon-braket-default-simulator==1.5.0", "amazon-braket-ocean-plugin==1.0.8", "amazon-braket-schemas==1.7.1", "amazon-braket-sdk==1.15.0", "amqp==5.1.0", "anyio==3.4.0", "appdirs==1.4.4", "argon2-cffi==21.1.0", "arrow==1.2.1", "asttokens==2.0.5", "async-timeout==3.0.1", "attrs==21.2.0", "auth0-python==3.18.0", "backcall==0.2.0", "backoff==1.11.1", "backports.entry-points-selectable==1.1.1", "bcrypt==3.2.0", "billiard==3.6.4.0", "black==21.9b0", "bleach==4.1.0", "blinker==1.4", "boltons==21.0.0", "boto3==1.21.1", "botocore==1.24.1", "celery==5.1.2", "certifi==2021.10.8", "cffi==1.15.0", "cfgv==3.3.1", "chardet==4.0.0", "charset-normalizer==2.0.8", "click-didyoumean==0.3.0", "click-plugins==1.1.1", "click-repl==0.2.0", "click==7.1.2", "colorama==0.4.4", "coverage==5.5", "cryptography==36.0.0", "cycler==0.11.0", "debugpy==1.5.1", "decorator==4.4.2", "deepmerge==1.0.1", "defusedxml==0.7.1", "dill==0.3.4", "dimod==0.10.6", "diskcache==5.2.1", "distlib==0.3.3", "distro==1.6.0", "dlx==1.0.4", "dnspython==2.2.1", "dockerpty==0.4.1", "docopt==0.6.2", "docplex==2.23.222", "docutils==0.17.1", "dogpile.cache==1.1.4", "dwave-cloud-client==0.9.2", "dwave-networkx==0.8.10", "dwave-preprocessing==0.3.1.post0", "dwave-qbsolv==0.3.3.post0", "dwave-system==1.9.0", "dwave-tabu==0.4.2", "email-validator==1.1.3", "entrypoints==0.3", "execnet==1.9.0", "factory-boy==3.2.0", "fastdtw==0.3.4", "fasteners==0.16.3", "fastjsonschema==2.15.1", "fcache==0.4.7", "fenestrate==0.3.0", "filelock==3.4.0", "flask-oidc==1.4.0", "gunicorn==20.1.0", "h11==0.13.0", "h2==4.1.0", "h5py==3.2.1", "hiredis==2.0.0", "homebase==1.0.1", "hpack==4.0.0", "httplib2==0.20.4", "hypercorn==0.13.2", "hyperframe==6.0.1", "hypothesis==6.21.6", "icdiff==2.0.4", "icontract==2.6.0", "identify==2.4.0", "idna==3.3", "imagesize==1.3.0", "importlib-metadata==4.8.1", "inflection==0.5.1", "iniconfig==1.1.1", "intervaltree==3.1.0", "invoke==1.6.0", "ipykernel==6.5.1", "ipython-genutils==0.2.0", "ipython==7.30.0", "ipywidgets==7.6.5", "itsdangerous==2.0.1", "jedi==0.18.1", "jinja2-time==0.2.0", "jmespath==0.10.0", "joblib==1.1.0", "json5==0.9.6", "jsonref==0.2", "jsonschema==3.2.0", "jupyter-client==7.1.0", "jupyter-core==4.9.1", "jupyter-packaging==0.7.9", "jupyter-server==1.12.1", "jupyterlab-pygments==0.1.2", "jupyterlab-server==2.8.2", "jupyterlab-widgets==1.1.0", "jupyterlab==3.1.12", "kiwisolver==1.3.2", "kombu==5.2.4", "llvmlite==0.36.0", "lxml==4.8.0", "lz4==3.1.3", "markdown2==2.4.1", "matplotlib-inline==0.1.3", "matplotlib==3.4.3", "memory-profiler==0.58.0", "minorminer==0.2.6", "mistune==0.8.4", "mmh3==3.0.0", "more-itertools==8.12.0", "moto==2.2.7", "mpmath==1.2.1", "multidict==5.2.0", "multitasking==0.0.10", "mypy-extensions==0.4.3", "nbclassic==0.3.4", "nbclient==0.5.9", "nbconvert==6.3.0", "nbformat==5.1.3", "nbmake==1.0", "nest-asyncio==1.5.1", "networkx==2.6.3", "nodeenv==1.6.0", "notebook==6.4.6", "ntlm-auth==1.5.0", "numba==0.53.1", "numpy==1.21.2", "oauth2client==4.1.3", "okta==0.0.4", "opentelemetry-api==1.8.0", "opentelemetry-exporter-zipkin-json==1.8.0", "opentelemetry-exporter-zipkin-proto-http==1.8.0", "opentelemetry-exporter-zipkin==1.8.0", "opentelemetry-instrumentation-sqlalchemy==0.27b0", "opentelemetry-instrumentation==0.27b0", "opentelemetry-sdk==1.8.0", "opentelemetry-semantic-conventions==0.27b0", "opt-einsum==3.3.0", "packaging==21.0", "pandas==1.3.4", "pandocfilters==1.5.0", "paramiko==2.8.1", "parse==1.19.0", "parso==0.8.2", "pathspec==0.9.0", "pbr==5.8.1", "pexpect==4.8.0", "pickleshare==0.7.5", "platformdirs==2.4.0", "plucky==0.4.3", "pluggy==1.0.0", "ply==3.11", "pprintpp==0.4.0", "pre-commit==2.15.0", "priority==2.0.0", "prometheus-client==0.12.0", "prompt-toolkit==3.0.23", "protobuf==3.19.4", "psutil==5.8.0", "psycopg2-binary==2.9.2", "ptyprocess==0.7.0", "py==1.11.0", "pyasn1-modules==0.2.8", "pyasn1==0.4.8", "pybind11==2.8.1", "pycodestyle==2.7.0", "pycparser==2.21", "pydantic==1.8.2", "pynvml==11.0.0", "pyparsing==3.0.6", "pyrsistent==0.18.0", "pytest-asyncio==0.15.1", "pytest-cov==3.0.0", "pytest-forked==1.3.0", "pytest-icdiff==0.5", "pytest-mock==3.6.1", "pytest-repeat==0.9.1", "pytest-xdist==2.4.0", "pytest==6.2.5", "python-constraint==1.4.0", "python-dateutil==2.8.2", "python-decouple==3.4", "python-dotenv==0.19.2", "python-http-client==3.3.7", "python-lsp-black==1.0.0", "python-lsp-jsonrpc==1.0.0", "python-lsp-server==1.3.2", "pytz-deprecation-shim==0.1.0.post0", "pytz==2021.3", "pyzmq==22.3.0", "qcware-quasar==1.0.6", "qcware-transpile==0.1.1a19", "qiskit-aer==0.9.0", "qiskit-aqua==0.9.5", "qiskit-ibmq-provider==0.16.0", "qiskit-ignis==0.6.0", "qiskit-terra==0.18.3", "qiskit==0.30.1", "qubovert==1.2.3", "redis==3.5.3", "regex==2021.11.10", "requests-ntlm==1.1.0", "requests==2.26.0", "responses==0.18.0", "retworkx==0.10.2", "rsa==4.8", "s3transfer==0.5.0", "scikit-learn==0.24.2", "scipy==1.7.1", "sendgrid==6.8.1", "setuptools==58.0.4", "six==1.16.0", "sniffio==1.2.0", "snowballstemmer==2.2.0", "sortedcontainers==2.4.0", "sphinx-rtd-theme==1.0.0", "sphinxcontrib-applehelp==1.0.2", "sphinxcontrib-devhelp==1.0.2", "sphinxcontrib-htmlhelp==2.0.0", "sphinxcontrib-jsmath==1.0.1", "sphinxcontrib-qthelp==1.0.3", "sphinxcontrib-serializinghtml==1.1.5", "sqlalchemy-diff==0.1.5", "starkbank-ecdsa==2.0.3", "stevedore==3.5.0", "structlog==21.5.0", "symengine==0.8.1", "sympy==1.9", "tabulate==0.8.9", "terminado==0.12.1", "testpath==0.5.0", "texttable==1.6.4", "threadpoolctl==3.0.0", "toml==0.10.2", "tomli==1.2.2", "toolz==0.11.2", "torch==1.10.2", "tornado==6.1", "tqdm==4.62.3", "traitlets==5.1.1", "tweedledum==1.1.1", "types-requests==2.25.8", "typing_extensions==4.0.0", "tzdata==2021.5", "tzlocal==4.1", "ujson==4.3.0", "unitair==0.1.6", "urllib3==1.26.7", "vine==5.0.0", "virtualenv==20.10.0", "watchdog==2.1.5", "wcwidth==0.2.5", "webencodings==0.5.1", "websocket-client==1.3.2", "wheel==0.37.0", "widgetsnbextension==3.5.2", "wrapt==1.13.3", "wsproto==1.1.0", "xmltodict==0.12.0", "yapf==0.31.0", "yarl==1.7.2", "yattag==1.14.0", "yfinance==0.1.70", "zipp==3.7.0"], env: { <<...some env vars...>> , "CPPFLAGS": "", "LDFLAGS": "", "PATH": "<< ... project path...>> :$HOME.pyenv/shims:$HOME.cargo/bin:$HOME.local/bin:$HOME.pyenv/bin:/opt/cuda/bin:$HOME.dotnet/tools:/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl", "PEX_IGNORE_RCFILES": "true", "PEX_PYTHON_PATH": "$HOME.pyenv/versions/2.7.18/bin:$HOME.pyenv/versions/3.10.2/bin:$HOME.pyenv/versions/3.6.15/bin:$HOME.pyenv/versions/3.7.11/bin:$HOME.pyenv/versions/3.8.12/bin:$HOME.pyenv/versions/3.9.9/bin", "PEX_ROOT": ".cache/pex_root", "PEX_SCRIPT": "pex3", "SSH_AUTH_SOCK": "<<...>>"}, working_directory: None, input_digests: InputDigests { complete: DirectoryDigest { digest: Digest { hash: Fingerprint<d81aa8a79bc3999a9e01b5df226e7649edb5bc4e16f187a17cdcc480e3f3196c>, size_bytes: 158 }, tree: "Some(..)" }, nailgun: DirectoryDigest { digest: Digest { hash: Fingerprint<e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855>, size_bytes: 0 }, tree: "Some(..)" }, input_files: DirectoryDigest { digest: Digest { hash: Fingerprint<d81aa8a79bc3999a9e01b5df226e7649edb5bc4e16f187a17cdcc480e3f3196c>, size_bytes: 158 }, tree: "Some(..)" }, immutable_inputs: {}, use_nailgun: [] }, output_files: {RelativePath("lock.json")}, output_directories: {}, timeout: None, execution_slot_variable: None, concurrency_available: 0, description: "Generate lockfile for python-default", level: Info, append_only_caches: {CacheName("pex_root"): RelativePath(".cache/pex_root")}, jdk_home: None, platform_constraint: None, cache_scope: PerSession }

With poetry, it completed after 25 seconds.

jsirois commented 2 years ago

A quick check shows (still running for me many minutes later):

$ python -mpex.cli lock create -vvv --output=lock.json --no-emit-warnings --style=universal --resolver-version pip-2020-resolver --indent=2 --no-pypi --index=https://pypi.org/simple/ --manylinux manylinux2014 --interpreter-constraint "CPython==3.9.*" APScheduler==3.8.1 Babel==2.9.1 Cython==0.29.24 Deprecated==1.2.13 Faker==13.3.4 Flask-Admin==1.5.8 Flask-SQLAlchemy-Session==1.1 Flask-SQLAlchemy==2.5.1 Flask-WTF==0.15.1 Flask==2.0.1 Jinja2==3.0.1 Mako==1.2.0 MarkupSafe==2.0.1 Pillow==8.4.0 PyGithub==1.55 PyJWT==2.3.0 PyNaCl==1.4.0 PySocks==1.7.1 PyYAML==5.4.1 Pygments==2.11.2 Quandl==3.7.0 Quart==0.15.1 SQLAlchemy-Paginator==0.2 SQLAlchemy-Utils==0.38.2 SQLAlchemy==1.3.24 Send2Trash==1.8.0 Sphinx==4.2.0 UnleashClient==4.4.1 WTForms-SQLAlchemy==0.2 WTForms==2.3.3 Werkzeug==2.0.3 aiofiles==0.8.0 aiohttp==3.7.4.post0 aioredis==1.3.1 alabaster==0.7.12 alembic-verify==0.1.4 alembic==1.7.3 amazon-braket-default-simulator==1.5.0 amazon-braket-ocean-plugin==1.0.8 amazon-braket-schemas==1.7.1 amazon-braket-sdk==1.15.0 amqp==5.1.0 anyio==3.4.0 appdirs==1.4.4 argon2-cffi==21.1.0 arrow==1.2.1 asttokens==2.0.5 async-timeout==3.0.1 attrs==21.2.0 auth0-python==3.18.0 backcall==0.2.0 backoff==1.11.1 backports.entry-points-selectable==1.1.1 bcrypt==3.2.0 billiard==3.6.4.0 black==21.9b0 bleach==4.1.0 blinker==1.4 boltons==21.0.0 boto3==1.21.1 botocore==1.24.1 celery==5.1.2 certifi==2021.10.8 cffi==1.15.0 cfgv==3.3.1 chardet==4.0.0 charset-normalizer==2.0.8 click-didyoumean==0.3.0 click-plugins==1.1.1 click-repl==0.2.0 click==7.1.2 colorama==0.4.4 coverage==5.5 cryptography==36.0.0 cycler==0.11.0 debugpy==1.5.1 decorator==4.4.2 deepmerge==1.0.1 defusedxml==0.7.1 dill==0.3.4 dimod==0.10.6 diskcache==5.2.1 distlib==0.3.3 distro==1.6.0 dlx==1.0.4 dnspython==2.2.1 dockerpty==0.4.1 docopt==0.6.2 docplex==2.23.222 docutils==0.17.1 dogpile.cache==1.1.4 dwave-cloud-client==0.9.2 dwave-networkx==0.8.10 dwave-preprocessing==0.3.1.post0 dwave-qbsolv==0.3.3.post0 dwave-system==1.9.0 dwave-tabu==0.4.2 email-validator==1.1.3 entrypoints==0.3 execnet==1.9.0 factory-boy==3.2.0 fastdtw==0.3.4 fasteners==0.16.3 fastjsonschema==2.15.1 fcache==0.4.7 fenestrate==0.3.0 filelock==3.4.0 flask-oidc==1.4.0 gunicorn==20.1.0 h11==0.13.0 h2==4.1.0 h5py==3.2.1 hiredis==2.0.0 homebase==1.0.1 hpack==4.0.0 httplib2==0.20.4 hypercorn==0.13.2 hyperframe==6.0.1 hypothesis==6.21.6 icdiff==2.0.4 icontract==2.6.0 identify==2.4.0 idna==3.3 imagesize==1.3.0 importlib-metadata==4.8.1 inflection==0.5.1 iniconfig==1.1.1 intervaltree==3.1.0 invoke==1.6.0 ipykernel==6.5.1 ipython-genutils==0.2.0 ipython==7.30.0 ipywidgets==7.6.5 itsdangerous==2.0.1 jedi==0.18.1 jinja2-time==0.2.0 jmespath==0.10.0 joblib==1.1.0 json5==0.9.6 jsonref==0.2 jsonschema==3.2.0 jupyter-client==7.1.0 jupyter-core==4.9.1 jupyter-packaging==0.7.9 jupyter-server==1.12.1 jupyterlab-pygments==0.1.2 jupyterlab-server==2.8.2 jupyterlab-widgets==1.1.0 jupyterlab==3.1.12 kiwisolver==1.3.2 kombu==5.2.4 llvmlite==0.36.0 lxml==4.8.0 lz4==3.1.3 markdown2==2.4.1 matplotlib-inline==0.1.3 matplotlib==3.4.3 memory-profiler==0.58.0 minorminer==0.2.6 mistune==0.8.4 mmh3==3.0.0 more-itertools==8.12.0 moto==2.2.7 mpmath==1.2.1 multidict==5.2.0 multitasking==0.0.10 mypy-extensions==0.4.3 nbclassic==0.3.4 nbclient==0.5.9 nbconvert==6.3.0 nbformat==5.1.3 nbmake==1.0 nest-asyncio==1.5.1 networkx==2.6.3 nodeenv==1.6.0 notebook==6.4.6 ntlm-auth==1.5.0 numba==0.53.1 numpy==1.21.2 oauth2client==4.1.3 okta==0.0.4 opentelemetry-api==1.8.0 opentelemetry-exporter-zipkin-json==1.8.0 opentelemetry-exporter-zipkin-proto-http==1.8.0 opentelemetry-exporter-zipkin==1.8.0 opentelemetry-instrumentation-sqlalchemy==0.27b0 opentelemetry-instrumentation==0.27b0 opentelemetry-sdk==1.8.0 opentelemetry-semantic-conventions==0.27b0 opt-einsum==3.3.0 packaging==21.0 pandas==1.3.4 pandocfilters==1.5.0 paramiko==2.8.1 parse==1.19.0 parso==0.8.2 pathspec==0.9.0 pbr==5.8.1 pexpect==4.8.0 pickleshare==0.7.5 platformdirs==2.4.0 plucky==0.4.3 pluggy==1.0.0 ply==3.11 pprintpp==0.4.0 pre-commit==2.15.0 priority==2.0.0 prometheus-client==0.12.0 prompt-toolkit==3.0.23 protobuf==3.19.4 psutil==5.8.0 psycopg2-binary==2.9.2 ptyprocess==0.7.0 py==1.11.0 pyasn1-modules==0.2.8 pyasn1==0.4.8 pybind11==2.8.1 pycodestyle==2.7.0 pycparser==2.21 pydantic==1.8.2 pynvml==11.0.0 pyparsing==3.0.6 pyrsistent==0.18.0 pytest-asyncio==0.15.1 pytest-cov==3.0.0 pytest-forked==1.3.0 pytest-icdiff==0.5 pytest-mock==3.6.1 pytest-repeat==0.9.1 pytest-xdist==2.4.0 pytest==6.2.5 python-constraint==1.4.0 python-dateutil==2.8.2 python-decouple==3.4 python-dotenv==0.19.2 python-http-client==3.3.7 python-lsp-black==1.0.0 python-lsp-jsonrpc==1.0.0 python-lsp-server==1.3.2 pytz-deprecation-shim==0.1.0.post0 pytz==2021.3 pyzmq==22.3.0 qcware-quasar==1.0.6 qcware-transpile==0.1.1a19 qiskit-aer==0.9.0 qiskit-aqua==0.9.5 qiskit-ibmq-provider==0.16.0 qiskit-ignis==0.6.0 qiskit-terra==0.18.3 qiskit==0.30.1 qubovert==1.2.3 redis==3.5.3 regex==2021.11.10 requests-ntlm==1.1.0 requests==2.26.0 responses==0.18.0 retworkx==0.10.2 rsa==4.8 s3transfer==0.5.0 scikit-learn==0.24.2 scipy==1.7.1 sendgrid==6.8.1 setuptools==58.0.4 six==1.16.0 sniffio==1.2.0 snowballstemmer==2.2.0 sortedcontainers==2.4.0 sphinx-rtd-theme==1.0.0 sphinxcontrib-applehelp==1.0.2 sphinxcontrib-devhelp==1.0.2 sphinxcontrib-htmlhelp==2.0.0 sphinxcontrib-jsmath==1.0.1 sphinxcontrib-qthelp==1.0.3 sphinxcontrib-serializinghtml==1.1.5 sqlalchemy-diff==0.1.5 starkbank-ecdsa==2.0.3 stevedore==3.5.0 structlog==21.5.0 symengine==0.8.1 sympy==1.9 tabulate==0.8.9 terminado==0.12.1 testpath==0.5.0 texttable==1.6.4 threadpoolctl==3.0.0 toml==0.10.2 tomli==1.2.2 toolz==0.11.2 torch==1.10.2 tornado==6.1 tqdm==4.62.3 traitlets==5.1.1 tweedledum==1.1.1 types-requests==2.25.8 typing_extensions==4.0.0 tzdata==2021.5 tzlocal==4.1 ujson==4.3.0 unitair==0.1.6 urllib3==1.26.7 vine==5.0.0 virtualenv==20.10.0 watchdog==2.1.5 wcwidth==0.2.5 webencodings==0.5.1 websocket-client==1.3.2 wheel==0.37.0 widgetsnbextension==3.5.2 wrapt==1.13.3 wsproto==1.1.0 xmltodict==0.12.0 yapf==0.31.0 yarl==1.7.2 yattag==1.14.0 yfinance==0.1.70 zipp==3.7.0
pex: Resolving for:
pex: Hashing pex: 17.0ms                                                 
pex: Isolating pex: 0.0ms
pex: Calculating compatible python versions for ('==3.9.*',): 5.6ms
    Running command python setup.py egg_info
    running egg_info
    creating /tmp/pip-pip-egg-info-r5ug9ht4/Flask_Admin.egg-info
    writing /tmp/pip-pip-egg-info-r5ug9ht4/Flask_Admin.egg-info/PKG-INFO
    writing dependency_links to /tmp/pip-pip-egg-info-r5ug9ht4/Flask_Admin.egg-info/dependency_links.txt
    writing requirements to /tmp/pip-pip-egg-info-r5ug9ht4/Flask_Admin.egg-info/requires.txt
    writing top-level names to /tmp/pip-pip-egg-info-r5ug9ht4/Flask_Admin.egg-info/top_level.txt
    writing manifest file '/tmp/pip-pip-egg-info-r5ug9ht4/Flask_Admin.egg-info/SOURCES.txt'
    reading manifest file '/tmp/pip-pip-egg-info-r5ug9ht4/Flask_Admin.egg-info/SOURCES.txt'
    reading manifest template 'MANIFEST.in'
    warning: no previously-included files matching '*.pyc' found under directory 'flask_admin'
    writing manifest file '/tmp/pip-pip-egg-info-r5ug9ht4/Flask_Admin.egg-info/SOURCES.txt'
    Running command python setup.py egg_info
    running egg_info
    creating /tmp/pip-pip-egg-info-9090dfjr/Flask_SQLAlchemy_Session.egg-info
    writing /tmp/pip-pip-egg-info-9090dfjr/Flask_SQLAlchemy_Session.egg-info/PKG-INFO
    writing dependency_links to /tmp/pip-pip-egg-info-9090dfjr/Flask_SQLAlchemy_Session.egg-info/dependency_links.txt
    writing requirements to /tmp/pip-pip-egg-info-9090dfjr/Flask_SQLAlchemy_Session.egg-info/requires.txt
    writing top-level names to /tmp/pip-pip-egg-info-9090dfjr/Flask_SQLAlchemy_Session.egg-info/top_level.txt
    writing manifest file '/tmp/pip-pip-egg-info-9090dfjr/Flask_SQLAlchemy_Session.egg-info/SOURCES.txt'
    reading manifest file '/tmp/pip-pip-egg-info-9090dfjr/Flask_SQLAlchemy_Session.egg-info/SOURCES.txt'
    writing manifest file '/tmp/pip-pip-egg-info-9090dfjr/Flask_SQLAlchemy_Session.egg-info/SOURCES.txt'
    Running command python setup.py egg_info
    running egg_info
    creating /tmp/pip-pip-egg-info-y0jy8zq7/SQLAlchemy_Paginator.egg-info
    writing /tmp/pip-pip-egg-info-y0jy8zq7/SQLAlchemy_Paginator.egg-info/PKG-INFO
    writing dependency_links to /tmp/pip-pip-egg-info-y0jy8zq7/SQLAlchemy_Paginator.egg-info/dependency_links.txt
    writing top-level names to /tmp/pip-pip-egg-info-y0jy8zq7/SQLAlchemy_Paginator.egg-info/top_level.txt
    writing manifest file '/tmp/pip-pip-egg-info-y0jy8zq7/SQLAlchemy_Paginator.egg-info/SOURCES.txt'
    reading manifest file '/tmp/pip-pip-egg-info-y0jy8zq7/SQLAlchemy_Paginator.egg-info/SOURCES.txt'
    reading manifest template 'MANIFEST.in'
    writing manifest file '/tmp/pip-pip-egg-info-y0jy8zq7/SQLAlchemy_Paginator.egg-info/SOURCES.txt'
    Running command python setup.py egg_info
    running egg_info
    creating /tmp/pip-pip-egg-info-9sh05p8w/alembic_verify.egg-info
    writing /tmp/pip-pip-egg-info-9sh05p8w/alembic_verify.egg-info/PKG-INFO
    writing dependency_links to /tmp/pip-pip-egg-info-9sh05p8w/alembic_verify.egg-info/dependency_links.txt
    writing entry points to /tmp/pip-pip-egg-info-9sh05p8w/alembic_verify.egg-info/entry_points.txt
    writing requirements to /tmp/pip-pip-egg-info-9sh05p8w/alembic_verify.egg-info/requires.txt
    writing top-level names to /tmp/pip-pip-egg-info-9sh05p8w/alembic_verify.egg-info/top_level.txt
    writing manifest file '/tmp/pip-pip-egg-info-9sh05p8w/alembic_verify.egg-info/SOURCES.txt'
    reading manifest file '/tmp/pip-pip-egg-info-9sh05p8w/alembic_verify.egg-info/SOURCES.txt'
    writing manifest file '/tmp/pip-pip-egg-info-9sh05p8w/alembic_verify.egg-info/SOURCES.txt'
    Running command python setup.py egg_info
    running egg_info
    creating /tmp/pip-pip-egg-info-7ge12l08/blinker.egg-info
    writing /tmp/pip-pip-egg-info-7ge12l08/blinker.egg-info/PKG-INFO
    writing dependency_links to /tmp/pip-pip-egg-info-7ge12l08/blinker.egg-info/dependency_links.txt
    writing top-level names to /tmp/pip-pip-egg-info-7ge12l08/blinker.egg-info/top_level.txt
    writing manifest file '/tmp/pip-pip-egg-info-7ge12l08/blinker.egg-info/SOURCES.txt'
    reading manifest file '/tmp/pip-pip-egg-info-7ge12l08/blinker.egg-info/SOURCES.txt'
    reading manifest template 'MANIFEST.in'
    writing manifest file '/tmp/pip-pip-egg-info-7ge12l08/blinker.egg-info/SOURCES.txt'
    Running command python setup.py egg_info
    running egg_info
    creating /tmp/pip-pip-egg-info-_0wad5o7/dlx.egg-info
    writing /tmp/pip-pip-egg-info-_0wad5o7/dlx.egg-info/PKG-INFO
    writing dependency_links to /tmp/pip-pip-egg-info-_0wad5o7/dlx.egg-info/dependency_links.txt
    writing top-level names to /tmp/pip-pip-egg-info-_0wad5o7/dlx.egg-info/top_level.txt
    writing manifest file '/tmp/pip-pip-egg-info-_0wad5o7/dlx.egg-info/SOURCES.txt'
    reading manifest file '/tmp/pip-pip-egg-info-_0wad5o7/dlx.egg-info/SOURCES.txt'
    writing manifest file '/tmp/pip-pip-egg-info-_0wad5o7/dlx.egg-info/SOURCES.txt'
    Running command python setup.py egg_info
    running egg_info
    creating /tmp/pip-pip-egg-info-z1v_fmyq/dockerpty.egg-info
    writing /tmp/pip-pip-egg-info-z1v_fmyq/dockerpty.egg-info/PKG-INFO
    writing dependency_links to /tmp/pip-pip-egg-info-z1v_fmyq/dockerpty.egg-info/dependency_links.txt
    writing requirements to /tmp/pip-pip-egg-info-z1v_fmyq/dockerpty.egg-info/requires.txt
    writing top-level names to /tmp/pip-pip-egg-info-z1v_fmyq/dockerpty.egg-info/top_level.txt
    writing manifest file '/tmp/pip-pip-egg-info-z1v_fmyq/dockerpty.egg-info/SOURCES.txt'
    reading manifest file '/tmp/pip-pip-egg-info-z1v_fmyq/dockerpty.egg-info/SOURCES.txt'
    reading manifest template 'MANIFEST.in'
    writing manifest file '/tmp/pip-pip-egg-info-z1v_fmyq/dockerpty.egg-info/SOURCES.txt'
    Running command python setup.py egg_info
    running egg_info
    creating /tmp/pip-pip-egg-info-_y9gl5pz/docopt.egg-info
    writing /tmp/pip-pip-egg-info-_y9gl5pz/docopt.egg-info/PKG-INFO
    writing dependency_links to /tmp/pip-pip-egg-info-_y9gl5pz/docopt.egg-info/dependency_links.txt
    writing top-level names to /tmp/pip-pip-egg-info-_y9gl5pz/docopt.egg-info/top_level.txt
    writing manifest file '/tmp/pip-pip-egg-info-_y9gl5pz/docopt.egg-info/SOURCES.txt'
    reading manifest file '/tmp/pip-pip-egg-info-_y9gl5pz/docopt.egg-info/SOURCES.txt'
    reading manifest template 'MANIFEST.in'
    writing manifest file '/tmp/pip-pip-egg-info-_y9gl5pz/docopt.egg-info/SOURCES.txt'
    Running command python setup.py egg_info
    running egg_info
    creating /tmp/pip-pip-egg-info-ft7mbrbl/docplex.egg-info
    writing /tmp/pip-pip-egg-info-ft7mbrbl/docplex.egg-info/PKG-INFO
    writing dependency_links to /tmp/pip-pip-egg-info-ft7mbrbl/docplex.egg-info/dependency_links.txt
    writing requirements to /tmp/pip-pip-egg-info-ft7mbrbl/docplex.egg-info/requires.txt
    writing top-level names to /tmp/pip-pip-egg-info-ft7mbrbl/docplex.egg-info/top_level.txt
    writing manifest file '/tmp/pip-pip-egg-info-ft7mbrbl/docplex.egg-info/SOURCES.txt'
    reading manifest file '/tmp/pip-pip-egg-info-ft7mbrbl/docplex.egg-info/SOURCES.txt'
    writing manifest file '/tmp/pip-pip-egg-info-ft7mbrbl/docplex.egg-info/SOURCES.txt'
    ** The documentation can be found here: http://ibmdecisionoptimization.github.io/docplex-doc/
    ** The examples can be found here: https://github.com/IBMDecisionOptimization/docplex-examples
    Running command python setup.py egg_info
    running egg_info
    creating /tmp/pip-pip-egg-info-dgmy053o/fastdtw.egg-info
    writing /tmp/pip-pip-egg-info-dgmy053o/fastdtw.egg-info/PKG-INFO
    writing dependency_links to /tmp/pip-pip-egg-info-dgmy053o/fastdtw.egg-info/dependency_links.txt
    writing requirements to /tmp/pip-pip-egg-info-dgmy053o/fastdtw.egg-info/requires.txt
    writing top-level names to /tmp/pip-pip-egg-info-dgmy053o/fastdtw.egg-info/top_level.txt
    writing manifest file '/tmp/pip-pip-egg-info-dgmy053o/fastdtw.egg-info/SOURCES.txt'
    reading manifest file '/tmp/pip-pip-egg-info-dgmy053o/fastdtw.egg-info/SOURCES.txt'
    reading manifest template 'MANIFEST.in'
    writing manifest file '/tmp/pip-pip-egg-info-dgmy053o/fastdtw.egg-info/SOURCES.txt'
    Running command python setup.py egg_info
    running egg_info
    creating /tmp/pip-pip-egg-info-9g6e3xlg/flask_oidc.egg-info
    writing /tmp/pip-pip-egg-info-9g6e3xlg/flask_oidc.egg-info/PKG-INFO
    writing dependency_links to /tmp/pip-pip-egg-info-9g6e3xlg/flask_oidc.egg-info/dependency_links.txt
    writing entry points to /tmp/pip-pip-egg-info-9g6e3xlg/flask_oidc.egg-info/entry_points.txt
    writing requirements to /tmp/pip-pip-egg-info-9g6e3xlg/flask_oidc.egg-info/requires.txt
    writing top-level names to /tmp/pip-pip-egg-info-9g6e3xlg/flask_oidc.egg-info/top_level.txt
    writing manifest file '/tmp/pip-pip-egg-info-9g6e3xlg/flask_oidc.egg-info/SOURCES.txt'
    reading manifest file '/tmp/pip-pip-egg-info-9g6e3xlg/flask_oidc.egg-info/SOURCES.txt'
    reading manifest template 'MANIFEST.in'
    warning: no files found matching 'LICENSE.rst'
    warning: no files found matching 'CHANGES.rst'
    warning: no previously-included files matching '*.pyc' found under directory 'docs'
    warning: no previously-included files matching '*.pyo' found under directory 'docs'
    warning: no files found matching '*' under directory 'example'
    warning: no previously-included files matching '*.pyc' found under directory 'example'
    warning: no previously-included files matching '*.pyo' found under directory 'example'
    no previously-included directories found matching 'docs/_build'
    no previously-included directories found matching 'docs/_themes/.git'
    writing manifest file '/tmp/pip-pip-egg-info-9g6e3xlg/flask_oidc.egg-info/SOURCES.txt'
    Running command python setup.py egg_info
    /home/jsirois/.pex/venvs/s/2b881b3a/venv/lib/python3.10/site-packages/pkg_resources/__init__.py:1940: UserWarning: /tmp/pip-download-54t7__ab/icdiff_288816c41fb24014af65406ab7b05e06/._icdiff.egg-info could not be properly decoded in UTF-8
      warnings.warn(msg)
    running egg_info
    creating /tmp/pip-pip-egg-info-5rffm18y/icdiff.egg-info
    writing /tmp/pip-pip-egg-info-5rffm18y/icdiff.egg-info/PKG-INFO
    writing dependency_links to /tmp/pip-pip-egg-info-5rffm18y/icdiff.egg-info/dependency_links.txt
    writing entry points to /tmp/pip-pip-egg-info-5rffm18y/icdiff.egg-info/entry_points.txt
    writing top-level names to /tmp/pip-pip-egg-info-5rffm18y/icdiff.egg-info/top_level.txt
    writing manifest file '/tmp/pip-pip-egg-info-5rffm18y/icdiff.egg-info/SOURCES.txt'
    reading manifest file '/tmp/pip-pip-egg-info-5rffm18y/icdiff.egg-info/SOURCES.txt'
    reading manifest template 'MANIFEST.in'
    writing manifest file '/tmp/pip-pip-egg-info-5rffm18y/icdiff.egg-info/SOURCES.txt'
    Running command python setup.py egg_info
    running egg_info
    creating /tmp/pip-pip-egg-info-32jwmt8s/icontract.egg-info
    writing /tmp/pip-pip-egg-info-32jwmt8s/icontract.egg-info/PKG-INFO
    writing dependency_links to /tmp/pip-pip-egg-info-32jwmt8s/icontract.egg-info/dependency_links.txt
    writing requirements to /tmp/pip-pip-egg-info-32jwmt8s/icontract.egg-info/requires.txt
    writing top-level names to /tmp/pip-pip-egg-info-32jwmt8s/icontract.egg-info/top_level.txt
    writing manifest file '/tmp/pip-pip-egg-info-32jwmt8s/icontract.egg-info/SOURCES.txt'
    file icontract.py (for module icontract) not found
    reading manifest file '/tmp/pip-pip-egg-info-32jwmt8s/icontract.egg-info/SOURCES.txt'
    writing manifest file '/tmp/pip-pip-egg-info-32jwmt8s/icontract.egg-info/SOURCES.txt'
    Running command python setup.py egg_info
    !!!>>> This is a RELEASE version <<<!!!

    Version: 3.1.0
    running egg_info
    creating /tmp/pip-pip-egg-info-x1jcx4hv/intervaltree.egg-info
    writing /tmp/pip-pip-egg-info-x1jcx4hv/intervaltree.egg-info/PKG-INFO
    writing dependency_links to /tmp/pip-pip-egg-info-x1jcx4hv/intervaltree.egg-info/dependency_links.txt
    writing requirements to /tmp/pip-pip-egg-info-x1jcx4hv/intervaltree.egg-info/requires.txt
    writing top-level names to /tmp/pip-pip-egg-info-x1jcx4hv/intervaltree.egg-info/top_level.txt
    writing manifest file '/tmp/pip-pip-egg-info-x1jcx4hv/intervaltree.egg-info/SOURCES.txt'
    reading manifest file '/tmp/pip-pip-egg-info-x1jcx4hv/intervaltree.egg-info/SOURCES.txt'
    reading manifest template 'MANIFEST.in'
    writing manifest file '/tmp/pip-pip-egg-info-x1jcx4hv/intervaltree.egg-info/SOURCES.txt'
    Running command python setup.py egg_info
    running egg_info
    creating /tmp/pip-pip-egg-info-einpg7p5/memory_profiler.egg-info
    writing /tmp/pip-pip-egg-info-einpg7p5/memory_profiler.egg-info/PKG-INFO
    writing dependency_links to /tmp/pip-pip-egg-info-einpg7p5/memory_profiler.egg-info/dependency_links.txt
    writing entry points to /tmp/pip-pip-egg-info-einpg7p5/memory_profiler.egg-info/entry_points.txt
    writing requirements to /tmp/pip-pip-egg-info-einpg7p5/memory_profiler.egg-info/requires.txt
    writing top-level names to /tmp/pip-pip-egg-info-einpg7p5/memory_profiler.egg-info/top_level.txt
    writing manifest file '/tmp/pip-pip-egg-info-einpg7p5/memory_profiler.egg-info/SOURCES.txt'
    reading manifest file '/tmp/pip-pip-egg-info-einpg7p5/memory_profiler.egg-info/SOURCES.txt'
    reading manifest template 'MANIFEST.in'
    warning: no files found matching 'mprof.bat'
    writing manifest file '/tmp/pip-pip-egg-info-einpg7p5/memory_profiler.egg-info/SOURCES.txt'
    Running command python setup.py egg_info
    running egg_info
    creating /tmp/pip-pip-egg-info-m9ye2_h2/multitasking.egg-info
    writing /tmp/pip-pip-egg-info-m9ye2_h2/multitasking.egg-info/PKG-INFO
    writing dependency_links to /tmp/pip-pip-egg-info-m9ye2_h2/multitasking.egg-info/dependency_links.txt
    writing top-level names to /tmp/pip-pip-egg-info-m9ye2_h2/multitasking.egg-info/top_level.txt
    writing manifest file '/tmp/pip-pip-egg-info-m9ye2_h2/multitasking.egg-info/SOURCES.txt'
    reading manifest file '/tmp/pip-pip-egg-info-m9ye2_h2/multitasking.egg-info/SOURCES.txt'
    reading manifest template 'MANIFEST.in'
    warning: no files found matching '*' under directory 'data'
    writing manifest file '/tmp/pip-pip-egg-info-m9ye2_h2/multitasking.egg-info/SOURCES.txt'
    Running command python setup.py egg_info
    running egg_info
    creating /tmp/pip-pip-egg-info-wf5vje5b/okta.egg-info
    writing /tmp/pip-pip-egg-info-wf5vje5b/okta.egg-info/PKG-INFO
    writing dependency_links to /tmp/pip-pip-egg-info-wf5vje5b/okta.egg-info/dependency_links.txt
    writing requirements to /tmp/pip-pip-egg-info-wf5vje5b/okta.egg-info/requires.txt
    writing top-level names to /tmp/pip-pip-egg-info-wf5vje5b/okta.egg-info/top_level.txt
    writing manifest file '/tmp/pip-pip-egg-info-wf5vje5b/okta.egg-info/SOURCES.txt'
    reading manifest file '/tmp/pip-pip-egg-info-wf5vje5b/okta.egg-info/SOURCES.txt'
    writing manifest file '/tmp/pip-pip-egg-info-wf5vje5b/okta.egg-info/SOURCES.txt'
    Running command python setup.py egg_info
    running egg_info
    creating /tmp/pip-pip-egg-info-8ei_5q67/parse.egg-info
    writing /tmp/pip-pip-egg-info-8ei_5q67/parse.egg-info/PKG-INFO
    writing dependency_links to /tmp/pip-pip-egg-info-8ei_5q67/parse.egg-info/dependency_links.txt
    writing top-level names to /tmp/pip-pip-egg-info-8ei_5q67/parse.egg-info/top_level.txt
    writing manifest file '/tmp/pip-pip-egg-info-8ei_5q67/parse.egg-info/SOURCES.txt'
    reading manifest file '/tmp/pip-pip-egg-info-8ei_5q67/parse.egg-info/SOURCES.txt'
    reading manifest template 'MANIFEST.in'
    writing manifest file '/tmp/pip-pip-egg-info-8ei_5q67/parse.egg-info/SOURCES.txt'
    Running command python setup.py egg_info
    running egg_info
    creating /tmp/pip-pip-egg-info-xuzjae6m/pytest_icdiff.egg-info
    writing /tmp/pip-pip-egg-info-xuzjae6m/pytest_icdiff.egg-info/PKG-INFO
    writing dependency_links to /tmp/pip-pip-egg-info-xuzjae6m/pytest_icdiff.egg-info/dependency_links.txt
    writing entry points to /tmp/pip-pip-egg-info-xuzjae6m/pytest_icdiff.egg-info/entry_points.txt
    writing requirements to /tmp/pip-pip-egg-info-xuzjae6m/pytest_icdiff.egg-info/requires.txt
    writing top-level names to /tmp/pip-pip-egg-info-xuzjae6m/pytest_icdiff.egg-info/top_level.txt
    writing manifest file '/tmp/pip-pip-egg-info-xuzjae6m/pytest_icdiff.egg-info/SOURCES.txt'
    reading manifest file '/tmp/pip-pip-egg-info-xuzjae6m/pytest_icdiff.egg-info/SOURCES.txt'
    reading manifest template 'MANIFEST.in'
    warning: no previously-included files matching '__pycache__' found under directory '*'
    warning: no previously-included files matching '*.py[co]' found under directory '*'
    writing manifest file '/tmp/pip-pip-egg-info-xuzjae6m/pytest_icdiff.egg-info/SOURCES.txt'
    Running command python setup.py egg_info
    running egg_info
    creating /tmp/pip-pip-egg-info-y54uai4r/python_constraint.egg-info
    writing /tmp/pip-pip-egg-info-y54uai4r/python_constraint.egg-info/PKG-INFO
    writing dependency_links to /tmp/pip-pip-egg-info-y54uai4r/python_constraint.egg-info/dependency_links.txt
    writing requirements to /tmp/pip-pip-egg-info-y54uai4r/python_constraint.egg-info/requires.txt
    writing top-level names to /tmp/pip-pip-egg-info-y54uai4r/python_constraint.egg-info/top_level.txt
    writing manifest file '/tmp/pip-pip-egg-info-y54uai4r/python_constraint.egg-info/SOURCES.txt'
    reading manifest file '/tmp/pip-pip-egg-info-y54uai4r/python_constraint.egg-info/SOURCES.txt'
    writing manifest file '/tmp/pip-pip-egg-info-y54uai4r/python_constraint.egg-info/SOURCES.txt'
    Running command python setup.py egg_info
    running egg_info
    creating /tmp/pip-pip-egg-info-3fgpamhl/qiskit.egg-info
    writing /tmp/pip-pip-egg-info-3fgpamhl/qiskit.egg-info/PKG-INFO
    writing dependency_links to /tmp/pip-pip-egg-info-3fgpamhl/qiskit.egg-info/dependency_links.txt
    writing requirements to /tmp/pip-pip-egg-info-3fgpamhl/qiskit.egg-info/requires.txt
    writing top-level names to /tmp/pip-pip-egg-info-3fgpamhl/qiskit.egg-info/top_level.txt
    writing manifest file '/tmp/pip-pip-egg-info-3fgpamhl/qiskit.egg-info/SOURCES.txt'
    reading manifest file '/tmp/pip-pip-egg-info-3fgpamhl/qiskit.egg-info/SOURCES.txt'
    writing manifest file '/tmp/pip-pip-egg-info-3fgpamhl/qiskit.egg-info/SOURCES.txt'
    Running command python setup.py egg_info
    running egg_info
    creating /tmp/pip-pip-egg-info-n1rcbm1f/qubovert.egg-info
    writing /tmp/pip-pip-egg-info-n1rcbm1f/qubovert.egg-info/PKG-INFO
    writing dependency_links to /tmp/pip-pip-egg-info-n1rcbm1f/qubovert.egg-info/dependency_links.txt
    writing requirements to /tmp/pip-pip-egg-info-n1rcbm1f/qubovert.egg-info/requires.txt
    writing top-level names to /tmp/pip-pip-egg-info-n1rcbm1f/qubovert.egg-info/top_level.txt
    writing manifest file '/tmp/pip-pip-egg-info-n1rcbm1f/qubovert.egg-info/SOURCES.txt'
    reading manifest file '/tmp/pip-pip-egg-info-n1rcbm1f/qubovert.egg-info/SOURCES.txt'
    reading manifest template 'MANIFEST.in'
    writing manifest file '/tmp/pip-pip-egg-info-n1rcbm1f/qubovert.egg-info/SOURCES.txt'
    Running command python setup.py egg_info
    running egg_info
    creating /tmp/pip-pip-egg-info-4q1cw68m/starkbank_ecdsa.egg-info
    writing /tmp/pip-pip-egg-info-4q1cw68m/starkbank_ecdsa.egg-info/PKG-INFO
    writing dependency_links to /tmp/pip-pip-egg-info-4q1cw68m/starkbank_ecdsa.egg-info/dependency_links.txt
    writing top-level names to /tmp/pip-pip-egg-info-4q1cw68m/starkbank_ecdsa.egg-info/top_level.txt
    writing manifest file '/tmp/pip-pip-egg-info-4q1cw68m/starkbank_ecdsa.egg-info/SOURCES.txt'
    reading manifest file '/tmp/pip-pip-egg-info-4q1cw68m/starkbank_ecdsa.egg-info/SOURCES.txt'
    reading manifest template 'MANIFEST.in'
    warning: no files found matching 'LICENSE.txt'
    no previously-included directories found matching 'tests'
    writing manifest file '/tmp/pip-pip-egg-info-4q1cw68m/starkbank_ecdsa.egg-info/SOURCES.txt'
    Running command python setup.py egg_info
    running egg_info
    creating /tmp/pip-pip-egg-info-nlywm5z3/yattag.egg-info
    writing /tmp/pip-pip-egg-info-nlywm5z3/yattag.egg-info/PKG-INFO
    writing dependency_links to /tmp/pip-pip-egg-info-nlywm5z3/yattag.egg-info/dependency_links.txt
    writing requirements to /tmp/pip-pip-egg-info-nlywm5z3/yattag.egg-info/requires.txt
    writing top-level names to /tmp/pip-pip-egg-info-nlywm5z3/yattag.egg-info/top_level.txt
    writing manifest file '/tmp/pip-pip-egg-info-nlywm5z3/yattag.egg-info/SOURCES.txt'
    reading manifest file '/tmp/pip-pip-egg-info-nlywm5z3/yattag.egg-info/SOURCES.txt'
    writing manifest file '/tmp/pip-pip-egg-info-nlywm5z3/yattag.egg-info/SOURCES.txt'
...

That says: lots of sdists being built to extract dependency metadata during backtracking. So there are:

  1. Lots of sdists in the resolve.
  2. The pip-2020-resolver in play which needs to backtrack to find a correct solution (pip-legacy-resolver will not backtrack and potentaily find an incorrect solution, but much faster).
jsirois commented 2 years ago

Even the legacy resolver is slow, but it does terminate:

$ time python -mpex.cli lock create -vvv --output=lock.json --no-emit-warnings --style=universal --resolver-version pip-legacy-resolver --indent=2 --no-pypi --index=https://pypi.org/simple/ --manylinux manylinux2014 --interpreter-constraint "CPython==3.9.*" APScheduler==3.8.1 Babel==2.9.1 Cython==0.29.24 Deprecated==1.2.13 Faker==13.3.4 Flask-Admin==1.5.8 Flask-SQLAlchemy-Session==1.1 Flask-SQLAlchemy==2.5.1 Flask-WTF==0.15.1 Flask==2.0.1 Jinja2==3.0.1 Mako==1.2.0 MarkupSafe==2.0.1 Pillow==8.4.0 PyGithub==1.55 PyJWT==2.3.0 PyNaCl==1.4.0 PySocks==1.7.1 PyYAML==5.4.1 Pygments==2.11.2 Quandl==3.7.0 Quart==0.15.1 SQLAlchemy-Paginator==0.2 SQLAlchemy-Utils==0.38.2 SQLAlchemy==1.3.24 Send2Trash==1.8.0 Sphinx==4.2.0 UnleashClient==4.4.1 WTForms-SQLAlchemy==0.2 WTForms==2.3.3 Werkzeug==2.0.3 aiofiles==0.8.0 aiohttp==3.7.4.post0 aioredis==1.3.1 alabaster==0.7.12 alembic-verify==0.1.4 alembic==1.7.3 amazon-braket-default-simulator==1.5.0 amazon-braket-ocean-plugin==1.0.8 amazon-braket-schemas==1.7.1 amazon-braket-sdk==1.15.0 amqp==5.1.0 anyio==3.4.0 appdirs==1.4.4 argon2-cffi==21.1.0 arrow==1.2.1 asttokens==2.0.5 async-timeout==3.0.1 attrs==21.2.0 auth0-python==3.18.0 backcall==0.2.0 backoff==1.11.1 backports.entry-points-selectable==1.1.1 bcrypt==3.2.0 billiard==3.6.4.0 black==21.9b0 bleach==4.1.0 blinker==1.4 boltons==21.0.0 boto3==1.21.1 botocore==1.24.1 celery==5.1.2 certifi==2021.10.8 cffi==1.15.0 cfgv==3.3.1 chardet==4.0.0 charset-normalizer==2.0.8 click-didyoumean==0.3.0 click-plugins==1.1.1 click-repl==0.2.0 click==7.1.2 colorama==0.4.4 coverage==5.5 cryptography==36.0.0 cycler==0.11.0 debugpy==1.5.1 decorator==4.4.2 deepmerge==1.0.1 defusedxml==0.7.1 dill==0.3.4 dimod==0.10.6 diskcache==5.2.1 distlib==0.3.3 distro==1.6.0 dlx==1.0.4 dnspython==2.2.1 dockerpty==0.4.1 docopt==0.6.2 docplex==2.23.222 docutils==0.17.1 dogpile.cache==1.1.4 dwave-cloud-client==0.9.2 dwave-networkx==0.8.10 dwave-preprocessing==0.3.1.post0 dwave-qbsolv==0.3.3.post0 dwave-system==1.9.0 dwave-tabu==0.4.2 email-validator==1.1.3 entrypoints==0.3 execnet==1.9.0 factory-boy==3.2.0 fastdtw==0.3.4 fasteners==0.16.3 fastjsonschema==2.15.1 fcache==0.4.7 fenestrate==0.3.0 filelock==3.4.0 flask-oidc==1.4.0 gunicorn==20.1.0 h11==0.13.0 h2==4.1.0 h5py==3.2.1 hiredis==2.0.0 homebase==1.0.1 hpack==4.0.0 httplib2==0.20.4 hypercorn==0.13.2 hyperframe==6.0.1 hypothesis==6.21.6 icdiff==2.0.4 icontract==2.6.0 identify==2.4.0 idna==3.3 imagesize==1.3.0 importlib-metadata==4.8.1 inflection==0.5.1 iniconfig==1.1.1 intervaltree==3.1.0 invoke==1.6.0 ipykernel==6.5.1 ipython-genutils==0.2.0 ipython==7.30.0 ipywidgets==7.6.5 itsdangerous==2.0.1 jedi==0.18.1 jinja2-time==0.2.0 jmespath==0.10.0 joblib==1.1.0 json5==0.9.6 jsonref==0.2 jsonschema==3.2.0 jupyter-client==7.1.0 jupyter-core==4.9.1 jupyter-packaging==0.7.9 jupyter-server==1.12.1 jupyterlab-pygments==0.1.2 jupyterlab-server==2.8.2 jupyterlab-widgets==1.1.0 jupyterlab==3.1.12 kiwisolver==1.3.2 kombu==5.2.4 llvmlite==0.36.0 lxml==4.8.0 lz4==3.1.3 markdown2==2.4.1 matplotlib-inline==0.1.3 matplotlib==3.4.3 memory-profiler==0.58.0 minorminer==0.2.6 mistune==0.8.4 mmh3==3.0.0 more-itertools==8.12.0 moto==2.2.7 mpmath==1.2.1 multidict==5.2.0 multitasking==0.0.10 mypy-extensions==0.4.3 nbclassic==0.3.4 nbclient==0.5.9 nbconvert==6.3.0 nbformat==5.1.3 nbmake==1.0 nest-asyncio==1.5.1 networkx==2.6.3 nodeenv==1.6.0 notebook==6.4.6 ntlm-auth==1.5.0 numba==0.53.1 numpy==1.21.2 oauth2client==4.1.3 okta==0.0.4 opentelemetry-api==1.8.0 opentelemetry-exporter-zipkin-json==1.8.0 opentelemetry-exporter-zipkin-proto-http==1.8.0 opentelemetry-exporter-zipkin==1.8.0 opentelemetry-instrumentation-sqlalchemy==0.27b0 opentelemetry-instrumentation==0.27b0 opentelemetry-sdk==1.8.0 opentelemetry-semantic-conventions==0.27b0 opt-einsum==3.3.0 packaging==21.0 pandas==1.3.4 pandocfilters==1.5.0 paramiko==2.8.1 parse==1.19.0 parso==0.8.2 pathspec==0.9.0 pbr==5.8.1 pexpect==4.8.0 pickleshare==0.7.5 platformdirs==2.4.0 plucky==0.4.3 pluggy==1.0.0 ply==3.11 pprintpp==0.4.0 pre-commit==2.15.0 priority==2.0.0 prometheus-client==0.12.0 prompt-toolkit==3.0.23 protobuf==3.19.4 psutil==5.8.0 psycopg2-binary==2.9.2 ptyprocess==0.7.0 py==1.11.0 pyasn1-modules==0.2.8 pyasn1==0.4.8 pybind11==2.8.1 pycodestyle==2.7.0 pycparser==2.21 pydantic==1.8.2 pynvml==11.0.0 pyparsing==3.0.6 pyrsistent==0.18.0 pytest-asyncio==0.15.1 pytest-cov==3.0.0 pytest-forked==1.3.0 pytest-icdiff==0.5 pytest-mock==3.6.1 pytest-repeat==0.9.1 pytest-xdist==2.4.0 pytest==6.2.5 python-constraint==1.4.0 python-dateutil==2.8.2 python-decouple==3.4 python-dotenv==0.19.2 python-http-client==3.3.7 python-lsp-black==1.0.0 python-lsp-jsonrpc==1.0.0 python-lsp-server==1.3.2 pytz-deprecation-shim==0.1.0.post0 pytz==2021.3 pyzmq==22.3.0 qcware-quasar==1.0.6 qcware-transpile==0.1.1a19 qiskit-aer==0.9.0 qiskit-aqua==0.9.5 qiskit-ibmq-provider==0.16.0 qiskit-ignis==0.6.0 qiskit-terra==0.18.3 qiskit==0.30.1 qubovert==1.2.3 redis==3.5.3 regex==2021.11.10 requests-ntlm==1.1.0 requests==2.26.0 responses==0.18.0 retworkx==0.10.2 rsa==4.8 s3transfer==0.5.0 scikit-learn==0.24.2 scipy==1.7.1 sendgrid==6.8.1 setuptools==58.0.4 six==1.16.0 sniffio==1.2.0 snowballstemmer==2.2.0 sortedcontainers==2.4.0 sphinx-rtd-theme==1.0.0 sphinxcontrib-applehelp==1.0.2 sphinxcontrib-devhelp==1.0.2 sphinxcontrib-htmlhelp==2.0.0 sphinxcontrib-jsmath==1.0.1 sphinxcontrib-qthelp==1.0.3 sphinxcontrib-serializinghtml==1.1.5 sqlalchemy-diff==0.1.5 starkbank-ecdsa==2.0.3 stevedore==3.5.0 structlog==21.5.0 symengine==0.8.1 sympy==1.9 tabulate==0.8.9 terminado==0.12.1 testpath==0.5.0 texttable==1.6.4 threadpoolctl==3.0.0 toml==0.10.2 tomli==1.2.2 toolz==0.11.2 torch==1.10.2 tornado==6.1 tqdm==4.62.3 traitlets==5.1.1 tweedledum==1.1.1 types-requests==2.25.8 typing_extensions==4.0.0 tzdata==2021.5 tzlocal==4.1 ujson==4.3.0 unitair==0.1.6 urllib3==1.26.7 vine==5.0.0 virtualenv==20.10.0 watchdog==2.1.5 wcwidth==0.2.5 webencodings==0.5.1 websocket-client==1.3.2 wheel==0.37.0 widgetsnbextension==3.5.2 wrapt==1.13.3 wsproto==1.1.0 xmltodict==0.12.0 yapf==0.31.0 yarl==1.7.2 yattag==1.14.0 yfinance==0.1.70 zipp==3.7.0
pex: Resolving for:
pex: Hashing pex: 17.0ms                                                 
pex: Isolating pex: 0.0ms
pex: Calculating compatible python versions for ('==3.9.*',): 5.5ms
...
real    4m55.927s
user    1m25.957s
sys 0m8.694s
jsirois commented 2 years ago

Ok, the missing data, approximated (presumably it would take even longer if we handed Poetry the root requirement set and not all the pinned transitive versions):

$ poetry init -q -n --name poetry_lock_timing --author "John Sirois <john.sirois@gmail.com>" --python "==3.9.*"
$ rm -rf ~/.cache/pypoetry && jq -r .requirements[] ~/dev/pantsbuild/pex/lock.json | time xargs ~/.venv/poetry/bin/poetry add
The currently activated Python version 3.10.2 is not supported by the project (==3.9.*).
Trying to find and use a compatible version. 
Using python3.9 (3.9.10)
Creating virtualenv poetry-lock-timing-B7fwsm4C-py3.9 in /home/jsirois/.cache/pypoetry/virtualenvs

Updating dependencies
Resolving dependencies... (351.3s)

Writing lock file

Package operations: 314 installs, 0 updates, 0 removals

  • Installing attrs (21.2.0)
  • Installing pyrsistent (0.18.0)
  • Installing six (1.16.0)
  • Installing traitlets (5.1.1)
  • Installing entrypoints (0.3)
  • Installing ipython-genutils (0.2.0)
  • Installing jsonschema (3.2.0)
  • Installing jupyter-core (4.9.1)
  • Installing nest-asyncio (1.5.1)
  • Installing parso (0.8.2)
  • Installing ptyprocess (0.7.0)
  • Installing pyparsing (3.0.6)
  • Installing python-dateutil (2.8.2)
  • Installing pyzmq (22.3.0)
  • Installing tornado (6.1)
  • Installing wcwidth (0.2.5)
  • Installing wrapt (1.13.3)
  • Installing backcall (0.2.0)
  • Installing decorator (4.4.2)
  • Installing deprecated (1.2.13)
  • Installing jedi (0.18.1)
  • Installing jmespath (0.10.0)
  • Installing jupyter-client (7.1.0)
  • Installing markupsafe (2.0.1)
  • Installing matplotlib-inline (0.1.3)
  • Installing mpmath (1.2.1)
  • Installing nbformat (5.1.3)
  • Installing numpy (1.21.2)
  • Installing packaging (21.0)
  • Installing pexpect (4.8.0)
  • Installing pickleshare (0.7.5)
  • Installing prompt-toolkit (3.0.23)
  • Installing pycparser (2.21)
  • Installing pygments (2.11.2)
  • Installing typing-extensions (4.0.0)
  • Installing urllib3 (1.26.7)
  • Installing webencodings (0.5.1)
  • Installing bleach (4.1.0)
  • Installing botocore (1.24.1)
  • Installing certifi (2021.10.8)
  • Installing cffi (1.15.0)
  • Installing charset-normalizer (2.0.8)
  • Installing debugpy (1.5.1)
  • Installing defusedxml (0.7.1)
  • Installing dill (0.3.4)
  • Installing dwave-preprocessing (0.3.1.post0)
  • Installing fastjsonschema (2.15.1)
  • Installing idna (3.3)
  • Installing ipython (7.30.0)
  • Installing jinja2 (3.0.1)
  • Installing jupyterlab-pygments (0.1.2)
  • Installing mistune (0.8.4)
  • Installing nbclient (0.5.9)
  • Installing opentelemetry-api (1.8.0)
  • Installing opentelemetry-semantic-conventions (0.27b0)
  • Installing pandocfilters (1.5.0)
  • Installing ply (3.11)
  • Installing psutil (5.8.0)
  • Installing pydantic (1.8.2)
  • Installing python-constraint (1.4.0)
  • Installing pytz (2021.3)
  • Installing retworkx (0.10.2)
  • Installing scipy (1.7.1)
  • Installing sniffio (1.2.0)
  • Installing symengine (0.8.1)
  • Installing sympy (1.9)
  • Installing testpath (0.5.0)
  • Installing tweedledum (1.1.1)
  • Installing tzdata (2021.5)
  • Installing amazon-braket-schemas (1.7.1)
  • Installing anyio (3.4.0)
  • Installing argon2-cffi (21.1.0)
  • Installing cryptography (36.0.0)
  • Installing dimod (0.10.6)
  • Installing h11 (0.13.0)
  • Installing hpack (4.0.0)
  • Installing hyperframe (6.0.1)
  • Installing inflection (0.5.1)
  • Installing iniconfig (1.1.1)
  • Installing ipykernel (6.5.1)
  • Installing joblib (1.1.0)
  • Installing lxml (4.8.0)
  • Installing more-itertools (8.12.0)
  • Installing multitasking (0.0.10)
  • Installing nbconvert (6.3.0)
  • Installing networkx (2.6.3)
  • Installing ntlm-auth (1.5.0)
  • Installing opentelemetry-sdk (1.8.0)
  • Installing opt-einsum (3.3.0)
  • Installing pandas (1.3.4)
  • Installing pluggy (1.0.0)
  • Installing prometheus-client (0.12.0)
  • Installing py (1.11.0)
  • Installing pyasn1 (0.4.8)
  • Installing pysocks (1.7.1)
  • Installing pytz-deprecation-shim (0.1.0.post0)
  • Installing qiskit-terra (0.18.3)
  • Installing requests (2.26.0)
  • Installing s3transfer (0.5.0)
  • Installing send2trash (1.8.0)
  • Installing sqlalchemy (1.3.24)
  • Installing terminado (0.12.1)
  • Installing threadpoolctl (3.0.0)
  • Installing toml (0.10.2)
  • Installing ujson (4.3.0)
  • Installing vine (5.0.0)
  • Installing websocket-client (1.3.2)
  • Installing alabaster (0.7.12)
  • Installing amazon-braket-default-simulator (1.5.0)
  • Installing amqp (5.1.0)
  • Installing appdirs (1.4.4)
  • Installing asttokens (2.0.5)
  • Installing babel (2.9.1): Installing...
  • Installing backoff (1.11.1)
  • Installing backports.entry-points-selectable (1.1.1)
  • Installing boltons (21.0.0)
  • Installing boto3 (1.21.1)
  • Installing click (7.1.2)
  • Installing coverage (5.5)
  • Installing diskcache (5.2.1)
  • Installing distlib (0.3.3)
  • Installing dlx (1.0.4)
  • Installing docplex (2.23.222): Installing...
  • Installing docutils (0.17.1)
  • Installing backoff (1.11.1)
  • Installing backports.entry-points-selectable (1.1.1)
  • Installing boltons (21.0.0)
  • Installing boto3 (1.21.1)
  • Installing click (7.1.2)
  • Installing coverage (5.5)
  • Installing diskcache (5.2.1)
  • Installing distlib (0.3.3)
  • Installing dlx (1.0.4)
  • Installing docplex (2.23.222): Installing...
  • Installing docutils (0.17.1)
  • Installing babel (2.9.1)
  • Installing backoff (1.11.1)
  • Installing backports.entry-points-selectable (1.1.1)
  • Installing boltons (21.0.0)
  • Installing boto3 (1.21.1)
  • Installing click (7.1.2)
  • Installing coverage (5.5)
  • Installing diskcache (5.2.1)
  • Installing distlib (0.3.3)
  • Installing dlx (1.0.4)
  • Installing docplex (2.23.222): Installing...
  • Installing docutils (0.17.1)
  • Installing docutils (0.17.1)
  • Installing docplex (2.23.222)
  • Installing docutils (0.17.1)
  • Installing dwave-networkx (0.8.10)
  • Installing fastdtw (0.3.4)
  • Installing fasteners (0.16.3)
  • Installing filelock (3.4.0)
  • Installing h5py (3.2.1)
  • Installing httplib2 (0.20.4)
  • Installing homebase (1.0.1)
  • Installing h2 (4.1.0)
  • Installing imagesize (1.3.0)
  • Installing itsdangerous (2.0.1)
  • Installing json5 (0.9.6)
  • Installing jupyter-server (1.12.1)
  • Installing mako (1.2.0)
  • Installing multidict (5.2.0)
  • Installing mypy-extensions (0.4.3)
  • Installing notebook (6.4.6)
  • Installing opentelemetry-exporter-zipkin-json (1.8.0)
  • Installing pathspec (0.9.0)
  • Installing pbr (5.8.1)
  • Installing platformdirs (2.4.0)
  • Installing plucky (0.4.3)
  • Installing priority (2.0.0)
  • Installing protobuf (3.19.4)
  • Installing pyasn1-modules (0.2.8)
  • Installing pybind11 (2.8.1)
  • Installing pytest (6.2.5)
  • Installing python-lsp-jsonrpc (1.0.0)
  • Installing qiskit-ignis (0.6.0)
  • Installing quandl (3.7.0)
  • Installing regex (2021.11.10)
  • Installing requests-ntlm (1.1.0)
  • Installing rsa (4.8)
  • Installing scikit-learn (0.24.2)
  • Installing snowballstemmer (2.2.0)
  • Installing sortedcontainers (2.4.0)
  • Installing sphinxcontrib-applehelp (1.0.2)
  • Installing sphinxcontrib-devhelp (1.0.2)
  • Installing sphinxcontrib-htmlhelp (2.0.0)
  • Installing sphinxcontrib-jsmath (1.0.1)
  • Installing sphinxcontrib-qthelp (1.0.3)
  • Installing sphinxcontrib-serializinghtml (1.1.5)
  • Installing sqlalchemy-utils (0.38.2)
  • Installing tomli (1.2.2)
  • Installing tzlocal (4.1)
  • Installing werkzeug (2.0.3)
  • Installing wsproto (1.1.0)
  • Installing yfinance (0.1.70)
  • Installing aiofiles (0.8.0)
  • Installing alembic (1.7.3)
  • Installing amazon-braket-sdk (1.15.0)
  • Installing apscheduler (3.8.1)
  • Installing arrow (1.2.1)
  • Installing async-timeout (3.0.1)
  • Installing bcrypt (3.2.0)
  • Installing billiard (3.6.4.0)
  • Installing black (21.9b0)
  • Installing blinker (1.4)
  • Installing cfgv (3.3.1)
  • Installing chardet (4.0.0)
  • Installing click-didyoumean (0.3.0)
  • Installing click-plugins (1.1.1)
  • Installing click-repl (0.2.0)
  • Installing colorama (0.4.4)
  • Installing cycler (0.11.0)
  • Installing dnspython (2.2.1)
  • Installing dwave-cloud-client (0.9.2)
  • Installing dwave-tabu (0.4.2)
  • Installing execnet (1.9.0)
  • Installing faker (13.3.4)
  • Installing fcache (0.4.7)
  • Installing flask (2.0.1)
  • Installing hiredis (2.0.0)
  • Installing hypercorn (0.13.2)
  • Installing icdiff (2.0.4)
  • Installing icontract (2.6.0)
  • Installing identify (2.4.0)
  • Installing intervaltree (3.1.0)
  • Installing jsonref (0.2)
  • Installing jupyterlab-server (2.8.2)
  • Installing jupyterlab-widgets (1.1.0)
  • Installing kiwisolver (1.3.2)
  • Installing kombu (5.2.4)
  • Installing llvmlite (0.36.0)
  • Installing minorminer (0.2.6)
  • Installing mmh3 (3.0.0)
  • Installing nbclassic (0.3.4)
  • Installing nodeenv (1.6.0)
  • Installing oauth2client (4.1.3)
  • Installing opentelemetry-exporter-zipkin-proto-http (1.8.0)
  • Installing opentelemetry-instrumentation (0.27b0)
  • Installing parse (1.19.0)
  • Installing pillow (8.4.0)
  • Installing pprintpp (0.4.0)
  • Installing pyjwt (2.3.0)
  • Installing pynacl (1.4.0)
  • Installing pytest-forked (1.3.0)
  • Installing python-http-client (3.3.7)
  • Installing python-lsp-server (1.3.2)
  • Installing pyyaml (5.4.1)
  • Installing qiskit-aer (0.9.0)
  • Installing qiskit-aqua (0.9.5)
  • Installing qiskit-ibmq-provider (0.16.0)
  • Installing responses (0.18.0)
  • Installing sphinx (4.2.0)
  • Installing sqlalchemy-diff (0.1.5)
  • Installing starkbank-ecdsa (2.0.3)
  • Installing stevedore (3.5.0)
  • Installing toolz (0.11.2)
  • Installing torch (1.10.2)
  • Installing virtualenv (20.10.0)
  • Installing widgetsnbextension (3.5.2)
  • Installing wtforms (2.3.3)
  • Installing xmltodict (0.12.0)
  • Installing yarl (1.7.2)
  • Installing zipp (3.7.0)
  • Installing aiohttp (3.7.4.post0)
  • Installing aioredis (1.3.1)
  • Installing alembic-verify (0.1.4)
  • Installing amazon-braket-ocean-plugin (1.0.8)
  • Installing auth0-python (3.18.0)
  • Installing celery (5.1.2)
  • Installing cython (0.29.24)
  • Installing deepmerge (1.0.1)
  • Installing distro (1.6.0)
  • Installing dockerpty (0.4.1)
  • Installing docopt (0.6.2)
  • Installing dogpile.cache (1.1.4)
  • Installing dwave-qbsolv (0.3.3.post0)
  • Installing dwave-system (1.9.0)
  • Installing email-validator (1.1.3)
  • Installing factory-boy (3.2.0)
  • Installing fenestrate (0.3.0)
  • Installing flask-admin (1.5.8): Installing...
  • Installing flask-oidc (1.4.0)
  • Installing flask-sqlalchemy (2.5.1)
  • Installing flask-sqlalchemy-session (1.1)
  • Installing flask-wtf (0.15.1)
  • Installing gunicorn (20.1.0)
  • Installing hypothesis (6.21.6)
  • Installing importlib-metadata (4.8.1)
  • Installing invoke (1.6.0)
  • Installing ipywidgets (7.6.5)
  • Installing jinja2-time (0.2.0)
  • Installing flask-oidc (1.4.0)
  • Installing flask-sqlalchemy (2.5.1)
  • Installing flask-sqlalchemy-session (1.1)
  • Installing flask-wtf (0.15.1)
  • Installing gunicorn (20.1.0)
  • Installing hypothesis (6.21.6)
  • Installing importlib-metadata (4.8.1)
  • Installing invoke (1.6.0)
  • Installing ipywidgets (7.6.5)
  • Installing jinja2-time (0.2.0)
  • Installing flask-admin (1.5.8)
  • Installing flask-oidc (1.4.0)
  • Installing flask-sqlalchemy (2.5.1)
  • Installing flask-sqlalchemy-session (1.1)
  • Installing flask-wtf (0.15.1)
  • Installing gunicorn (20.1.0)
  • Installing hypothesis (6.21.6)
  • Installing importlib-metadata (4.8.1)
  • Installing invoke (1.6.0)
  • Installing ipywidgets (7.6.5)
  • Installing jinja2-time (0.2.0)
  • Installing jupyter-packaging (0.7.9)
  • Installing jupyterlab (3.1.12)
  • Installing lz4 (3.1.3)
  • Installing markdown2 (2.4.1)
  • Installing matplotlib (3.4.3)
  • Installing memory-profiler (0.58.0)
  • Installing moto (2.2.7)
  • Installing nbmake (1.0)
  • Installing numba (0.53.1)
  • Installing okta (0.0.4)
  • Installing opentelemetry-exporter-zipkin (1.8.0)
  • Installing opentelemetry-instrumentation-sqlalchemy (0.27b0)
  • Installing paramiko (2.8.1)
  • Installing pre-commit (2.15.0)
  • Installing psycopg2-binary (2.9.2)
  • Installing pycodestyle (2.7.0)
  • Installing pygithub (1.55)
  • Installing pynvml (11.0.0)
  • Installing pytest-asyncio (0.15.1)
  • Installing pytest-cov (3.0.0)
  • Installing pytest-icdiff (0.5)
  • Installing pytest-mock (3.6.1)
  • Installing pytest-repeat (0.9.1)
  • Installing pytest-xdist (2.4.0)
  • Installing python-decouple (3.4)
  • Installing python-dotenv (0.19.2)
  • Installing python-lsp-black (1.0.0)
  • Installing qcware-quasar (1.0.6)
  • Installing qcware-transpile (0.1.1a19)
  • Installing qiskit (0.30.1)
  • Installing quart (0.15.1)
  • Installing qubovert (1.2.3)
  • Installing redis (3.5.3)
  • Installing sendgrid (6.8.1)
  • Installing sphinx-rtd-theme (1.0.0)
  • Installing sqlalchemy-paginator (0.2)
  • Installing structlog (21.5.0)
  • Installing tabulate (0.8.9)
  • Installing texttable (1.6.4)
  • Installing tqdm (4.62.3)
  • Installing types-requests (2.25.8)
  • Installing unitair (0.1.6)
  • Installing unleashclient (4.4.1)
  • Installing watchdog (2.1.5)
  • Installing wtforms-sqlalchemy (0.2)
  • Installing yapf (0.31.0)
  • Installing yattag (1.14.0)
764.98user 75.88system 11:34.37elapsed 121%CPU (0avgtext+0avgdata 399600maxresident)k
65600inputs+10051704outputs (1187major+16455519minor)pagefaults 0swaps

So, 11.5 minutes is our baseline here from Poetry.

Eric-Arellano commented 2 years ago

If you have not already tried, I'm curious if pip alone can resolve this in a timely manner.

jsirois commented 2 years ago 
rm -rf ~/.cache/pip && PYTHONPATH=pex/vendor/_vendored/pip time python3.9 -mpip download --dest /tmp/pip-download --use-feature 2020-resolver APScheduler==3.8.1 Babel==2.9.1 Cython==0.29.24 Deprecated==1.2.13 Faker==13.3.4 Flask-Admin==1.5.8 Flask-SQLAlchemy-Session==1.1 Flask-SQLAlchemy==2.5.1 Flask-WTF==0.15.1 Flask==2.0.1 Jinja2==3.0.1 Mako==1.2.0 MarkupSafe==2.0.1 Pillow==8.4.0 PyGithub==1.55 PyJWT==2.3.0 PyNaCl==1.4.0 PySocks==1.7.1 PyYAML==5.4.1 Pygments==2.11.2 Quandl==3.7.0 Quart==0.15.1 SQLAlchemy-Paginator==0.2 SQLAlchemy-Utils==0.38.2 SQLAlchemy==1.3.24 Send2Trash==1.8.0 Sphinx==4.2.0 UnleashClient==4.4.1 WTForms-SQLAlchemy==0.2 WTForms==2.3.3 Werkzeug==2.0.3 aiofiles==0.8.0 aiohttp==3.7.4.post0 aioredis==1.3.1 alabaster==0.7.12 alembic-verify==0.1.4 alembic==1.7.3 amazon-braket-default-simulator==1.5.0 amazon-braket-ocean-plugin==1.0.8 amazon-braket-schemas==1.7.1 amazon-braket-sdk==1.15.0 amqp==5.1.0 anyio==3.4.0 appdirs==1.4.4 argon2-cffi==21.1.0 arrow==1.2.1 asttokens==2.0.5 async-timeout==3.0.1 attrs==21.2.0 auth0-python==3.18.0 backcall==0.2.0 backoff==1.11.1 backports.entry-points-selectable==1.1.1 bcrypt==3.2.0 billiard==3.6.4.0 black==21.9b0 bleach==4.1.0 blinker==1.4 boltons==21.0.0 boto3==1.21.1 botocore==1.24.1 celery==5.1.2 certifi==2021.10.8 cffi==1.15.0 cfgv==3.3.1 chardet==4.0.0 charset-normalizer==2.0.8 click-didyoumean==0.3.0 click-plugins==1.1.1 click-repl==0.2.0 click==7.1.2 colorama==0.4.4 coverage==5.5 cryptography==36.0.0 cycler==0.11.0 debugpy==1.5.1 decorator==4.4.2 deepmerge==1.0.1 defusedxml==0.7.1 dill==0.3.4 dimod==0.10.6 diskcache==5.2.1 distlib==0.3.3 distro==1.6.0 dlx==1.0.4 dnspython==2.2.1 dockerpty==0.4.1 docopt==0.6.2 docplex==2.23.222 docutils==0.17.1 dogpile.cache==1.1.4 dwave-cloud-client==0.9.2 dwave-networkx==0.8.10 dwave-preprocessing==0.3.1.post0 dwave-qbsolv==0.3.3.post0 dwave-system==1.9.0 dwave-tabu==0.4.2 email-validator==1.1.3 entrypoints==0.3 execnet==1.9.0 factory-boy==3.2.0 fastdtw==0.3.4 fasteners==0.16.3 fastjsonschema==2.15.1 fcache==0.4.7 fenestrate==0.3.0 filelock==3.4.0 flask-oidc==1.4.0 gunicorn==20.1.0 h11==0.13.0 h2==4.1.0 h5py==3.2.1 hiredis==2.0.0 homebase==1.0.1 hpack==4.0.0 httplib2==0.20.4 hypercorn==0.13.2 hyperframe==6.0.1 hypothesis==6.21.6 icdiff==2.0.4 icontract==2.6.0 identify==2.4.0 idna==3.3 imagesize==1.3.0 importlib-metadata==4.8.1 inflection==0.5.1 iniconfig==1.1.1 intervaltree==3.1.0 invoke==1.6.0 ipykernel==6.5.1 ipython-genutils==0.2.0 ipython==7.30.0 ipywidgets==7.6.5 itsdangerous==2.0.1 jedi==0.18.1 jinja2-time==0.2.0 jmespath==0.10.0 joblib==1.1.0 json5==0.9.6 jsonref==0.2 jsonschema==3.2.0 jupyter-client==7.1.0 jupyter-core==4.9.1 jupyter-packaging==0.7.9 jupyter-server==1.12.1 jupyterlab-pygments==0.1.2 jupyterlab-server==2.8.2 jupyterlab-widgets==1.1.0 jupyterlab==3.1.12 kiwisolver==1.3.2 kombu==5.2.4 llvmlite==0.36.0 lxml==4.8.0 lz4==3.1.3 markdown2==2.4.1 matplotlib-inline==0.1.3 matplotlib==3.4.3 memory-profiler==0.58.0 minorminer==0.2.6 mistune==0.8.4 mmh3==3.0.0 more-itertools==8.12.0 moto==2.2.7 mpmath==1.2.1 multidict==5.2.0 multitasking==0.0.10 mypy-extensions==0.4.3 nbclassic==0.3.4 nbclient==0.5.9 nbconvert==6.3.0 nbformat==5.1.3 nbmake==1.0 nest-asyncio==1.5.1 networkx==2.6.3 nodeenv==1.6.0 notebook==6.4.6 ntlm-auth==1.5.0 numba==0.53.1 numpy==1.21.2 oauth2client==4.1.3 okta==0.0.4 opentelemetry-api==1.8.0 opentelemetry-exporter-zipkin-json==1.8.0 opentelemetry-exporter-zipkin-proto-http==1.8.0 opentelemetry-exporter-zipkin==1.8.0 opentelemetry-instrumentation-sqlalchemy==0.27b0 opentelemetry-instrumentation==0.27b0 opentelemetry-sdk==1.8.0 opentelemetry-semantic-conventions==0.27b0 opt-einsum==3.3.0 packaging==21.0 pandas==1.3.4 pandocfilters==1.5.0 paramiko==2.8.1 parse==1.19.0 parso==0.8.2 pathspec==0.9.0 pbr==5.8.1 pexpect==4.8.0 pickleshare==0.7.5 platformdirs==2.4.0 plucky==0.4.3 pluggy==1.0.0 ply==3.11 pprintpp==0.4.0 pre-commit==2.15.0 priority==2.0.0 prometheus-client==0.12.0 prompt-toolkit==3.0.23 protobuf==3.19.4 psutil==5.8.0 psycopg2-binary==2.9.2 ptyprocess==0.7.0 py==1.11.0 pyasn1-modules==0.2.8 pyasn1==0.4.8 pybind11==2.8.1 pycodestyle==2.7.0 pycparser==2.21 pydantic==1.8.2 pynvml==11.0.0 pyparsing==3.0.6 pyrsistent==0.18.0 pytest-asyncio==0.15.1 pytest-cov==3.0.0 pytest-forked==1.3.0 pytest-icdiff==0.5 pytest-mock==3.6.1 pytest-repeat==0.9.1 pytest-xdist==2.4.0 pytest==6.2.5 python-constraint==1.4.0 python-dateutil==2.8.2 python-decouple==3.4 python-dotenv==0.19.2 python-http-client==3.3.7 python-lsp-black==1.0.0 python-lsp-jsonrpc==1.0.0 python-lsp-server==1.3.2 pytz-deprecation-shim==0.1.0.post0 pytz==2021.3 pyzmq==22.3.0 qcware-quasar==1.0.6 qcware-transpile==0.1.1a19 qiskit-aer==0.9.0 qiskit-aqua==0.9.5 qiskit-ibmq-provider==0.16.0 qiskit-ignis==0.6.0 qiskit-terra==0.18.3 qiskit==0.30.1 qubovert==1.2.3 redis==3.5.3 regex==2021.11.10 requests-ntlm==1.1.0 requests==2.26.0 responses==0.18.0 retworkx==0.10.2 rsa==4.8 s3transfer==0.5.0 scikit-learn==0.24.2 scipy==1.7.1 sendgrid==6.8.1 setuptools==58.0.4 six==1.16.0 sniffio==1.2.0 snowballstemmer==2.2.0 sortedcontainers==2.4.0 sphinx-rtd-theme==1.0.0 sphinxcontrib-applehelp==1.0.2 sphinxcontrib-devhelp==1.0.2 sphinxcontrib-htmlhelp==2.0.0 sphinxcontrib-jsmath==1.0.1 sphinxcontrib-qthelp==1.0.3 sphinxcontrib-serializinghtml==1.1.5 sqlalchemy-diff==0.1.5 starkbank-ecdsa==2.0.3 stevedore==3.5.0 structlog==21.5.0 symengine==0.8.1 sympy==1.9 tabulate==0.8.9 terminado==0.12.1 testpath==0.5.0 texttable==1.6.4 threadpoolctl==3.0.0 toml==0.10.2 tomli==1.2.2 toolz==0.11.2 torch==1.10.2 tornado==6.1 tqdm==4.62.3 traitlets==5.1.1 tweedledum==1.1.1 types-requests==2.25.8 typing_extensions==4.0.0 tzdata==2021.5 tzlocal==4.1 ujson==4.3.0 unitair==0.1.6 urllib3==1.26.7 vine==5.0.0 virtualenv==20.10.0 watchdog==2.1.5 wcwidth==0.2.5 webencodings==0.5.1 websocket-client==1.3.2 wheel==0.37.0 widgetsnbextension==3.5.2 wrapt==1.13.3 wsproto==1.1.0 xmltodict==0.12.0 yapf==0.31.0 yarl==1.7.2 yattag==1.14.0 yfinance==0.1.70 zipp==3.7.0
...
Successfully downloaded aiofiles aiohttp aioredis alabaster alembic alembic-verify amazon-braket-default-simulator amazon-braket-ocean-plugin amazon-braket-schemas amazon-braket-sdk amqp anyio appdirs APScheduler argon2-cffi arrow asttokens async-timeout attrs auth0-python Babel backcall backoff backports.entry-points-selectable bcrypt billiard black bleach blinker boltons boto3 botocore celery certifi cffi cfgv chardet charset-normalizer click click-didyoumean click-plugins click-repl colorama coverage cryptography cycler Cython debugpy decorator deepmerge defusedxml Deprecated dill dimod diskcache distlib distro dlx dnspython dockerpty docopt docplex docutils dogpile.cache dwave-cloud-client dwave-networkx dwave-preprocessing dwave-qbsolv dwave-system dwave-tabu email-validator entrypoints execnet factory-boy Faker fastdtw fasteners fastjsonschema fcache fenestrate filelock Flask Flask-Admin flask-oidc Flask-SQLAlchemy Flask-SQLAlchemy-Session Flask-WTF gunicorn h11 h2 h5py hiredis homebase hpack httplib2 hypercorn hyperframe hypothesis icdiff icontract identify idna imagesize importlib-metadata inflection iniconfig intervaltree invoke ipykernel ipython ipython-genutils ipywidgets itsdangerous jedi Jinja2 jinja2-time jmespath joblib json5 jsonref jsonschema jupyter-client jupyter-core jupyter-packaging jupyter-server jupyterlab jupyterlab-pygments jupyterlab-server jupyterlab-widgets kiwisolver kombu llvmlite lxml lz4 Mako markdown2 MarkupSafe matplotlib matplotlib-inline memory-profiler minorminer mistune mmh3 more-itertools moto mpmath multidict multitasking mypy-extensions nbclassic nbclient nbconvert nbformat nbmake nest-asyncio networkx nodeenv notebook ntlm-auth numba numpy oauth2client okta opentelemetry-api opentelemetry-exporter-zipkin opentelemetry-exporter-zipkin-json opentelemetry-exporter-zipkin-proto-http opentelemetry-instrumentation opentelemetry-instrumentation-sqlalchemy opentelemetry-sdk opentelemetry-semantic-conventions opt-einsum packaging pandas pandocfilters paramiko parse parso pathspec pbr pexpect pickleshare Pillow platformdirs plucky pluggy ply pprintpp pre-commit priority prometheus-client prompt-toolkit protobuf psutil psycopg2-binary ptyprocess py pyasn1 pyasn1-modules pybind11 pycodestyle pycparser pydantic PyGithub Pygments PyJWT PyNaCl pynvml pyparsing pyrsistent PySocks pytest pytest-asyncio pytest-cov pytest-forked pytest-icdiff pytest-mock pytest-repeat pytest-xdist python-constraint python-dateutil python-decouple python-dotenv python-http-client python-lsp-black python-lsp-jsonrpc python-lsp-server pytz pytz-deprecation-shim PyYAML pyzmq qcware-quasar qcware-transpile qiskit qiskit-aer qiskit-aqua qiskit-ibmq-provider qiskit-ignis qiskit-terra Quandl Quart qubovert redis regex requests requests-ntlm responses retworkx rsa s3transfer scikit-learn scipy Send2Trash sendgrid six sniffio snowballstemmer sortedcontainers Sphinx sphinx-rtd-theme sphinxcontrib-applehelp sphinxcontrib-devhelp sphinxcontrib-htmlhelp sphinxcontrib-jsmath sphinxcontrib-qthelp sphinxcontrib-serializinghtml SQLAlchemy sqlalchemy-diff SQLAlchemy-Paginator SQLAlchemy-Utils starkbank-ecdsa stevedore structlog symengine sympy tabulate terminado testpath texttable threadpoolctl toml tomli toolz torch tornado tqdm traitlets tweedledum types-requests typing-extensions tzdata tzlocal ujson unitair UnleashClient urllib3 vine virtualenv watchdog wcwidth webencodings websocket-client Werkzeug wheel widgetsnbextension wrapt wsproto WTForms WTForms-SQLAlchemy xmltodict yapf yarl yattag yfinance zipp setuptools
WARNING: You are using pip version 20.3.4; however, version 22.0.4 is available.
You should consider upgrading via the '/home/jsirois/.pyenv/versions/3.9.10/bin/python3.9 -m pip install --upgrade pip' command.
36.90user 7.05system 5:28.44elapsed 13%CPU (0avgtext+0avgdata 2796716maxresident)k
5952inputs+2415928outputs (20major+1202935minor)pagefaults 0swaps

So 5.5 minutes for a pinned resolve for the local interpreter using the pip-2020-resolver. That's not very interesting though. The resolve is pinned, and its for 1 interpreter. Pretty far from a universal lock.

jsirois commented 2 years ago

Its obvious this is a big resolve, and that much is inherently slow (network time), but in these tests probably only accounts for ~5 minutes worth of the total time:

$ du -sh /tmp/pip-download/
1.2G    /tmp/pip-download/

There is something pathological clearly with the pip-2020-resolver and the gauntlet I put it through with the universal patching. I've snapshotted the pip log after 43 minutes of the pathological case and will stare at it and some pip resolvelib source to see what exactly is pathological here.

jsirois commented 2 years ago

This appears to be ~non-terminating even with --intransitive (which is translated to pip's --no-deps). This has run >30 minutes for me:

rm -rf ~/.pex && time python -mpex.cli lock create -v --intransitive --output=lock.json --no-emit-warnings --style=universal --resolver-version pip-2020-resolver --indent=2 --no-pypi --index=https://pypi.org/simple/ --manylinux manylinux2014 --interpreter-constraint "CPython==3.9.*" APScheduler==3.8.1 Babel==2.9.1 Cython==0.29.24 Deprecated==1.2.13 Faker==13.3.4 Flask-Admin==1.5.8 Flask-SQLAlchemy-Session==1.1 Flask-SQLAlchemy==2.5.1 Flask-WTF==0.15.1 Flask==2.0.1 Jinja2==3.0.1 Mako==1.2.0 MarkupSafe==2.0.1 Pillow==8.4.0 PyGithub==1.55 PyJWT==2.3.0 PyNaCl==1.4.0 PySocks==1.7.1 PyYAML==5.4.1 Pygments==2.11.2 Quandl==3.7.0 Quart==0.15.1 SQLAlchemy-Paginator==0.2 SQLAlchemy-Utils==0.38.2 SQLAlchemy==1.3.24 Send2Trash==1.8.0 Sphinx==4.2.0 UnleashClient==4.4.1 WTForms-SQLAlchemy==0.2 WTForms==2.3.3 Werkzeug==2.0.3 aiofiles==0.8.0 aiohttp==3.7.4.post0 aioredis==1.3.1 alabaster==0.7.12 alembic-verify==0.1.4 alembic==1.7.3 amazon-braket-default-simulator==1.5.0 amazon-braket-ocean-plugin==1.0.8 amazon-braket-schemas==1.7.1 amazon-braket-sdk==1.15.0 amqp==5.1.0 anyio==3.4.0 appdirs==1.4.4 argon2-cffi==21.1.0 arrow==1.2.1 asttokens==2.0.5 async-timeout==3.0.1 attrs==21.2.0 auth0-python==3.18.0 backcall==0.2.0 backoff==1.11.1 backports.entry-points-selectable==1.1.1 bcrypt==3.2.0 billiard==3.6.4.0 black==21.9b0 bleach==4.1.0 blinker==1.4 boltons==21.0.0 boto3==1.21.1 botocore==1.24.1 celery==5.1.2 certifi==2021.10.8 cffi==1.15.0 cfgv==3.3.1 chardet==4.0.0 charset-normalizer==2.0.8 click-didyoumean==0.3.0 click-plugins==1.1.1 click-repl==0.2.0 click==7.1.2 colorama==0.4.4 coverage==5.5 cryptography==36.0.0 cycler==0.11.0 debugpy==1.5.1 decorator==4.4.2 deepmerge==1.0.1 defusedxml==0.7.1 dill==0.3.4 dimod==0.10.6 diskcache==5.2.1 distlib==0.3.3 distro==1.6.0 dlx==1.0.4 dnspython==2.2.1 dockerpty==0.4.1 docopt==0.6.2 docplex==2.23.222 docutils==0.17.1 dogpile.cache==1.1.4 dwave-cloud-client==0.9.2 dwave-networkx==0.8.10 dwave-preprocessing==0.3.1.post0 dwave-qbsolv==0.3.3.post0 dwave-system==1.9.0 dwave-tabu==0.4.2 email-validator==1.1.3 entrypoints==0.3 execnet==1.9.0 factory-boy==3.2.0 fastdtw==0.3.4 fasteners==0.16.3 fastjsonschema==2.15.1 fcache==0.4.7 fenestrate==0.3.0 filelock==3.4.0 flask-oidc==1.4.0 gunicorn==20.1.0 h11==0.13.0 h2==4.1.0 h5py==3.2.1 hiredis==2.0.0 homebase==1.0.1 hpack==4.0.0 httplib2==0.20.4 hypercorn==0.13.2 hyperframe==6.0.1 hypothesis==6.21.6 icdiff==2.0.4 icontract==2.6.0 identify==2.4.0 idna==3.3 imagesize==1.3.0 importlib-metadata==4.8.1 inflection==0.5.1 iniconfig==1.1.1 intervaltree==3.1.0 invoke==1.6.0 ipykernel==6.5.1 ipython-genutils==0.2.0 ipython==7.30.0 ipywidgets==7.6.5 itsdangerous==2.0.1 jedi==0.18.1 jinja2-time==0.2.0 jmespath==0.10.0 joblib==1.1.0 json5==0.9.6 jsonref==0.2 jsonschema==3.2.0 jupyter-client==7.1.0 jupyter-core==4.9.1 jupyter-packaging==0.7.9 jupyter-server==1.12.1 jupyterlab-pygments==0.1.2 jupyterlab-server==2.8.2 jupyterlab-widgets==1.1.0 jupyterlab==3.1.12 kiwisolver==1.3.2 kombu==5.2.4 llvmlite==0.36.0 lxml==4.8.0 lz4==3.1.3 markdown2==2.4.1 matplotlib-inline==0.1.3 matplotlib==3.4.3 memory-profiler==0.58.0 minorminer==0.2.6 mistune==0.8.4 mmh3==3.0.0 more-itertools==8.12.0 moto==2.2.7 mpmath==1.2.1 multidict==5.2.0 multitasking==0.0.10 mypy-extensions==0.4.3 nbclassic==0.3.4 nbclient==0.5.9 nbconvert==6.3.0 nbformat==5.1.3 nbmake==1.0 nest-asyncio==1.5.1 networkx==2.6.3 nodeenv==1.6.0 notebook==6.4.6 ntlm-auth==1.5.0 numba==0.53.1 numpy==1.21.2 oauth2client==4.1.3 okta==0.0.4 opentelemetry-api==1.8.0 opentelemetry-exporter-zipkin-json==1.8.0 opentelemetry-exporter-zipkin-proto-http==1.8.0 opentelemetry-exporter-zipkin==1.8.0 opentelemetry-instrumentation-sqlalchemy==0.27b0 opentelemetry-instrumentation==0.27b0 opentelemetry-sdk==1.8.0 opentelemetry-semantic-conventions==0.27b0 opt-einsum==3.3.0 packaging==21.0 pandas==1.3.4 pandocfilters==1.5.0 paramiko==2.8.1 parse==1.19.0 parso==0.8.2 pathspec==0.9.0 pbr==5.8.1 pexpect==4.8.0 pickleshare==0.7.5 platformdirs==2.4.0 plucky==0.4.3 pluggy==1.0.0 ply==3.11 pprintpp==0.4.0 pre-commit==2.15.0 priority==2.0.0 prometheus-client==0.12.0 prompt-toolkit==3.0.23 protobuf==3.19.4 psutil==5.8.0 psycopg2-binary==2.9.2 ptyprocess==0.7.0 py==1.11.0 pyasn1-modules==0.2.8 pyasn1==0.4.8 pybind11==2.8.1 pycodestyle==2.7.0 pycparser==2.21 pydantic==1.8.2 pynvml==11.0.0 pyparsing==3.0.6 pyrsistent==0.18.0 pytest-asyncio==0.15.1 pytest-cov==3.0.0 pytest-forked==1.3.0 pytest-icdiff==0.5 pytest-mock==3.6.1 pytest-repeat==0.9.1 pytest-xdist==2.4.0 pytest==6.2.5 python-constraint==1.4.0 python-dateutil==2.8.2 python-decouple==3.4 python-dotenv==0.19.2 python-http-client==3.3.7 python-lsp-black==1.0.0 python-lsp-jsonrpc==1.0.0 python-lsp-server==1.3.2 pytz-deprecation-shim==0.1.0.post0 pytz==2021.3 pyzmq==22.3.0 qcware-quasar==1.0.6 qcware-transpile==0.1.1a19 qiskit-aer==0.9.0 qiskit-aqua==0.9.5 qiskit-ibmq-provider==0.16.0 qiskit-ignis==0.6.0 qiskit-terra==0.18.3 qiskit==0.30.1 qubovert==1.2.3 redis==3.5.3 regex==2021.11.10 requests-ntlm==1.1.0 requests==2.26.0 responses==0.18.0 retworkx==0.10.2 rsa==4.8 s3transfer==0.5.0 scikit-learn==0.24.2 scipy==1.7.1 sendgrid==6.8.1 setuptools==58.0.4 six==1.16.0 sniffio==1.2.0 snowballstemmer==2.2.0 sortedcontainers==2.4.0 sphinx-rtd-theme==1.0.0 sphinxcontrib-applehelp==1.0.2 sphinxcontrib-devhelp==1.0.2 sphinxcontrib-htmlhelp==2.0.0 sphinxcontrib-jsmath==1.0.1 sphinxcontrib-qthelp==1.0.3 sphinxcontrib-serializinghtml==1.1.5 sqlalchemy-diff==0.1.5 starkbank-ecdsa==2.0.3 stevedore==3.5.0 structlog==21.5.0 symengine==0.8.1 sympy==1.9 tabulate==0.8.9 terminado==0.12.1 testpath==0.5.0 texttable==1.6.4 threadpoolctl==3.0.0 toml==0.10.2 tomli==1.2.2 toolz==0.11.2 torch==1.10.2 tornado==6.1 tqdm==4.62.3 traitlets==5.1.1 tweedledum==1.1.1 types-requests==2.25.8 typing_extensions==4.0.0 tzdata==2021.5 tzlocal==4.1 ujson==4.3.0 unitair==0.1.6 urllib3==1.26.7 vine==5.0.0 virtualenv==20.10.0 watchdog==2.1.5 wcwidth==0.2.5 webencodings==0.5.1 websocket-client==1.3.2 wheel==0.37.0 widgetsnbextension==3.5.2 wrapt==1.13.3 wsproto==1.1.0 xmltodict==0.12.0 yapf==0.31.0 yarl==1.7.2 yattag==1.14.0 yfinance==0.1.70 zipp==3.7.0
...
jsirois commented 2 years ago

If I revert the fix for https://github.com/pantsbuild/pex/issues/1667:

diff --git a/pex/pip/runtime_patches.py b/pex/pip/runtime_patches.py
index f3fece0..6a30900 100644
--- a/pex/pip/runtime_patches.py
+++ b/pex/pip/runtime_patches.py
@@ -149,7 +149,7 @@ if skip_markers:
                 # specified since we have no way to force Pip to backtrack and follow paths for any
                 # divergences. Most (all?) true divergences should be covered by forked environment
                 # markers.
-                return all(
+                return any(
                     orig_check_requires_python(requires_python, python_full_version)
                     for python_full_version in python_full_versions
                 )
@@ -185,7 +185,7 @@ if skip_markers:
                 # specified since we have no way to force Pip to backtrack and follow paths for any
                 # divergences. Most (all?) true divergences should be covered by forked environment
                 # markers.
-                return all(
+                return any(
                     orig_is_satisfied_by(self, RequiresPythonCandidate(python_full_version))
                     for python_full_version in python_full_versions
                 )

Then the resolve completes in reasonable time given baselines:

$ rm -rf ~/.pex && time python -mpex.cli lock create -v --output=lock.json --no-emit-warnings --style=universal --resolver-version pip-2020-resolver --indent=2 --no-pypi --index=https://pypi.org/simple/ --manylinux manylinux2014 --interpreter-constraint "CPython==3.9.*" APScheduler==3.8.1 Babel==2.9.1 Cython==0.29.24 Deprecated==1.2.13 Faker==13.3.4 Flask-Admin==1.5.8 Flask-SQLAlchemy-Session==1.1 Flask-SQLAlchemy==2.5.1 Flask-WTF==0.15.1 Flask==2.0.1 Jinja2==3.0.1 Mako==1.2.0 MarkupSafe==2.0.1 Pillow==8.4.0 PyGithub==1.55 PyJWT==2.3.0 PyNaCl==1.4.0 PySocks==1.7.1 PyYAML==5.4.1 Pygments==2.11.2 Quandl==3.7.0 Quart==0.15.1 SQLAlchemy-Paginator==0.2 SQLAlchemy-Utils==0.38.2 SQLAlchemy==1.3.24 Send2Trash==1.8.0 Sphinx==4.2.0 UnleashClient==4.4.1 WTForms-SQLAlchemy==0.2 WTForms==2.3.3 Werkzeug==2.0.3 aiofiles==0.8.0 aiohttp==3.7.4.post0 aioredis==1.3.1 alabaster==0.7.12 alembic-verify==0.1.4 alembic==1.7.3 amazon-braket-default-simulator==1.5.0 amazon-braket-ocean-plugin==1.0.8 amazon-braket-schemas==1.7.1 amazon-braket-sdk==1.15.0 amqp==5.1.0 anyio==3.4.0 appdirs==1.4.4 argon2-cffi==21.1.0 arrow==1.2.1 asttokens==2.0.5 async-timeout==3.0.1 attrs==21.2.0 auth0-python==3.18.0 backcall==0.2.0 backoff==1.11.1 backports.entry-points-selectable==1.1.1 bcrypt==3.2.0 billiard==3.6.4.0 black==21.9b0 bleach==4.1.0 blinker==1.4 boltons==21.0.0 boto3==1.21.1 botocore==1.24.1 celery==5.1.2 certifi==2021.10.8 cffi==1.15.0 cfgv==3.3.1 chardet==4.0.0 charset-normalizer==2.0.8 click-didyoumean==0.3.0 click-plugins==1.1.1 click-repl==0.2.0 click==7.1.2 colorama==0.4.4 coverage==5.5 cryptography==36.0.0 cycler==0.11.0 debugpy==1.5.1 decorator==4.4.2 deepmerge==1.0.1 defusedxml==0.7.1 dill==0.3.4 dimod==0.10.6 diskcache==5.2.1 distlib==0.3.3 distro==1.6.0 dlx==1.0.4 dnspython==2.2.1 dockerpty==0.4.1 docopt==0.6.2 docplex==2.23.222 docutils==0.17.1 dogpile.cache==1.1.4 dwave-cloud-client==0.9.2 dwave-networkx==0.8.10 dwave-preprocessing==0.3.1.post0 dwave-qbsolv==0.3.3.post0 dwave-system==1.9.0 dwave-tabu==0.4.2 email-validator==1.1.3 entrypoints==0.3 execnet==1.9.0 factory-boy==3.2.0 fastdtw==0.3.4 fasteners==0.16.3 fastjsonschema==2.15.1 fcache==0.4.7 fenestrate==0.3.0 filelock==3.4.0 flask-oidc==1.4.0 gunicorn==20.1.0 h11==0.13.0 h2==4.1.0 h5py==3.2.1 hiredis==2.0.0 homebase==1.0.1 hpack==4.0.0 httplib2==0.20.4 hypercorn==0.13.2 hyperframe==6.0.1 hypothesis==6.21.6 icdiff==2.0.4 icontract==2.6.0 identify==2.4.0 idna==3.3 imagesize==1.3.0 importlib-metadata==4.8.1 inflection==0.5.1 iniconfig==1.1.1 intervaltree==3.1.0 invoke==1.6.0 ipykernel==6.5.1 ipython-genutils==0.2.0 ipython==7.30.0 ipywidgets==7.6.5 itsdangerous==2.0.1 jedi==0.18.1 jinja2-time==0.2.0 jmespath==0.10.0 joblib==1.1.0 json5==0.9.6 jsonref==0.2 jsonschema==3.2.0 jupyter-client==7.1.0 jupyter-core==4.9.1 jupyter-packaging==0.7.9 jupyter-server==1.12.1 jupyterlab-pygments==0.1.2 jupyterlab-server==2.8.2 jupyterlab-widgets==1.1.0 jupyterlab==3.1.12 kiwisolver==1.3.2 kombu==5.2.4 llvmlite==0.36.0 lxml==4.8.0 lz4==3.1.3 markdown2==2.4.1 matplotlib-inline==0.1.3 matplotlib==3.4.3 memory-profiler==0.58.0 minorminer==0.2.6 mistune==0.8.4 mmh3==3.0.0 more-itertools==8.12.0 moto==2.2.7 mpmath==1.2.1 multidict==5.2.0 multitasking==0.0.10 mypy-extensions==0.4.3 nbclassic==0.3.4 nbclient==0.5.9 nbconvert==6.3.0 nbformat==5.1.3 nbmake==1.0 nest-asyncio==1.5.1 networkx==2.6.3 nodeenv==1.6.0 notebook==6.4.6 ntlm-auth==1.5.0 numba==0.53.1 numpy==1.21.2 oauth2client==4.1.3 okta==0.0.4 opentelemetry-api==1.8.0 opentelemetry-exporter-zipkin-json==1.8.0 opentelemetry-exporter-zipkin-proto-http==1.8.0 opentelemetry-exporter-zipkin==1.8.0 opentelemetry-instrumentation-sqlalchemy==0.27b0 opentelemetry-instrumentation==0.27b0 opentelemetry-sdk==1.8.0 opentelemetry-semantic-conventions==0.27b0 opt-einsum==3.3.0 packaging==21.0 pandas==1.3.4 pandocfilters==1.5.0 paramiko==2.8.1 parse==1.19.0 parso==0.8.2 pathspec==0.9.0 pbr==5.8.1 pexpect==4.8.0 pickleshare==0.7.5 platformdirs==2.4.0 plucky==0.4.3 pluggy==1.0.0 ply==3.11 pprintpp==0.4.0 pre-commit==2.15.0 priority==2.0.0 prometheus-client==0.12.0 prompt-toolkit==3.0.23 protobuf==3.19.4 psutil==5.8.0 psycopg2-binary==2.9.2 ptyprocess==0.7.0 py==1.11.0 pyasn1-modules==0.2.8 pyasn1==0.4.8 pybind11==2.8.1 pycodestyle==2.7.0 pycparser==2.21 pydantic==1.8.2 pynvml==11.0.0 pyparsing==3.0.6 pyrsistent==0.18.0 pytest-asyncio==0.15.1 pytest-cov==3.0.0 pytest-forked==1.3.0 pytest-icdiff==0.5 pytest-mock==3.6.1 pytest-repeat==0.9.1 pytest-xdist==2.4.0 pytest==6.2.5 python-constraint==1.4.0 python-dateutil==2.8.2 python-decouple==3.4 python-dotenv==0.19.2 python-http-client==3.3.7 python-lsp-black==1.0.0 python-lsp-jsonrpc==1.0.0 python-lsp-server==1.3.2 pytz-deprecation-shim==0.1.0.post0 pytz==2021.3 pyzmq==22.3.0 qcware-quasar==1.0.6 qcware-transpile==0.1.1a19 qiskit-aer==0.9.0 qiskit-aqua==0.9.5 qiskit-ibmq-provider==0.16.0 qiskit-ignis==0.6.0 qiskit-terra==0.18.3 qiskit==0.30.1 qubovert==1.2.3 redis==3.5.3 regex==2021.11.10 requests-ntlm==1.1.0 requests==2.26.0 responses==0.18.0 retworkx==0.10.2 rsa==4.8 s3transfer==0.5.0 scikit-learn==0.24.2 scipy==1.7.1 sendgrid==6.8.1 setuptools==58.0.4 six==1.16.0 sniffio==1.2.0 snowballstemmer==2.2.0 sortedcontainers==2.4.0 sphinx-rtd-theme==1.0.0 sphinxcontrib-applehelp==1.0.2 sphinxcontrib-devhelp==1.0.2 sphinxcontrib-htmlhelp==2.0.0 sphinxcontrib-jsmath==1.0.1 sphinxcontrib-qthelp==1.0.3 sphinxcontrib-serializinghtml==1.1.5 sqlalchemy-diff==0.1.5 starkbank-ecdsa==2.0.3 stevedore==3.5.0 structlog==21.5.0 symengine==0.8.1 sympy==1.9 tabulate==0.8.9 terminado==0.12.1 testpath==0.5.0 texttable==1.6.4 threadpoolctl==3.0.0 toml==0.10.2 tomli==1.2.2 toolz==0.11.2 torch==1.10.2 tornado==6.1 tqdm==4.62.3 traitlets==5.1.1 tweedledum==1.1.1 types-requests==2.25.8 typing_extensions==4.0.0 tzdata==2021.5 tzlocal==4.1 ujson==4.3.0 unitair==0.1.6 urllib3==1.26.7 vine==5.0.0 virtualenv==20.10.0 watchdog==2.1.5 wcwidth==0.2.5 webencodings==0.5.1 websocket-client==1.3.2 wheel==0.37.0 widgetsnbextension==3.5.2 wrapt==1.13.3 wsproto==1.1.0 xmltodict==0.12.0 yapf==0.31.0 yarl==1.7.2 yattag==1.14.0 yfinance==0.1.70 zipp==3.7.0
...
real    4m56.917s
user    3m45.577s
sys 0m5.302s

So that's a clue...

Eric-Arellano commented 2 years ago

That seems fishy to me given the exact interpreter constraints of Python 3.9. Great find!

jsirois commented 2 years ago

The constraints are not exact. The patch version is * and the fact that the python_full_version environment marker is a real thing means what looks like just check "3.9", means check all 30 patch versions we scan for "3.9".

jsirois commented 2 years ago

Reverting that patch and actually using an exact IC (--interpreter-constraint "CPython==3.9.10" instead of --interpreter-constraint "CPython==3.9.*") also works for example:

$ rm -rf ~/.pex && time python -mpex.cli lock create -v --output=lock.json --no-emit-warnings --style=universal --resolver-version pip-2020-resolver --indent=2 --no-pypi --index=https://pypi.org/simple/ --manylinux manylinux2014 --interpreter-constraint "CPython==3.9.10" APScheduler==3.8.1 Babel==2.9.1 Cython==0.29.24 Deprecated==1.2.13 Faker==13.3.4 Flask-Admin==1.5.8 Flask-SQLAlchemy-Session==1.1 Flask-SQLAlchemy==2.5.1 Flask-WTF==0.15.1 Flask==2.0.1 Jinja2==3.0.1 Mako==1.2.0 MarkupSafe==2.0.1 Pillow==8.4.0 PyGithub==1.55 PyJWT==2.3.0 PyNaCl==1.4.0 PySocks==1.7.1 PyYAML==5.4.1 Pygments==2.11.2 Quandl==3.7.0 Quart==0.15.1 SQLAlchemy-Paginator==0.2 SQLAlchemy-Utils==0.38.2 SQLAlchemy==1.3.24 Send2Trash==1.8.0 Sphinx==4.2.0 UnleashClient==4.4.1 WTForms-SQLAlchemy==0.2 WTForms==2.3.3 Werkzeug==2.0.3 aiofiles==0.8.0 aiohttp==3.7.4.post0 aioredis==1.3.1 alabaster==0.7.12 alembic-verify==0.1.4 alembic==1.7.3 amazon-braket-default-simulator==1.5.0 amazon-braket-ocean-plugin==1.0.8 amazon-braket-schemas==1.7.1 amazon-braket-sdk==1.15.0 amqp==5.1.0 anyio==3.4.0 appdirs==1.4.4 argon2-cffi==21.1.0 arrow==1.2.1 asttokens==2.0.5 async-timeout==3.0.1 attrs==21.2.0 auth0-python==3.18.0 backcall==0.2.0 backoff==1.11.1 backports.entry-points-selectable==1.1.1 bcrypt==3.2.0 billiard==3.6.4.0 black==21.9b0 bleach==4.1.0 blinker==1.4 boltons==21.0.0 boto3==1.21.1 botocore==1.24.1 celery==5.1.2 certifi==2021.10.8 cffi==1.15.0 cfgv==3.3.1 chardet==4.0.0 charset-normalizer==2.0.8 click-didyoumean==0.3.0 click-plugins==1.1.1 click-repl==0.2.0 click==7.1.2 colorama==0.4.4 coverage==5.5 cryptography==36.0.0 cycler==0.11.0 debugpy==1.5.1 decorator==4.4.2 deepmerge==1.0.1 defusedxml==0.7.1 dill==0.3.4 dimod==0.10.6 diskcache==5.2.1 distlib==0.3.3 distro==1.6.0 dlx==1.0.4 dnspython==2.2.1 dockerpty==0.4.1 docopt==0.6.2 docplex==2.23.222 docutils==0.17.1 dogpile.cache==1.1.4 dwave-cloud-client==0.9.2 dwave-networkx==0.8.10 dwave-preprocessing==0.3.1.post0 dwave-qbsolv==0.3.3.post0 dwave-system==1.9.0 dwave-tabu==0.4.2 email-validator==1.1.3 entrypoints==0.3 execnet==1.9.0 factory-boy==3.2.0 fastdtw==0.3.4 fasteners==0.16.3 fastjsonschema==2.15.1 fcache==0.4.7 fenestrate==0.3.0 filelock==3.4.0 flask-oidc==1.4.0 gunicorn==20.1.0 h11==0.13.0 h2==4.1.0 h5py==3.2.1 hiredis==2.0.0 homebase==1.0.1 hpack==4.0.0 httplib2==0.20.4 hypercorn==0.13.2 hyperframe==6.0.1 hypothesis==6.21.6 icdiff==2.0.4 icontract==2.6.0 identify==2.4.0 idna==3.3 imagesize==1.3.0 importlib-metadata==4.8.1 inflection==0.5.1 iniconfig==1.1.1 intervaltree==3.1.0 invoke==1.6.0 ipykernel==6.5.1 ipython-genutils==0.2.0 ipython==7.30.0 ipywidgets==7.6.5 itsdangerous==2.0.1 jedi==0.18.1 jinja2-time==0.2.0 jmespath==0.10.0 joblib==1.1.0 json5==0.9.6 jsonref==0.2 jsonschema==3.2.0 jupyter-client==7.1.0 jupyter-core==4.9.1 jupyter-packaging==0.7.9 jupyter-server==1.12.1 jupyterlab-pygments==0.1.2 jupyterlab-server==2.8.2 jupyterlab-widgets==1.1.0 jupyterlab==3.1.12 kiwisolver==1.3.2 kombu==5.2.4 llvmlite==0.36.0 lxml==4.8.0 lz4==3.1.3 markdown2==2.4.1 matplotlib-inline==0.1.3 matplotlib==3.4.3 memory-profiler==0.58.0 minorminer==0.2.6 mistune==0.8.4 mmh3==3.0.0 more-itertools==8.12.0 moto==2.2.7 mpmath==1.2.1 multidict==5.2.0 multitasking==0.0.10 mypy-extensions==0.4.3 nbclassic==0.3.4 nbclient==0.5.9 nbconvert==6.3.0 nbformat==5.1.3 nbmake==1.0 nest-asyncio==1.5.1 networkx==2.6.3 nodeenv==1.6.0 notebook==6.4.6 ntlm-auth==1.5.0 numba==0.53.1 numpy==1.21.2 oauth2client==4.1.3 okta==0.0.4 opentelemetry-api==1.8.0 opentelemetry-exporter-zipkin-json==1.8.0 opentelemetry-exporter-zipkin-proto-http==1.8.0 opentelemetry-exporter-zipkin==1.8.0 opentelemetry-instrumentation-sqlalchemy==0.27b0 opentelemetry-instrumentation==0.27b0 opentelemetry-sdk==1.8.0 opentelemetry-semantic-conventions==0.27b0 opt-einsum==3.3.0 packaging==21.0 pandas==1.3.4 pandocfilters==1.5.0 paramiko==2.8.1 parse==1.19.0 parso==0.8.2 pathspec==0.9.0 pbr==5.8.1 pexpect==4.8.0 pickleshare==0.7.5 platformdirs==2.4.0 plucky==0.4.3 pluggy==1.0.0 ply==3.11 pprintpp==0.4.0 pre-commit==2.15.0 priority==2.0.0 prometheus-client==0.12.0 prompt-toolkit==3.0.23 protobuf==3.19.4 psutil==5.8.0 psycopg2-binary==2.9.2 ptyprocess==0.7.0 py==1.11.0 pyasn1-modules==0.2.8 pyasn1==0.4.8 pybind11==2.8.1 pycodestyle==2.7.0 pycparser==2.21 pydantic==1.8.2 pynvml==11.0.0 pyparsing==3.0.6 pyrsistent==0.18.0 pytest-asyncio==0.15.1 pytest-cov==3.0.0 pytest-forked==1.3.0 pytest-icdiff==0.5 pytest-mock==3.6.1 pytest-repeat==0.9.1 pytest-xdist==2.4.0 pytest==6.2.5 python-constraint==1.4.0 python-dateutil==2.8.2 python-decouple==3.4 python-dotenv==0.19.2 python-http-client==3.3.7 python-lsp-black==1.0.0 python-lsp-jsonrpc==1.0.0 python-lsp-server==1.3.2 pytz-deprecation-shim==0.1.0.post0 pytz==2021.3 pyzmq==22.3.0 qcware-quasar==1.0.6 qcware-transpile==0.1.1a19 qiskit-aer==0.9.0 qiskit-aqua==0.9.5 qiskit-ibmq-provider==0.16.0 qiskit-ignis==0.6.0 qiskit-terra==0.18.3 qiskit==0.30.1 qubovert==1.2.3 redis==3.5.3 regex==2021.11.10 requests-ntlm==1.1.0 requests==2.26.0 responses==0.18.0 retworkx==0.10.2 rsa==4.8 s3transfer==0.5.0 scikit-learn==0.24.2 scipy==1.7.1 sendgrid==6.8.1 setuptools==58.0.4 six==1.16.0 sniffio==1.2.0 snowballstemmer==2.2.0 sortedcontainers==2.4.0 sphinx-rtd-theme==1.0.0 sphinxcontrib-applehelp==1.0.2 sphinxcontrib-devhelp==1.0.2 sphinxcontrib-htmlhelp==2.0.0 sphinxcontrib-jsmath==1.0.1 sphinxcontrib-qthelp==1.0.3 sphinxcontrib-serializinghtml==1.1.5 sqlalchemy-diff==0.1.5 starkbank-ecdsa==2.0.3 stevedore==3.5.0 structlog==21.5.0 symengine==0.8.1 sympy==1.9 tabulate==0.8.9 terminado==0.12.1 testpath==0.5.0 texttable==1.6.4 threadpoolctl==3.0.0 toml==0.10.2 tomli==1.2.2 toolz==0.11.2 torch==1.10.2 tornado==6.1 tqdm==4.62.3 traitlets==5.1.1 tweedledum==1.1.1 types-requests==2.25.8 typing_extensions==4.0.0 tzdata==2021.5 tzlocal==4.1 ujson==4.3.0 unitair==0.1.6 urllib3==1.26.7 vine==5.0.0 virtualenv==20.10.0 watchdog==2.1.5 wcwidth==0.2.5 webencodings==0.5.1 websocket-client==1.3.2 wheel==0.37.0 widgetsnbextension==3.5.2 wrapt==1.13.3 wsproto==1.1.0 xmltodict==0.12.0 yapf==0.31.0 yarl==1.7.2 yattag==1.14.0 yfinance==0.1.70 zipp==3.7.0
...

real    4m41.382s
user    3m45.565s
sys 0m5.421s
jsirois commented 2 years ago

Ok, the current Pip universal patches for Requires-Python handling blindly use python full versions when checking Requires-Python just in case the Requires-Python specifier uses full versions. If I parse out the specifier set components and only do this when there is a (non *) patch version somewhere in the specifier set; then things terminate in much the same time (~5 minutes) as the legacy resolver. I'll fork off a Pex issue and fix.

jsirois commented 2 years ago

https://github.com/pantsbuild/pex/issues/1706