uv pip and python -m pip resolve different versions of tensorflow in pyhf developer environment

matthewfeickert commented 4 months ago

Hi. :wave: While trying to adopt uv for use in the pyhf CI system (https://github.com/scikit-hep/pyhf/pull/2444) we've found that we're getting different environment solves between python -m pip install and uv pip install. The result of uv pip install results in an unusable environment of tensorflow v2.11.1 and tensorflow-probability v0.20.1 while python -m pip install resolves in the (in our view "correct") usable environment of tensorflow v2.13.1 and tensorflow-probability v0.20.1.

To avoid overwhelming wall of text I've broken the reproducible example in a Docker container into two sections that can be expanded below.

Here's the other requested info up front though:

The current uv platform: The python:3.8 Docker container image

# uname -a
Linux 2c24021d2969 6.5.0-15-generic #15~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Fri Jan 12 18:54:30 UTC 2 x86_64 GNU/Linux

The current uv version (uv --version): uv 0.1.12

Using pip

Using pip:

```console $ docker run --rm -ti python:3.8 /bin/bash root@2c24021d2969:/# python -m venv venv && . venv/bin/activate (venv) root@2c24021d2969:/# python -m pip --quiet install --upgrade pip setuptools wheel (venv) root@2c24021d2969:/# git clone https://github.com/scikit-hep/pyhf.git --branch main --single-branch && cd pyhf (venv) root@2c24021d2969:/pyhf# git reset --hard fc5e83018d2a2c98156e5073a6e630793a1edd55 (venv) root@2c24021d2969:/pyhf# python -m pip --verbose --no-cache-dir install --upgrade '.[all,test]' &> /tmp/python-pip-install-verbose.txt (venv) root@2c24021d2969:/pyhf# python -m pip list Package Version ---------------------------- --------------- absl-py 2.1.0 ansiwrap 0.8.4 anyio 4.3.0 argon2-cffi 23.1.0 argon2-cffi-bindings 21.2.0 arrow 1.3.0 asttokens 2.4.1 astunparse 1.6.3 async-lru 2.0.4 attrs 23.2.0 awkward 2.6.1 awkward-cpp 29 Babel 2.14.0 backcall 0.2.0 beautifulsoup4 4.12.3 bleach 6.1.0 cachetools 5.3.3 certifi 2024.2.2 cffi 1.16.0 charset-normalizer 3.3.2 click 8.1.7 click-completion 0.5.2 cloudpickle 3.0.0 comm 0.2.1 contourpy 1.1.1 coverage 7.4.3 cramjam 2.8.1 cycler 0.12.1 debugpy 1.8.1 decorator 5.1.1 defusedxml 0.7.1 dm-tree 0.1.8 entrypoints 0.4 exceptiongroup 1.2.0 executing 2.0.1 fastjsonschema 2.19.1 filelock 3.13.1 flatbuffers 23.5.26 fonttools 4.49.0 fqdn 1.5.1 fsspec 2024.2.0 gast 0.4.0 google-auth 2.28.1 google-auth-oauthlib 1.0.0 google-pasta 0.2.0 graphviz 0.20.1 grpcio 1.62.0 h11 0.14.0 h5py 3.10.0 httpcore 1.0.4 httpx 0.27.0 idna 3.6 iminuit 2.25.2 importlib-metadata 7.0.1 importlib_resources 6.1.2 iniconfig 2.0.0 ipykernel 6.29.3 ipympl 0.9.3 ipython 8.12.3 ipython-genutils 0.2.0 ipywidgets 8.1.2 isoduration 20.11.0 jax 0.4.13 jaxlib 0.4.13 jedi 0.19.1 Jinja2 3.1.3 json5 0.9.17 jsonpatch 1.33 jsonpointer 2.4 jsonschema 4.21.1 jsonschema-specifications 2023.12.1 jupyter 1.0.0 jupyter_client 8.6.0 jupyter-console 6.6.3 jupyter_core 5.7.1 jupyter-events 0.9.0 jupyter-lsp 2.2.3 jupyter_server 2.12.5 jupyter_server_terminals 0.5.2 jupyterlab 4.1.2 jupyterlab_pygments 0.3.0 jupyterlab_server 2.25.3 jupyterlab_widgets 3.0.10 keras 2.13.1 kiwisolver 1.4.5 libclang 16.0.6 Markdown 3.5.2 MarkupSafe 2.1.5 matplotlib 3.7.5 matplotlib-inline 0.1.6 mistune 3.0.2 ml-dtypes 0.2.0 mpmath 1.3.0 nbclient 0.9.0 nbconvert 7.16.1 nbformat 5.9.2 nest-asyncio 1.6.0 networkx 3.1 notebook 7.1.1 notebook_shim 0.2.4 numpy 1.24.3 nvidia-cublas-cu12 12.1.3.1 nvidia-cuda-cupti-cu12 12.1.105 nvidia-cuda-nvrtc-cu12 12.1.105 nvidia-cuda-runtime-cu12 12.1.105 nvidia-cudnn-cu12 8.9.2.26 nvidia-cufft-cu12 11.0.2.54 nvidia-curand-cu12 10.3.2.106 nvidia-cusolver-cu12 11.4.5.107 nvidia-cusparse-cu12 12.1.0.106 nvidia-nccl-cu12 2.18.1 nvidia-nvjitlink-cu12 12.3.101 nvidia-nvtx-cu12 12.1.105 oauthlib 3.2.2 opt-einsum 3.3.0 overrides 7.7.0 packaging 23.2 pandas 2.0.3 pandocfilters 1.5.1 papermill 2.3.4 parso 0.8.3 pexpect 4.9.0 pickleshare 0.7.5 pillow 10.2.0 pip 24.0 pkgutil_resolve_name 1.3.10 platformdirs 4.2.0 pluggy 1.4.0 prometheus_client 0.20.0 prompt-toolkit 3.0.43 protobuf 4.25.3 psutil 5.9.8 ptyprocess 0.7.0 pure-eval 0.2.2 py-cpuinfo 9.0.0 pyarrow 15.0.0 pyasn1 0.5.1 pyasn1-modules 0.3.0 pycparser 2.21 pydocstyle 6.3.0 pygal 3.0.4 pygaljs 1.0.2 Pygments 2.17.2 pyhf 0.7.1.dev220 pyparsing 3.1.1 pytest 8.0.2 pytest-benchmark 4.0.0 pytest-console-scripts 1.4.1 pytest-mock 3.12.0 pytest-mpl 0.17.0 pytest-socket 0.7.0 python-dateutil 2.8.2 python-json-logger 2.0.7 pytz 2024.1 PyYAML 6.0.1 pyzmq 25.1.2 qtconsole 5.5.1 QtPy 2.4.1 referencing 0.33.0 requests 2.31.0 requests-mock 1.11.0 requests-oauthlib 1.3.1 rfc3339-validator 0.1.4 rfc3986-validator 0.1.1 rpds-py 0.18.0 rsa 4.9 scikit_hep_testdata 0.4.38 scipy 1.10.1 scrapbook 0.5.0 Send2Trash 1.8.2 setuptools 69.1.1 shellingham 1.5.4 six 1.16.0 sniffio 1.3.1 snowballstemmer 2.2.0 soupsieve 2.5 stack-data 0.6.3 sympy 1.12 tenacity 8.2.3 tensorboard 2.13.0 tensorboard-data-server 0.7.2 tensorflow 2.13.1 tensorflow-estimator 2.13.0 tensorflow-io-gcs-filesystem 0.34.0 tensorflow-probability 0.21.0 termcolor 2.4.0 terminado 0.18.0 textwrap3 0.9.2 tinycss2 1.2.1 tomli 2.0.1 torch 2.1.2 tornado 6.4 tqdm 4.66.2 traitlets 5.14.1 triton 2.1.0 types-python-dateutil 2.8.19.20240106 typing_extensions 4.5.0 tzdata 2024.1 uproot 5.3.1 uri-template 1.3.0 urllib3 2.2.1 wcwidth 0.2.13 webcolors 1.13 webencodings 0.5.1 websocket-client 1.7.0 Werkzeug 3.0.1 wheel 0.42.0 widgetsnbextension 4.0.10 wrapt 1.16.0 zipp 3.17.0 (venv) root@2c24021d2969:/pyhf# python -m pip list | grep 'tensor' tensorboard 2.13.0 tensorboard-data-server 0.7.2 tensorflow 2.13.1 tensorflow-estimator 2.13.0 tensorflow-io-gcs-filesystem 0.34.0 tensorflow-probability 0.21.0 (venv) root@2c24021d2969:/pyhf# python -m pip show tensorflow Name: tensorflow Version: 2.13.1 Summary: TensorFlow is an open source machine learning framework for everyone. Home-page: https://www.tensorflow.org/ Author: Google Inc. Author-email: packages@tensorflow.org License: Apache 2.0 Location: /venv/lib/python3.8/site-packages Requires: absl-py, astunparse, flatbuffers, gast, google-pasta, grpcio, h5py, keras, libclang, numpy, opt-einsum, packaging, protobuf, setuptools, six, tensorboard, tensorflow-estimator, tensorflow-io-gcs-filesystem, termcolor, typing-extensions, wrapt Required-by: (venv) root@2c24021d2969:/pyhf# python -c 'import tensorflow; import tensorflow_probability' # no error 2024-02-29 00:04:43.719791: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. 2024-02-29 00:04:43.764247: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used. 2024-02-29 00:04:44.089063: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used. 2024-02-29 00:04:44.090564: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations. To enable the following instructions: AVX2 AVX_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags. 2024-02-29 00:04:44.978663: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT (venv) root@2c24021d2969:/pyhf# ```

python-pip-install-verbose.txt

and now in a virtual environment with uv (version 0.1.12)

uv v0.1.12:

```console (venv) root@2c24021d2969:/pyhf# deactivate root@2c24021d2969:/pyhf# python -m venv /uv-venv && . /uv-venv/bin/activate (uv-venv) root@2c24021d2969:/pyhf# python -m pip --quiet install --upgrade pip setuptools wheel uv (uv-venv) root@2c24021d2969:/pyhf# uv --version uv 0.1.12 (uv-venv) root@2c24021d2969:/pyhf# uv pip --verbose --no-cache install --upgrade 'pyhf[all,test] @ .' &> /tmp/uv-pip-install-verbose.txt (uv-venv) root@2c24021d2969:/pyhf# uv pip list Package Version ---------------------------- --------------- absl-py 2.1.0 ansiwrap 0.8.4 anyio 4.3.0 argon2-cffi 23.1.0 argon2-cffi-bindings 21.2.0 arrow 1.3.0 asttokens 2.4.1 astunparse 1.6.3 async-lru 2.0.4 attrs 23.2.0 awkward 2.6.1 awkward-cpp 29 babel 2.14.0 backcall 0.2.0 beautifulsoup4 4.12.3 bleach 6.1.0 cachetools 5.3.3 certifi 2024.2.2 cffi 1.16.0 charset-normalizer 3.3.2 click 8.1.7 click-completion 0.5.2 cloudpickle 3.0.0 comm 0.2.1 contourpy 1.1.1 coverage 7.4.3 cramjam 2.8.1 cycler 0.12.1 debugpy 1.8.1 decorator 5.1.1 defusedxml 0.7.1 dm-tree 0.1.8 entrypoints 0.4 exceptiongroup 1.2.0 executing 2.0.1 fastjsonschema 2.19.1 filelock 3.13.1 flatbuffers 23.5.26 fonttools 4.49.0 fqdn 1.5.1 fsspec 2024.2.0 gast 0.4.0 google-auth 2.28.1 google-auth-oauthlib 0.4.6 google-pasta 0.2.0 graphviz 0.20.1 grpcio 1.62.0 h11 0.14.0 h5py 3.10.0 httpcore 1.0.4 httpx 0.27.0 idna 3.6 iminuit 2.25.2 importlib-metadata 7.0.1 importlib-resources 6.1.2 iniconfig 2.0.0 ipykernel 6.29.3 ipympl 0.9.3 ipython 8.12.3 ipython-genutils 0.2.0 ipywidgets 8.1.2 isoduration 20.11.0 jax 0.4.13 jaxlib 0.4.13 jedi 0.19.1 jinja2 3.1.3 json5 0.9.17 jsonpatch 1.33 jsonpointer 2.4 jsonschema 4.21.1 jsonschema-specifications 2023.12.1 jupyter 1.0.0 jupyter-client 8.6.0 jupyter-console 6.6.3 jupyter-core 5.7.1 jupyter-events 0.9.0 jupyter-lsp 2.2.3 jupyter-server 2.12.5 jupyter-server-terminals 0.5.2 jupyterlab 4.1.2 jupyterlab-pygments 0.3.0 jupyterlab-server 2.25.3 jupyterlab-widgets 3.0.10 keras 2.11.0 kiwisolver 1.4.5 libclang 16.0.6 markdown 3.5.2 markupsafe 2.1.5 matplotlib 3.7.5 matplotlib-inline 0.1.6 mistune 3.0.2 ml-dtypes 0.2.0 mpmath 1.3.0 nbclient 0.9.0 nbconvert 7.16.1 nbformat 5.9.2 nest-asyncio 1.6.0 networkx 3.1 notebook 7.1.1 notebook-shim 0.2.4 numpy 1.24.4 nvidia-cublas-cu12 12.1.3.1 nvidia-cuda-cupti-cu12 12.1.105 nvidia-cuda-nvrtc-cu12 12.1.105 nvidia-cuda-runtime-cu12 12.1.105 nvidia-cudnn-cu12 8.9.2.26 nvidia-cufft-cu12 11.0.2.54 nvidia-curand-cu12 10.3.2.106 nvidia-cusolver-cu12 11.4.5.107 nvidia-cusparse-cu12 12.1.0.106 nvidia-nccl-cu12 2.19.3 nvidia-nvjitlink-cu12 12.3.101 nvidia-nvtx-cu12 12.1.105 oauthlib 3.2.2 opt-einsum 3.3.0 overrides 7.7.0 packaging 23.2 pandas 2.0.3 pandocfilters 1.5.1 papermill 2.3.4 parso 0.8.3 pexpect 4.9.0 pickleshare 0.7.5 pillow 10.2.0 pip 24.0 pkgutil-resolve-name 1.3.10 platformdirs 4.2.0 pluggy 1.4.0 prometheus-client 0.20.0 prompt-toolkit 3.0.43 protobuf 3.19.6 psutil 5.9.8 ptyprocess 0.7.0 pure-eval 0.2.2 py-cpuinfo 9.0.0 pyarrow 15.0.0 pyasn1 0.5.1 pyasn1-modules 0.3.0 pycparser 2.21 pydocstyle 6.3.0 pygal 3.0.4 pygaljs 1.0.2 pygments 2.17.2 pyhf 0.7.1.dev220 pyparsing 3.1.1 pytest 8.0.2 pytest-benchmark 4.0.0 pytest-console-scripts 1.4.1 pytest-mock 3.12.0 pytest-mpl 0.17.0 pytest-socket 0.7.0 python-dateutil 2.8.2 python-json-logger 2.0.7 pytz 2024.1 pyyaml 6.0.1 pyzmq 25.1.2 qtconsole 5.5.1 qtpy 2.4.1 referencing 0.33.0 requests 2.31.0 requests-mock 1.11.0 requests-oauthlib 1.3.1 rfc3339-validator 0.1.4 rfc3986-validator 0.1.1 rpds-py 0.18.0 rsa 4.9 scikit-hep-testdata 0.4.38 scipy 1.10.1 scrapbook 0.5.0 send2trash 1.8.2 setuptools 69.1.1 shellingham 1.5.4 six 1.16.0 sniffio 1.3.1 snowballstemmer 2.2.0 soupsieve 2.5 stack-data 0.6.3 sympy 1.12 tenacity 8.2.3 tensorboard 2.11.2 tensorboard-data-server 0.6.1 tensorboard-plugin-wit 1.8.1 tensorflow 2.11.1 tensorflow-estimator 2.11.0 tensorflow-io-gcs-filesystem 0.34.0 tensorflow-probability 0.20.1 termcolor 2.4.0 terminado 0.18.0 textwrap3 0.9.2 tinycss2 1.2.1 tomli 2.0.1 torch 2.2.1 tornado 6.4 tqdm 4.66.2 traitlets 5.14.1 triton 2.2.0 types-python-dateutil 2.8.19.20240106 typing-extensions 4.10.0 tzdata 2024.1 uproot 5.3.1 uri-template 1.3.0 urllib3 2.2.1 uv 0.1.12 wcwidth 0.2.13 webcolors 1.13 webencodings 0.5.1 websocket-client 1.7.0 werkzeug 3.0.1 wheel 0.42.0 widgetsnbextension 4.0.10 wrapt 1.16.0 zipp 3.17.0 (uv-venv) root@2c24021d2969:/pyhf# uv pip list | grep tensorflow tensorflow 2.11.1 tensorflow-estimator 2.11.0 tensorflow-io-gcs-filesystem 0.34.0 tensorflow-probability 0.20.1 (uv-venv) root@2c24021d2969:/pyhf# python -m pip show tensorflow Name: tensorflow Version: 2.11.1 Summary: TensorFlow is an open source machine learning framework for everyone. Home-page: https://www.tensorflow.org/ Author: Google Inc. Author-email: packages@tensorflow.org License: Apache 2.0 Location: /uv-venv/lib/python3.8/site-packages Requires: absl-py, astunparse, flatbuffers, gast, google-pasta, grpcio, h5py, keras, libclang, numpy, opt-einsum, packaging, protobuf, setuptools, six, tensorboard, tensorflow-estimator, tensorflow-io-gcs-filesystem, termcolor, typing-extensions, wrapt Required-by: (uv-venv) root@2c24021d2969:/pyhf# python -c 'import tensorflow; import tensorflow_probability' 2024-02-29 00:04:00.444303: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 AVX_VNNI FMA To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 2024-02-29 00:04:00.527810: I tensorflow/core/util/port.cc:104] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. 2024-02-29 00:04:00.531317: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory 2024-02-29 00:04:00.531351: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine. 2024-02-29 00:04:01.047385: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory 2024-02-29 00:04:01.047437: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory 2024-02-29 00:04:01.047442: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly. Traceback (most recent call last): File "", line 1, in File "/uv-venv/lib/python3.8/site-packages/tensorflow_probability/__init__.py", line 20, in from tensorflow_probability import substrates File "/uv-venv/lib/python3.8/site-packages/tensorflow_probability/substrates/__init__.py", line 17, in from tensorflow_probability.python.internal import all_util File "/uv-venv/lib/python3.8/site-packages/tensorflow_probability/python/__init__.py", line 138, in dir(globals()[pkg_name]) # Forces loading the package from its lazy loader. File "/uv-venv/lib/python3.8/site-packages/tensorflow_probability/python/internal/lazy_loader.py", line 57, in __dir__ module = self._load() File "/uv-venv/lib/python3.8/site-packages/tensorflow_probability/python/internal/lazy_loader.py", line 37, in _load self._on_first_access() File "/uv-venv/lib/python3.8/site-packages/tensorflow_probability/python/__init__.py", line 59, in _validate_tf_environment raise ImportError( ImportError: This version of TensorFlow Probability requires TensorFlow version >= 2.12; Detected an installation of version 2.11.1. Please upgrade TensorFlow to proceed. (uv-venv) root@2c24021d2969:/pyhf# ```

uv-pip-install-verbose.txt

charliermarsh commented 4 months ago

Will take a look... My guess is that these are both "valid" resolutions (since uv did pin some packages to more recent versions), and the difference is due to semi-arbitrary choices in how resolution played out in the two systems. But there could be a bug somewhere.

Also confusing is that tensorflow_probability doesn't encode this constraint in its metadata, as far as I can tell? There's no tensorflow version constraint in tensorflow_probability.

matthewfeickert commented 4 months ago

Will take a look

Thanks!

Also confusing is that tensorflow_probability doesn't encode this constraint in its metadata, as far as I can tell? There's no tensorflow version constraint in tensorflow_probability.

Correct, tensorflow-probability doesn't specifically require tensorflow as a dependency as it can work with other libraries, but it then does the internal check on the tensorflow version seen in the error log if you attempt to import them together. I can make an additional reproducer later today, but I believe if you just create an environment with just tensorflow and tensorflow-probability then uv will match pip's solve.

notatallshaw commented 4 months ago

I'm speculating here, but I would make the guess that this is another example of uv backtracking on the package creating an upper bound requirement, rather than backtracking on the package being upper bounded, like in https://github.com/astral-sh/uv/issues/1398.

In this case tensorflow is upper bounding gast, grpcio, keras, numpy, protobuf, tensorboard, tensorflow-estimator, and wrapt. Skimming over the verbose logs I would say the most likely culprit is numpy. I can check later but I suspect the resolution is going to be better (and faster) with a pinned or lower bound requirement on numpy, or one of the other packages.

This kind of issue would came up a lot on the pip issue tracker after pip debuted it's resolver, that a new release of pip would cause a different solution. It's not that the resolver is incorrect, it's a kind of performance issue, if the user has strict lower bounds they want on certain packages they should include it in their requirements or constraints.

matthewfeickert commented 4 months ago

I can make an additional reproducer later today, but I believe if you just create an environment with just tensorflow and tensorflow-probability then uv will match pip's solve.

Yeah, for just installing tensorflow and tensorflow-probability the solves are the same

Using pip:

```console $ docker run --rm -ti python:3.8 /bin/bash root@0aa33f8bda2c:/# python -m venv venv && . venv/bin/activate (venv) root@0aa33f8bda2c:/# python -m pip --quiet install --upgrade pip setuptools wheel (venv) root@0aa33f8bda2c:/# python -m pip --verbose --no-cache-dir install --upgrade tensorflow tensorflow-probability &> /tmp/python-pip-install-verbose.txt (venv) root@0aa33f8bda2c:/# python -m pip list Package Version ---------------------------- -------- absl-py 2.1.0 astunparse 1.6.3 cachetools 5.3.3 certifi 2024.2.2 charset-normalizer 3.3.2 cloudpickle 3.0.0 decorator 5.1.1 dm-tree 0.1.8 flatbuffers 23.5.26 gast 0.4.0 google-auth 2.28.1 google-auth-oauthlib 1.0.0 google-pasta 0.2.0 grpcio 1.62.0 h5py 3.10.0 idna 3.6 importlib-metadata 7.0.1 keras 2.13.1 libclang 16.0.6 Markdown 3.5.2 MarkupSafe 2.1.5 numpy 1.24.3 oauthlib 3.2.2 opt-einsum 3.3.0 packaging 23.2 pip 24.0 protobuf 4.25.3 pyasn1 0.5.1 pyasn1-modules 0.3.0 requests 2.31.0 requests-oauthlib 1.3.1 rsa 4.9 setuptools 69.1.1 six 1.16.0 tensorboard 2.13.0 tensorboard-data-server 0.7.2 tensorflow 2.13.1 tensorflow-estimator 2.13.0 tensorflow-io-gcs-filesystem 0.34.0 tensorflow-probability 0.21.0 termcolor 2.4.0 typing_extensions 4.5.0 urllib3 2.2.1 Werkzeug 3.0.1 wheel 0.42.0 wrapt 1.16.0 zipp 3.17.0 (venv) root@0aa33f8bda2c:/# python -m pip list | grep tensor tensorboard 2.13.0 tensorboard-data-server 0.7.2 tensorflow 2.13.1 tensorflow-estimator 2.13.0 tensorflow-io-gcs-filesystem 0.34.0 tensorflow-probability 0.21.0 (venv) root@0aa33f8bda2c:/# ```

python-pip-install-verbose.txt

uv v0.1.13:

```console (venv) root@0aa33f8bda2c:/# deactivate root@0aa33f8bda2c:/# python -m venv /uv-venv && . /uv-venv/bin/activate (uv-venv) root@0aa33f8bda2c:/# python -m pip --quiet install --upgrade pip setuptools wheel uv (uv-venv) root@0aa33f8bda2c:/# uv --version uv 0.1.13 (uv-venv) root@0aa33f8bda2c:/# uv pip --verbose --no-cache install --upgrade tensorflow tensorflow-probability &> /tmp/uv-pip-install-verbose.txt (uv-venv) root@0aa33f8bda2c:/# uv pip list Package Version ---------------------------- -------- absl-py 2.1.0 astunparse 1.6.3 cachetools 5.3.3 certifi 2024.2.2 charset-normalizer 3.3.2 cloudpickle 3.0.0 decorator 5.1.1 dm-tree 0.1.8 flatbuffers 23.5.26 gast 0.4.0 google-auth 2.28.1 google-auth-oauthlib 1.0.0 google-pasta 0.2.0 grpcio 1.62.0 h5py 3.10.0 idna 3.6 importlib-metadata 7.0.1 keras 2.13.1 libclang 16.0.6 markdown 3.5.2 markupsafe 2.1.5 numpy 1.24.3 oauthlib 3.2.2 opt-einsum 3.3.0 packaging 23.2 pip 24.0 protobuf 4.25.3 pyasn1 0.5.1 pyasn1-modules 0.3.0 requests 2.31.0 requests-oauthlib 1.3.1 rsa 4.9 setuptools 69.1.1 six 1.16.0 tensorboard 2.13.0 tensorboard-data-server 0.7.2 tensorflow 2.13.1 tensorflow-estimator 2.13.0 tensorflow-io-gcs-filesystem 0.34.0 tensorflow-probability 0.21.0 termcolor 2.4.0 typing-extensions 4.5.0 urllib3 2.2.1 uv 0.1.13 werkzeug 3.0.1 wheel 0.42.0 wrapt 1.16.0 zipp 3.17.0 (uv-venv) root@0aa33f8bda2c:/# uv pip list | grep tensor tensorboard 2.13.0 tensorboard-data-server 0.7.2 tensorflow 2.13.1 tensorflow-estimator 2.13.0 tensorflow-io-gcs-filesystem 0.34.0 tensorflow-probability 0.21.0 (uv-venv) root@0aa33f8bda2c:/# ```

uv-pip-install-verbose.txt

Ghost-Terms commented 2 months ago

When trying to add tensorflow[and-cuda]==2.16.1, uv reports that nvidia-cublas-cu12==12.3.4.1 does not exist, even though it's right here: https://pypi.org/project/nvidia-cublas-cu12/12.3.4.1/

zanieb commented 2 months ago

@ImportTaste please open a new issue, that looks unrelated to this one.

matthewfeickert commented 2 months ago

Here's a more minimal reproducer of the problem, using pyhf at https://github.com/scikit-hep/pyhf/commit/08755e45ed134cbe6e5e6e238b7b5011700080b3 and

$ uv --version
uv 0.1.39

At this commit, pyhf's 'tensorflow' extra is the following

tensorflow = [
    # python == 3.8
    "tensorflow>=2.7.0; python_version < '3.9' and platform_machine != 'arm64'",  # c.f. PR #1962, #2452
    "tensorflow-macos>=2.7.0; python_version < '3.9' and platform_machine == 'arm64' and platform_system == 'Darwin'",  # c.f. PR #2119, #2452
    "tensorflow-probability>=0.11.0; python_version < '3.9'",  # c.f. PR #1657, #2452
    # python >= 3.9
    "tensorflow-probability[tf]>=0.24.0; python_version >= '3.9'"  # c.f. PR #2452
]

so on Python 3.8 (Linux) running

uv pip install '.[tensorflow]'

should be the same as running

uv pip install . 'tensorflow>=2.7.0' 'tensorflow-probability>=0.11.0'

yet these result in different environments, where I (as the developer) can say that the second environment is the expected solve.

uv v0.1.39 with tensorflow extra (failure case):

```console $ docker run --rm -ti -v /tmp:/tmp python:3.8 /bin/bash root@4935323db0ed:/# curl -LsSf https://astral.sh/uv/install.sh | sh downloading uv 0.1.39 x86_64-unknown-linux-gnu installing to /root/.cargo/bin uv everything's installed! To add $HOME/.cargo/bin to your PATH, either restart your shell or run: source $HOME/.cargo/env root@4935323db0ed:/# . ~/.cargo/env root@4935323db0ed:/# uv venv && . .venv/bin/activate Using Python 3.8.19 interpreter at: usr/local/bin/python3 Creating virtualenv at: .venv (.venv) root@4935323db0ed:/# git clone --single-branch https://github.com/scikit-hep/pyhf.git && cd pyhf && git reset --hard 08755e45ed134cbe6e5e6e238b7b5011700080b3 (.venv) root@4935323db0ed:/pyhf# uv --version uv 0.1.39 (.venv) root@4935323db0ed:/pyhf# uv pip --verbose --no-cache install '.[tensorflow]' &> /tmp/uv-pip-install-tensorflow-extra-verbose.txt (.venv) root@4935323db0ed:/pyhf# uv pip list Package Version ---------------------------- ------------ absl-py 2.1.0 astunparse 1.6.3 attrs 23.2.0 cachetools 5.3.3 certifi 2024.2.2 charset-normalizer 3.3.2 click 8.1.7 cloudpickle 3.0.0 decorator 5.1.1 dm-tree 0.1.8 flatbuffers 24.3.25 gast 0.4.0 google-auth 2.29.0 google-auth-oauthlib 0.4.6 google-pasta 0.2.0 grpcio 1.62.2 h5py 3.11.0 idna 3.7 importlib-metadata 7.1.0 importlib-resources 6.4.0 jsonpatch 1.33 jsonpointer 2.4 jsonschema 4.21.1 jsonschema-specifications 2023.12.1 keras 2.11.0 libclang 18.1.1 markdown 3.6 markupsafe 2.1.5 numpy 1.24.4 oauthlib 3.2.2 opt-einsum 3.3.0 packaging 24.0 pkgutil-resolve-name 1.3.10 protobuf 3.19.6 pyasn1 0.6.0 pyasn1-modules 0.4.0 pyhf 0.7.1.dev238 pyyaml 6.0.1 referencing 0.35.0 requests 2.31.0 requests-oauthlib 2.0.0 rpds-py 0.18.0 rsa 4.9 scipy 1.10.1 setuptools 69.5.1 six 1.16.0 tensorboard 2.11.2 tensorboard-data-server 0.6.1 tensorboard-plugin-wit 1.8.1 tensorflow 2.11.1 tensorflow-estimator 2.11.0 tensorflow-io-gcs-filesystem 0.34.0 tensorflow-probability 0.21.0 termcolor 2.4.0 tqdm 4.66.2 typing-extensions 4.5.0 urllib3 2.2.1 werkzeug 3.0.2 wheel 0.43.0 wrapt 1.16.0 zipp 3.18.1 (.venv) root@4935323db0ed:/pyhf# uv pip list | grep tensorflow tensorflow 2.11.1 tensorflow-estimator 2.11.0 tensorflow-io-gcs-filesystem 0.34.0 tensorflow-probability 0.21.0 (.venv) root@4935323db0ed:/pyhf# python -c 'import tensorflow; import tensorflow_probability' 2024-04-30 07:01:07.319640: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 AVX_VNNI FMA To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 2024-04-30 07:01:07.399553: I tensorflow/core/util/port.cc:104] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. 2024-04-30 07:01:07.402137: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory 2024-04-30 07:01:07.402156: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine. 2024-04-30 07:01:09.446572: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory 2024-04-30 07:01:09.446619: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory 2024-04-30 07:01:09.446626: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly. Traceback (most recent call last): File "", line 1, in File "/.venv/lib/python3.8/site-packages/tensorflow_probability/__init__.py", line 20, in from tensorflow_probability import substrates File "/.venv/lib/python3.8/site-packages/tensorflow_probability/substrates/__init__.py", line 17, in from tensorflow_probability.python.internal import all_util File "/.venv/lib/python3.8/site-packages/tensorflow_probability/python/__init__.py", line 138, in dir(globals()[pkg_name]) # Forces loading the package from its lazy loader. File "/.venv/lib/python3.8/site-packages/tensorflow_probability/python/internal/lazy_loader.py", line 57, in __dir__ module = self._load() File "/.venv/lib/python3.8/site-packages/tensorflow_probability/python/internal/lazy_loader.py", line 37, in _load self._on_first_access() File "/.venv/lib/python3.8/site-packages/tensorflow_probability/python/__init__.py", line 59, in _validate_tf_environment raise ImportError( ImportError: This version of TensorFlow Probability requires TensorFlow version >= 2.13; Detected an installation of version 2.11.1. Please upgrade TensorFlow to proceed. (.venv) root@4935323db0ed:/pyhf# ```

uv-pip-install-tensorflow-extra-verbose.txt

uv v0.1.39 with tensorflow,tfp on command line (success case):

```console $ docker run --rm -ti -v /tmp:/tmp python:3.8 /bin/bash root@edbb84aa61db:/# curl -LsSf https://astral.sh/uv/install.sh | sh downloading uv 0.1.39 x86_64-unknown-linux-gnu installing to /root/.cargo/bin uv everything's installed! To add $HOME/.cargo/bin to your PATH, either restart your shell or run: source $HOME/.cargo/env root@edbb84aa61db:/# . ~/.cargo/env root@edbb84aa61db:/# uv venv && . .venv/bin/activate Using Python 3.8.19 interpreter at: usr/local/bin/python3 Creating virtualenv at: .venv (.venv) root@edbb84aa61db:/# git clone --single-branch https://github.com/scikit-hep/pyhf.git && cd pyhf && git reset --hard 08755e45ed134cbe6e5e6e238b7b5011700080b3 (.venv) root@edbb84aa61db:/pyhf# uv pip --verbose --no-cache install . 'tensorflow>=2.7.0' 'tensorflow-probability>=0.11.0' &> /tmp/uv-pip-install-tensorflow-command-line-verbose.txt (.venv) root@edbb84aa61db:/pyhf# uv pip list Package Version ---------------------------- ------------ absl-py 2.1.0 astunparse 1.6.3 attrs 23.2.0 cachetools 5.3.3 certifi 2024.2.2 charset-normalizer 3.3.2 click 8.1.7 cloudpickle 3.0.0 decorator 5.1.1 dm-tree 0.1.8 flatbuffers 24.3.25 gast 0.4.0 google-auth 2.29.0 google-auth-oauthlib 1.0.0 google-pasta 0.2.0 grpcio 1.62.2 h5py 3.11.0 idna 3.7 importlib-metadata 7.1.0 importlib-resources 6.4.0 jsonpatch 1.33 jsonpointer 2.4 jsonschema 4.21.1 jsonschema-specifications 2023.12.1 keras 2.13.1 libclang 18.1.1 markdown 3.6 markupsafe 2.1.5 numpy 1.24.3 oauthlib 3.2.2 opt-einsum 3.3.0 packaging 24.0 pkgutil-resolve-name 1.3.10 protobuf 4.25.3 pyasn1 0.6.0 pyasn1-modules 0.4.0 pyhf 0.7.1.dev238 pyyaml 6.0.1 referencing 0.35.0 requests 2.31.0 requests-oauthlib 2.0.0 rpds-py 0.18.0 rsa 4.9 scipy 1.10.1 setuptools 69.5.1 six 1.16.0 tensorboard 2.13.0 tensorboard-data-server 0.7.2 tensorflow 2.13.1 tensorflow-estimator 2.13.0 tensorflow-io-gcs-filesystem 0.34.0 tensorflow-probability 0.21.0 termcolor 2.4.0 tqdm 4.66.2 typing-extensions 4.5.0 urllib3 2.2.1 werkzeug 3.0.2 wheel 0.43.0 wrapt 1.16.0 zipp 3.18.1 (.venv) root@edbb84aa61db:/pyhf# uv pip list | grep tensorflow tensorflow 2.13.1 tensorflow-estimator 2.13.0 tensorflow-io-gcs-filesystem 0.34.0 tensorflow-probability 0.21.0 (.venv) root@edbb84aa61db:/pyhf# python -c 'import tensorflow; import tensorflow_probability' 2024-04-30 07:08:21.163407: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. 2024-04-30 07:08:21.164774: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used. 2024-04-30 07:08:21.192250: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used. 2024-04-30 07:08:21.192613: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations. To enable the following instructions: AVX2 AVX_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags. 2024-04-30 07:08:23.284477: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT (.venv) root@edbb84aa61db:/pyhf# ```

uv-pip-install-tensorflow-command-line-verbose.txt

pip (success case):

```console $ docker run --rm -ti -v /tmp:/tmp python:3.8 /bin/bash root@0c9ba727a892:/# python -m venv .venv && . .venv/bin/activate (.venv) root@0c9ba727a892:/# python -m pip --quiet install --upgrade pip setuptools wheel (.venv) root@0c9ba727a892:/# git clone --single-branch https://github.com/scikit-hep/pyhf.git && cd pyhf && git reset --hard 08755e45ed134cbe6e5e6e238b7b5011700080b3 (.venv) root@0c9ba727a892:/pyhf# python -m pip --verbose --no-cache-dir install --upgrade '.[tensorflow]' &> /tmp/python-pip-install-tensorflow-extra-verbose.txt (.venv) root@0c9ba727a892:/pyhf# python -m pip list Package Version ---------------------------- ------------ absl-py 2.1.0 astunparse 1.6.3 attrs 23.2.0 cachetools 5.3.3 certifi 2024.2.2 charset-normalizer 3.3.2 click 8.1.7 cloudpickle 3.0.0 decorator 5.1.1 dm-tree 0.1.8 flatbuffers 24.3.25 gast 0.4.0 google-auth 2.29.0 google-auth-oauthlib 1.0.0 google-pasta 0.2.0 grpcio 1.62.2 h5py 3.11.0 idna 3.7 importlib_metadata 7.1.0 importlib_resources 6.4.0 jsonpatch 1.33 jsonpointer 2.4 jsonschema 4.21.1 jsonschema-specifications 2023.12.1 keras 2.13.1 libclang 18.1.1 Markdown 3.6 MarkupSafe 2.1.5 numpy 1.24.3 oauthlib 3.2.2 opt-einsum 3.3.0 packaging 24.0 pip 24.0 pkgutil_resolve_name 1.3.10 protobuf 4.25.3 pyasn1 0.6.0 pyasn1_modules 0.4.0 pyhf 0.7.1.dev238 PyYAML 6.0.1 referencing 0.35.0 requests 2.31.0 requests-oauthlib 2.0.0 rpds-py 0.18.0 rsa 4.9 scipy 1.10.1 setuptools 69.5.1 six 1.16.0 tensorboard 2.13.0 tensorboard-data-server 0.7.2 tensorflow 2.13.1 tensorflow-estimator 2.13.0 tensorflow-io-gcs-filesystem 0.34.0 tensorflow-probability 0.21.0 termcolor 2.4.0 tqdm 4.66.2 typing_extensions 4.5.0 urllib3 2.2.1 Werkzeug 3.0.2 wheel 0.43.0 wrapt 1.16.0 zipp 3.18.1 (.venv) root@0c9ba727a892:/pyhf# python -m pip list | grep tensorflow tensorflow 2.13.1 tensorflow-estimator 2.13.0 tensorflow-io-gcs-filesystem 0.34.0 tensorflow-probability 0.21.0 (.venv) root@0c9ba727a892:/pyhf# python -c 'import tensorflow; import tensorflow_probability' 2024-04-30 07:14:14.122645: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. 2024-04-30 07:14:14.124300: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used. 2024-04-30 07:14:14.155920: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used. 2024-04-30 07:14:14.156250: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations. To enable the following instructions: AVX2 AVX_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags. 2024-04-30 07:14:14.749372: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT (.venv) root@0c9ba727a892:/pyhf# ```

python-pip-install-tensorflow-extra-verbose.txt

astral-sh / uv

uv pip and python -m pip resolve different versions of tensorflow in pyhf developer environment #2062