search

tensorflow / tensorflow

An Open Source Machine Learning Framework for Everyone
https://tensorflow.org
Apache License 2.0
186.29k stars 74.31k forks source link

tensorflow-macos==2.15.0 still requesting ml-dtypes (~=0.2.0) instead of 0.3.1 #62746

Open mathpluscode opened 10 months ago

mathpluscode commented 10 months ago

Issue type

Bug

Have you reproduced the bug with TensorFlow Nightly?

No

Source

binary

TensorFlow version

2.15.0

Custom code

No

OS platform and distribution

MacOS 14.1.2

Mobile device

Mac

Python version

3.9

Bazel version

No response

GCC/compiler version

No response

CUDA/cuDNN version

No response

GPU model and memory

No response

Current behavior?

Currently, we could not install the following packages together

The conflict is caused by:
    tensorflow-macos 2.15.0 depends on ml-dtypes~=0.2.0
    jax 0.4.23 depends on ml-dtypes>=0.2.0
    jaxlib 0.4.23 depends on ml-dtypes>=0.2.0
    tensorstore 0.1.52 depends on ml-dtypes>=0.3.1
    tensorflow-macos 2.15.0 depends on ml-dtypes~=0.2.0
    jax 0.4.23 depends on ml-dtypes>=0.2.0
    jaxlib 0.4.23 depends on ml-dtypes>=0.2.0
    tensorstore 0.1.51 depends on ml-dtypes>=0.3.1

From the requirement, it seems that it should require ml-dtypes==0.3.1.

But if running

curl -s https://pypi.org/pypi/tensorflow/2.15.0/json | jq '.info.requires_dist' | grep 'ml-dtypes'

We will get "ml-dtypes (~=0.2.0)",.

Note, it does work without orbax-checkpoint which is now recommended by Flax for checkpointing.

Standalone code to reproduce the issue

name: debug
channels:
  - defaults
dependencies:
  - python=3.9
  - pip=23.3.1
  - pip:
- tensorflow-macos==2.15.0
- tensorflow-metal==1.1.0
- jax==0.4.23
- jaxlib==0.4.23
- orbax-checkpoint==0.4.8

Please put this into `environment_mac_m1_debug.yml` then execute `conda env create -f environment_mac_m1_debug.yml`.

Relevant log output

conda env create -f environment_mac_m1_debug.yml                                                                                                                                                 ok  base py
Channels:
 - defaults
 - conda-forge
Platform: osx-arm64
Collecting package metadata (repodata.json): done
Solving environment: done

==> WARNING: A newer version of conda exists. <==
    current version: 23.3.1
    latest version: 23.11.0

Please update conda by running

    $ conda update -n base -c conda-forge conda

Downloading and Extracting Packages

Preparing transaction: done
Verifying transaction: done
Executing transaction: done
Installing pip dependencies: \ Ran pip subprocess with arguments:
['/Users/user/miniforge3/envs/debug/bin/python', '-m', 'pip', 'install', '-U', '-r', '/Users/user/condaenv.1tnhkmib.requirements.txt', '--exists-action=b']
Pip subprocess output:
Collecting tensorflow-macos==2.15.0 (from -r /Users/user/condaenv.1tnhkmib.requirements.txt (line 1))
  Using cached tensorflow_macos-2.15.0-cp39-cp39-macosx_12_0_arm64.whl.metadata (4.2 kB)
Collecting tensorflow-metal==1.1.0 (from -r /Users/user/condaenv.1tnhkmib.requirements.txt (line 2))
  Downloading tensorflow_metal-1.1.0-cp39-cp39-macosx_12_0_arm64.whl.metadata (1.2 kB)
Collecting jax==0.4.23 (from -r /Users/user/condaenv.1tnhkmib.requirements.txt (line 3))
  Using cached jax-0.4.23-py3-none-any.whl.metadata (24 kB)
Collecting jaxlib==0.4.23 (from -r /Users/user/condaenv.1tnhkmib.requirements.txt (line 4))
  Using cached jaxlib-0.4.23-cp39-cp39-macosx_11_0_arm64.whl.metadata (2.1 kB)
Collecting orbax-checkpoint==0.4.8 (from -r /Users/user/condaenv.1tnhkmib.requirements.txt (line 5))
  Using cached orbax_checkpoint-0.4.8-py3-none-any.whl.metadata (1.7 kB)
Collecting tensorflow-datasets==4.9.3 (from -r /Users/user/condaenv.1tnhkmib.requirements.txt (line 6))
  Downloading tensorflow_datasets-4.9.3-py3-none-any.whl.metadata (9.3 kB)
Collecting absl-py>=1.0.0 (from tensorflow-macos==2.15.0->-r /Users/user/condaenv.1tnhkmib.requirements.txt (line 1))
  Downloading absl_py-2.0.0-py3-none-any.whl.metadata (2.3 kB)
Collecting astunparse>=1.6.0 (from tensorflow-macos==2.15.0->-r /Users/user/condaenv.1tnhkmib.requirements.txt (line 1))
  Downloading astunparse-1.6.3-py2.py3-none-any.whl (12 kB)
Collecting flatbuffers>=23.5.26 (from tensorflow-macos==2.15.0->-r /Users/user/condaenv.1tnhkmib.requirements.txt (line 1))
  Downloading flatbuffers-23.5.26-py2.py3-none-any.whl.metadata (850 bytes)
Collecting gast!=0.5.0,!=0.5.1,!=0.5.2,>=0.2.1 (from tensorflow-macos==2.15.0->-r /Users/user/condaenv.1tnhkmib.requirements.txt (line 1))
  Downloading gast-0.5.4-py3-none-any.whl (19 kB)
Collecting google-pasta>=0.1.1 (from tensorflow-macos==2.15.0->-r /Users/user/condaenv.1tnhkmib.requirements.txt (line 1))
  Downloading google_pasta-0.2.0-py3-none-any.whl (57 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 57.5/57.5 kB 45.2 kB/s eta 0:00:00
Collecting h5py>=2.9.0 (from tensorflow-macos==2.15.0->-r /Users/user/condaenv.1tnhkmib.requirements.txt (line 1))
  Downloading h5py-3.10.0-cp39-cp39-macosx_11_0_arm64.whl.metadata (2.5 kB)
Collecting libclang>=13.0.0 (from tensorflow-macos==2.15.0->-r /Users/user/condaenv.1tnhkmib.requirements.txt (line 1))
  Downloading libclang-16.0.6-py2.py3-none-macosx_11_0_arm64.whl.metadata (5.2 kB)
Collecting ml-dtypes~=0.2.0 (from tensorflow-macos==2.15.0->-r /Users/user/condaenv.1tnhkmib.requirements.txt (line 1))
  Downloading ml_dtypes-0.2.0-cp39-cp39-macosx_10_9_universal2.whl.metadata (20 kB)
Collecting numpy<2.0.0,>=1.23.5 (from tensorflow-macos==2.15.0->-r /Users/user/condaenv.1tnhkmib.requirements.txt (line 1))
  Downloading numpy-1.26.3-cp39-cp39-macosx_11_0_arm64.whl.metadata (61 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 61.2/61.2 kB 60.0 kB/s eta 0:00:00
Collecting opt-einsum>=2.3.2 (from tensorflow-macos==2.15.0->-r /Users/user/condaenv.1tnhkmib.requirements.txt (line 1))
  Using cached opt_einsum-3.3.0-py3-none-any.whl (65 kB)
Collecting packaging (from tensorflow-macos==2.15.0->-r /Users/user/condaenv.1tnhkmib.requirements.txt (line 1))
  Using cached packaging-23.2-py3-none-any.whl.metadata (3.2 kB)
Collecting protobuf!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.20.3 (from tensorflow-macos==2.15.0->-r /Users/user/condaenv.1tnhkmib.requirements.txt (line 1))
  Downloading protobuf-4.25.1-cp37-abi3-macosx_10_9_universal2.whl.metadata (541 bytes)
Requirement already satisfied: setuptools in /Users/user/miniforge3/envs/debug/lib/python3.9/site-packages (from tensorflow-macos==2.15.0->-r /Users/user/condaenv.1tnhkmib.requirements.txt (line 1)) (68.2.2)
Collecting six>=1.12.0 (from tensorflow-macos==2.15.0->-r /Users/user/condaenv.1tnhkmib.requirements.txt (line 1))
  Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting termcolor>=1.1.0 (from tensorflow-macos==2.15.0->-r /Users/user/condaenv.1tnhkmib.requirements.txt (line 1))
  Downloading termcolor-2.4.0-py3-none-any.whl.metadata (6.1 kB)
Collecting typing-extensions>=3.6.6 (from tensorflow-macos==2.15.0->-r /Users/user/condaenv.1tnhkmib.requirements.txt (line 1))
  Using cached typing_extensions-4.9.0-py3-none-any.whl.metadata (3.0 kB)
Collecting wrapt<1.15,>=1.11.0 (from tensorflow-macos==2.15.0->-r /Users/user/condaenv.1tnhkmib.requirements.txt (line 1))
  Downloading wrapt-1.14.1-cp39-cp39-macosx_11_0_arm64.whl (35 kB)
Collecting tensorflow-io-gcs-filesystem>=0.23.1 (from tensorflow-macos==2.15.0->-r /Users/user/condaenv.1tnhkmib.requirements.txt (line 1))
  Downloading tensorflow_io_gcs_filesystem-0.34.0-cp39-cp39-macosx_12_0_arm64.whl.metadata (14 kB)
Collecting grpcio<2.0,>=1.24.3 (from tensorflow-macos==2.15.0->-r /Users/user/condaenv.1tnhkmib.requirements.txt (line 1))
  Downloading grpcio-1.60.0-cp39-cp39-macosx_10_10_universal2.whl.metadata (4.0 kB)
Collecting tensorboard<2.16,>=2.15 (from tensorflow-macos==2.15.0->-r /Users/user/condaenv.1tnhkmib.requirements.txt (line 1))
  Using cached tensorboard-2.15.1-py3-none-any.whl.metadata (1.7 kB)
Collecting tensorflow-estimator<2.16,>=2.15.0 (from tensorflow-macos==2.15.0->-r /Users/user/condaenv.1tnhkmib.requirements.txt (line 1))
  Using cached tensorflow_estimator-2.15.0-py2.py3-none-any.whl.metadata (1.3 kB)
Collecting keras<2.16,>=2.15.0 (from tensorflow-macos==2.15.0->-r /Users/user/condaenv.1tnhkmib.requirements.txt (line 1))
  Using cached keras-2.15.0-py3-none-any.whl.metadata (2.4 kB)
Requirement already satisfied: wheel~=0.35 in /Users/user/miniforge3/envs/debug/lib/python3.9/site-packages (from tensorflow-metal==1.1.0->-r /Users/user/condaenv.1tnhkmib.requirements.txt (line 2)) (0.41.2)
Collecting scipy>=1.9 (from jax==0.4.23->-r /Users/user/condaenv.1tnhkmib.requirements.txt (line 3))
  Downloading scipy-1.11.4-cp39-cp39-macosx_12_0_arm64.whl.metadata (60 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 60.4/60.4 kB 21.7 kB/s eta 0:00:00
Collecting importlib-metadata>=4.6 (from jax==0.4.23->-r /Users/user/condaenv.1tnhkmib.requirements.txt (line 3))
  Downloading importlib_metadata-7.0.1-py3-none-any.whl.metadata (4.9 kB)
Collecting etils[epath,epy] (from orbax-checkpoint==0.4.8->-r /Users/user/condaenv.1tnhkmib.requirements.txt (line 5))
  Downloading etils-1.5.2-py3-none-any.whl.metadata (6.3 kB)
Collecting msgpack (from orbax-checkpoint==0.4.8->-r /Users/user/condaenv.1tnhkmib.requirements.txt (line 5))
  Downloading msgpack-1.0.7-cp39-cp39-macosx_11_0_arm64.whl.metadata (9.1 kB)
Collecting pyyaml (from orbax-checkpoint==0.4.8->-r /Users/user/condaenv.1tnhkmib.requirements.txt (line 5))
  Downloading PyYAML-6.0.1-cp39-cp39-macosx_11_0_arm64.whl.metadata (2.1 kB)
Collecting tensorstore>=0.1.51 (from orbax-checkpoint==0.4.8->-r /Users/user/condaenv.1tnhkmib.requirements.txt (line 5))
  Using cached tensorstore-0.1.52-cp39-cp39-macosx_11_0_arm64.whl.metadata (3.0 kB)
Collecting nest_asyncio (from orbax-checkpoint==0.4.8->-r /Users/user/condaenv.1tnhkmib.requirements.txt (line 5))
  Downloading nest_asyncio-1.5.8-py3-none-any.whl.metadata (2.8 kB)
Collecting array-record (from tensorflow-datasets==4.9.3->-r /Users/user/condaenv.1tnhkmib.requirements.txt (line 6))
  Downloading array_record-0.4.1-py39-none-any.whl.metadata (503 bytes)
Collecting click (from tensorflow-datasets==4.9.3->-r /Users/user/condaenv.1tnhkmib.requirements.txt (line 6))
  Using cached click-8.1.7-py3-none-any.whl.metadata (3.0 kB)
Collecting dm-tree (from tensorflow-datasets==4.9.3->-r /Users/user/condaenv.1tnhkmib.requirements.txt (line 6))
  Downloading dm_tree-0.1.8-cp39-cp39-macosx_11_0_arm64.whl (110 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 110.8/110.8 kB 28.6 kB/s eta 0:00:00
Collecting promise (from tensorflow-datasets==4.9.3->-r /Users/user/condaenv.1tnhkmib.requirements.txt (line 6))
  Using cached promise-2.3-py3-none-any.whl
Collecting psutil (from tensorflow-datasets==4.9.3->-r /Users/user/condaenv.1tnhkmib.requirements.txt (line 6))
  Downloading psutil-5.9.7-cp38-abi3-macosx_11_0_arm64.whl.metadata (21 kB)
Collecting requests>=2.19.0 (from tensorflow-datasets==4.9.3->-r /Users/user/condaenv.1tnhkmib.requirements.txt (line 6))
  Using cached requests-2.31.0-py3-none-any.whl.metadata (4.6 kB)
Collecting tensorflow-metadata (from tensorflow-datasets==4.9.3->-r /Users/user/condaenv.1tnhkmib.requirements.txt (line 6))
  Downloading tensorflow_metadata-1.14.0-py3-none-any.whl.metadata (2.1 kB)
Collecting toml (from tensorflow-datasets==4.9.3->-r /Users/user/condaenv.1tnhkmib.requirements.txt (line 6))
  Downloading toml-0.10.2-py2.py3-none-any.whl (16 kB)
Collecting tqdm (from tensorflow-datasets==4.9.3->-r /Users/user/condaenv.1tnhkmib.requirements.txt (line 6))
  Downloading tqdm-4.66.1-py3-none-any.whl.metadata (57 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 57.6/57.6 kB 40.7 kB/s eta 0:00:00
Collecting fsspec (from etils[enp,epath,etree]>=0.9.0->tensorflow-datasets==4.9.3->-r /Users/user/condaenv.1tnhkmib.requirements.txt (line 6))
  Downloading fsspec-2023.12.2-py3-none-any.whl.metadata (6.8 kB)
Collecting importlib_resources (from etils[enp,epath,etree]>=0.9.0->tensorflow-datasets==4.9.3->-r /Users/user/condaenv.1tnhkmib.requirements.txt (line 6))
  Downloading importlib_resources-6.1.1-py3-none-any.whl.metadata (4.1 kB)
Collecting zipp (from etils[enp,epath,etree]>=0.9.0->tensorflow-datasets==4.9.3->-r /Users/user/condaenv.1tnhkmib.requirements.txt (line 6))
  Downloading zipp-3.17.0-py3-none-any.whl.metadata (3.7 kB)
Collecting charset-normalizer<4,>=2 (from requests>=2.19.0->tensorflow-datasets==4.9.3->-r /Users/user/condaenv.1tnhkmib.requirements.txt (line 6))
  Downloading charset_normalizer-3.3.2-cp39-cp39-macosx_11_0_arm64.whl.metadata (33 kB)
Collecting idna<4,>=2.5 (from requests>=2.19.0->tensorflow-datasets==4.9.3->-r /Users/user/condaenv.1tnhkmib.requirements.txt (line 6))
  Downloading idna-3.6-py3-none-any.whl.metadata (9.9 kB)
Collecting urllib3<3,>=1.21.1 (from requests>=2.19.0->tensorflow-datasets==4.9.3->-r /Users/user/condaenv.1tnhkmib.requirements.txt (line 6))
  Downloading urllib3-2.1.0-py3-none-any.whl.metadata (6.4 kB)
Collecting certifi>=2017.4.17 (from requests>=2.19.0->tensorflow-datasets==4.9.3->-r /Users/user/condaenv.1tnhkmib.requirements.txt (line 6))
  Downloading certifi-2023.11.17-py3-none-any.whl.metadata (2.2 kB)
Collecting google-auth<3,>=1.6.3 (from tensorboard<2.16,>=2.15->tensorflow-macos==2.15.0->-r /Users/user/condaenv.1tnhkmib.requirements.txt (line 1))
  Downloading google_auth-2.26.1-py2.py3-none-any.whl.metadata (4.7 kB)
Collecting google-auth-oauthlib<2,>=0.5 (from tensorboard<2.16,>=2.15->tensorflow-macos==2.15.0->-r /Users/user/condaenv.1tnhkmib.requirements.txt (line 1))
  Downloading google_auth_oauthlib-1.2.0-py2.py3-none-any.whl.metadata (2.7 kB)
Collecting markdown>=2.6.8 (from tensorboard<2.16,>=2.15->tensorflow-macos==2.15.0->-r /Users/user/condaenv.1tnhkmib.requirements.txt (line 1))
  Downloading Markdown-3.5.1-py3-none-any.whl.metadata (7.1 kB)
Collecting protobuf!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.20.3 (from tensorflow-macos==2.15.0->-r /Users/user/condaenv.1tnhkmib.requirements.txt (line 1))
  Downloading protobuf-4.23.4-cp37-abi3-macosx_10_9_universal2.whl.metadata (540 bytes)
Collecting tensorboard-data-server<0.8.0,>=0.7.0 (from tensorboard<2.16,>=2.15->tensorflow-macos==2.15.0->-r /Users/user/condaenv.1tnhkmib.requirements.txt (line 1))
  Downloading tensorboard_data_server-0.7.2-py3-none-any.whl.metadata (1.1 kB)
Collecting werkzeug>=1.0.1 (from tensorboard<2.16,>=2.15->tensorflow-macos==2.15.0->-r /Users/user/condaenv.1tnhkmib.requirements.txt (line 1))
  Downloading werkzeug-3.0.1-py3-none-any.whl.metadata (4.1 kB)
INFO: pip is looking at multiple versions of tensorstore to determine which version is compatible with other requirements. This could take a while.
Collecting tensorstore>=0.1.51 (from orbax-checkpoint==0.4.8->-r /Users/user/condaenv.1tnhkmib.requirements.txt (line 5))
  Using cached tensorstore-0.1.51-cp39-cp39-macosx_11_0_arm64.whl.metadata (3.0 kB)

The conflict is caused by:
    tensorflow-macos 2.15.0 depends on ml-dtypes~=0.2.0
    jax 0.4.23 depends on ml-dtypes>=0.2.0
    jaxlib 0.4.23 depends on ml-dtypes>=0.2.0
    tensorstore 0.1.52 depends on ml-dtypes>=0.3.1
    tensorflow-macos 2.15.0 depends on ml-dtypes~=0.2.0
    jax 0.4.23 depends on ml-dtypes>=0.2.0
    jaxlib 0.4.23 depends on ml-dtypes>=0.2.0
    tensorstore 0.1.51 depends on ml-dtypes>=0.3.1

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

Pip subprocess error:
ERROR: Cannot install -r /Users/user/condaenv.1tnhkmib.requirements.txt (line 1), -r /Users/user/condaenv.1tnhkmib.requirements.txt (line 3), -r /Users/user/condaenv.1tnhkmib.requirements.txt (line 4) and orbax-checkpoint because these package versions have conflicting dependencies.
ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts

failed

CondaEnvException: Pip failed
SuryanarayanaY commented 10 months ago

Hi @mathpluscode ,

The requirements updated for master branch which is having ml-dtypes~= 0.3.1 now.

This is an issue with Tf2.15v with tensorflow package also apart from tensorflow-macos. In setup.py ml-dtypes~= 0.2.0 but for all 3 requirements_lock_3_x.txt it's still showing ml-dtypes~= 0.3.1 . Need to check on this. Thanks!

SuryanarayanaY commented 10 months ago

CC: @nitins17 , @learning-to-play , For your review please.

jakeBass commented 8 months ago

Any updates on a resolution to this issue? cc @nitins17 @learning-to-play

deeplearningrobotics commented 8 months ago

We are having the same problem on Linux. This issue is especially problematic as it affects Jax and related packages which demand a newer ml-dypes version.

jakeBass commented 8 months ago

If I'm understanding correctly, this appears to already be fixed on master:

So it seems like the operative questions are:

  1. Am I understanding correctly?
  2. When will 2.15.1 be released?
nitins17 commented 8 months ago

Thanks for letting us know! #63020 updates the ml-dtypes version to 0.3.1. It should be included in the upcoming patch release for TF 2.15.

When will 2.15.1 be released?

I do not know the exact date but believe it should be out by end of March. Adding @learning-to-play to confirm.

learning-to-play commented 8 months ago

@jakeBass TF 2.15 patch release is planned to be released before the end of March.

@nitins17 Thank you for cherry picking the fix to the r2.15 branch.