[bug] `platform_system ==` `"Darwin"`|`"Linux"` used on Win10 ⇒ `pdm add tensorflow>=2.16` fails

goyalyashpal commented 4 weeks ago

[x] I have searched the issue tracker and believe that this is not a duplicate.

Summaries & Titles

[bug] platform_system == "Darwin"|"Linux" used on Win10 ⇒ pdm add tensorflow>=2.16 fails
[bug] pdm add uses platform_system == "Darwin" or "Linux" on Win10 os hence pdm fails to resolve tensorflow to tensorflow-intel package
pdm add tensorflow>=2.16 fails, pip install tensorflow>=2.16 succeeds on windows

Steps to reproduce

run pdm lock with pyproject.toml having specifics as given:

# file: pyproject.toml
[project]
dependencies = [
    # "keras",
    "tensorflow>=2.16",
]
requires-python = "<3.13,>=3.12"

[build-system]
requires = ["pdm-backend"]
build-backend = "pdm.backend"

Actual behavior

> pdm lock -v
... STATUS ...
... NORMAL OUTPUT ...
... TRACEBACK ...

normal output

``` 🔒 Lock failed ERROR: Unable to find a resolution for tensorflow-cpu-aws because of the following conflicts: tensorflow-cpu-aws==2.16.1; (platform_machine == "arm64" or platform_machine == "aarch64") and platform_system == "Linux" (from tensorflow@2.16.1) To fix this, you could loosen the dependency version constraints in pyproject.toml. See https://pdm-project.org/en/latest/usage/lockfile/#solve-the-locking-failure for more details. See $LOCALAPPDATA\pdm\pdm\Logs\pdm-lock-kt_3eb6f.log for detailed debug log. [ResolutionImpossible]: Unable to find a resolution WARNING: Add '-v' to see the detailed traceback ```

verbose status

``` STATUS: Resolving dependencies pdm.termui: ======== Start resolving requirements ======== pdm.termui: tensorflow>=2.16 pdm.termui: python<3.13,>=3.12 pdm.termui: Adding requirement tensorflow>=2.16 pdm.termui: Adding requirement python<3.13,>=3.12 pdm.termui: ======== Starting round 0 ======== STATUS: Resolving: new pin python<3.13,>=3.12 pdm.termui: Pinning: python None pdm.termui: ======== Ending round 0 ======== pdm.termui: ======== Starting round 1 ======== pdm.termui: Adding requirement tensorflow==2.16.1; platform_system == "Darwin" and platform_machine == "arm64"(from tensorflow 2.16.1) pdm.termui: Adding requirement tensorflow-cpu-aws==2.16.1; (platform_machine == "arm64" or platform_machine == "aarch64") and platform_system == "Linux"(from tensorflow 2.16.1) pdm.termui: Candidate rejected: tensorflow@2.16.1 because it introduces a new requirement tensorflow-cpu-aws==2.16.1; (platform_machine == "arm64" or platform_machine == "aarch64") and platform_system == "Linux" that conflicts with other requirements: pdm.termui: Adding requirement tensorflow==2.16.1; platform_system == "Darwin" and platform_machine == "arm64"(from tensorflow 2.16.1) pdm.termui: Adding requirement tensorflow-cpu-aws==2.16.1; (platform_machine == "arm64" or platform_machine == "aarch64") and platform_system == "Linux"(from tensorflow 2.16.1) pdm.termui: Candidate rejected: tensorflow@2.16.1 because it introduces a new requirement tensorflow-cpu-aws==2.16.1; (platform_machine == "arm64" or platform_machine == "aarch64") and platform_system == "Linux" that conflicts with other requirements: pdm.termui: Adding requirement tensorflow==2.16.1; platform_system == "Darwin" and platform_machine == "arm64"(from tensorflow 2.16.1) pdm.termui: Adding requirement tensorflow-cpu-aws==2.16.1; (platform_machine == "arm64" or platform_machine == "aarch64") and platform_system == "Linux"(from tensorflow 2.16.1) pdm.termui: Candidate rejected: tensorflow@2.16.1 because it introduces a new requirement tensorflow-cpu-aws==2.16.1; (platform_machine == "arm64" or platform_machine == "aarch64") and platform_system == "Linux" that conflicts with other requirements: pdm.termui: Adding requirement tensorflow==2.16.1; platform_system == "Darwin" and platform_machine == "arm64"(from tensorflow 2.16.1) pdm.termui: Adding requirement tensorflow-cpu-aws==2.16.1; (platform_machine == "arm64" or platform_machine == "aarch64") and platform_system == "Linux"(from tensorflow 2.16.1) pdm.termui: Candidate rejected: tensorflow@2.16.1 because it introduces a new requirement tensorflow-cpu-aws==2.16.1; (platform_machine == "arm64" or platform_machine == "aarch64") and platform_system == "Linux" that conflicts with other requirements: pdm.termui: Adding requirement tensorflow==2.16.1; platform_system == "Darwin" and platform_machine == "arm64"(from tensorflow 2.16.1) pdm.termui: Adding requirement tensorflow-cpu-aws==2.16.1; (platform_machine == "arm64" or platform_machine == "aarch64") and platform_system == "Linux"(from tensorflow 2.16.1) pdm.termui: Candidate rejected: tensorflow@2.16.1 because it introduces a new requirement tensorflow-cpu-aws==2.16.1; (platform_machine == "arm64" or platform_machine == "aarch64") and platform_system == "Linux" that conflicts with other requirements: pdm.termui: Adding requirement tensorflow==2.16.1; platform_system == "Darwin" and platform_machine == "arm64"(from tensorflow 2.16.1) pdm.termui: Adding requirement tensorflow-cpu-aws==2.16.1; (platform_machine == "arm64" or platform_machine == "aarch64") and platform_system == "Linux"(from tensorflow 2.16.1) pdm.termui: Candidate rejected: tensorflow@2.16.1 because it introduces a new requirement tensorflow-cpu-aws==2.16.1; (platform_machine == "arm64" or platform_machine == "aarch64") and platform_system == "Linux" that conflicts with other requirements: pdm.termui: Adding requirement tensorflow==2.16.1; platform_system == "Darwin" and platform_machine == "arm64"(from tensorflow 2.16.1) pdm.termui: Adding requirement tensorflow-cpu-aws==2.16.1; (platform_machine == "arm64" or platform_machine == "aarch64") and platform_system == "Linux"(from tensorflow 2.16.1) pdm.termui: Candidate rejected: tensorflow@2.16.1 because it introduces a new requirement tensorflow-cpu-aws==2.16.1; (platform_machine == "arm64" or platform_machine == "aarch64") and platform_system == "Linux" that conflicts with other requirements: pdm.termui: Adding requirement tensorflow==2.16.1; platform_system == "Darwin" and platform_machine == "arm64"(from tensorflow 2.16.1) pdm.termui: Adding requirement tensorflow-cpu-aws==2.16.1; (platform_machine == "arm64" or platform_machine == "aarch64") and platform_system == "Linux"(from tensorflow 2.16.1) pdm.termui: Candidate rejected: tensorflow@2.16.1 because it introduces a new requirement tensorflow-cpu-aws==2.16.1; (platform_machine == "arm64" or platform_machine == "aarch64") and platform_system == "Linux" that conflicts with other requirements: pdm.termui: Adding requirement tensorflow==2.16.1; platform_system == "Darwin" and platform_machine == "arm64"(from tensorflow 2.16.1) pdm.termui: Adding requirement tensorflow-cpu-aws==2.16.1; (platform_machine == "arm64" or platform_machine == "aarch64") and platform_system == "Linux"(from tensorflow 2.16.1) pdm.termui: Candidate rejected: tensorflow@2.16.1 because it introduces a new requirement tensorflow-cpu-aws==2.16.1; (platform_machine == "arm64" or platform_machine == "aarch64") and platform_system == "Linux" that conflicts with other requirements: pdm.termui: Adding requirement tensorflow==2.16.1; platform_system == "Darwin" and platform_machine == "arm64"(from tensorflow 2.16.1) pdm.termui: Adding requirement tensorflow-cpu-aws==2.16.1; (platform_machine == "arm64" or platform_machine == "aarch64") and platform_system == "Linux"(from tensorflow 2.16.1) pdm.termui: Candidate rejected: tensorflow@2.16.1 because it introduces a new requirement tensorflow-cpu-aws==2.16.1; (platform_machine == "arm64" or platform_machine == "aarch64") and platform_system == "Linux" that conflicts with other requirements: pdm.termui: Adding requirement tensorflow==2.16.1; platform_system == "Darwin" and platform_machine == "arm64"(from tensorflow 2.16.1) pdm.termui: Adding requirement tensorflow-cpu-aws==2.16.1; (platform_machine == "arm64" or platform_machine == "aarch64") and platform_system == "Linux"(from tensorflow 2.16.1) pdm.termui: Candidate rejected: tensorflow@2.16.1 because it introduces a new requirement tensorflow-cpu-aws==2.16.1; (platform_machine == "arm64" or platform_machine == "aarch64") and platform_system == "Linux" that conflicts with other requirements: pdm.termui: Adding requirement tensorflow==2.16.1; platform_system == "Darwin" and platform_machine == "arm64"(from tensorflow 2.16.1) pdm.termui: Adding requirement tensorflow-cpu-aws==2.16.1; (platform_machine == "arm64" or platform_machine == "aarch64") and platform_system == "Linux"(from tensorflow 2.16.1) pdm.termui: Candidate rejected: tensorflow@2.16.1 because it introduces a new requirement tensorflow-cpu-aws==2.16.1; (platform_machine == "arm64" or platform_machine == "aarch64") and platform_system == "Linux" that conflicts with other requirements: pdm.termui: Adding requirement tensorflow==2.16.1; platform_system == "Darwin" and platform_machine == "arm64"(from tensorflow 2.16.1) pdm.termui: Adding requirement tensorflow-cpu-aws==2.16.1; (platform_machine == "arm64" or platform_machine == "aarch64") and platform_system == "Linux"(from tensorflow 2.16.1) pdm.termui: Candidate rejected: tensorflow@2.16.1 because it introduces a new requirement tensorflow-cpu-aws==2.16.1; (platform_machine == "arm64" or platform_machine == "aarch64") and platform_system == "Linux" that conflicts with other requirements: pdm.termui: Adding requirement tensorflow==2.16.1; platform_system == "Darwin" and platform_machine == "arm64"(from tensorflow 2.16.1) pdm.termui: Adding requirement tensorflow-cpu-aws==2.16.1; (platform_machine == "arm64" or platform_machine == "aarch64") and platform_system == "Linux"(from tensorflow 2.16.1) pdm.termui: Candidate rejected: tensorflow@2.16.1 because it introduces a new requirement tensorflow-cpu-aws==2.16.1; (platform_machine == "arm64" or platform_machine == "aarch64") and platform_system == "Linux" that conflicts with other requirements: pdm.termui: Adding requirement tensorflow==2.16.1; platform_system == "Darwin" and platform_machine == "arm64"(from tensorflow 2.16.1) pdm.termui: Adding requirement tensorflow-cpu-aws==2.16.1; (platform_machine == "arm64" or platform_machine == "aarch64") and platform_system == "Linux"(from tensorflow 2.16.1) pdm.termui: Candidate rejected: tensorflow@2.16.1 because it introduces a new requirement tensorflow-cpu-aws==2.16.1; (platform_machine == "arm64" or platform_machine == "aarch64") and platform_system == "Linux" that conflicts with other requirements: pdm.termui: Adding requirement tensorflow==2.16.1; platform_system == "Darwin" and platform_machine == "arm64"(from tensorflow 2.16.1) pdm.termui: Adding requirement tensorflow-cpu-aws==2.16.1; (platform_machine == "arm64" or platform_machine == "aarch64") and platform_system == "Linux"(from tensorflow 2.16.1) pdm.termui: Candidate rejected: tensorflow@2.16.1 because it introduces a new requirement tensorflow-cpu-aws==2.16.1; (platform_machine == "arm64" or platform_machine == "aarch64") and platform_system == "Linux" that conflicts with other requirements: pdm.termui: Adding requirement tensorflow==2.16.1; platform_system == "Darwin" and platform_machine == "arm64"(from tensorflow 2.16.1) pdm.termui: Adding requirement tensorflow-cpu-aws==2.16.1; (platform_machine == "arm64" or platform_machine == "aarch64") and platform_system == "Linux"(from tensorflow 2.16.1) pdm.termui: Candidate rejected: tensorflow@2.16.1 because it introduces a new requirement tensorflow-cpu-aws==2.16.1; (platform_machine == "arm64" or platform_machine == "aarch64") and platform_system == "Linux" that conflicts with other requirements: pdm.termui: Adding requirement tensorflow==2.16.1; platform_system == "Darwin" and platform_machine == "arm64"(from tensorflow 2.16.1) pdm.termui: Adding requirement tensorflow-cpu-aws==2.16.1; (platform_machine == "arm64" or platform_machine == "aarch64") and platform_system == "Linux"(from tensorflow 2.16.1) pdm.termui: Candidate rejected: tensorflow@2.16.1 because it introduces a new requirement tensorflow-cpu-aws==2.16.1; (platform_machine == "arm64" or platform_machine == "aarch64") and platform_system == "Linux" that conflicts with other requirements: pdm.termui: Adding requirement tensorflow==2.16.1; platform_system == "Darwin" and platform_machine == "arm64"(from tensorflow 2.16.1) pdm.termui: Adding requirement tensorflow-cpu-aws==2.16.1; (platform_machine == "arm64" or platform_machine == "aarch64") and platform_system == "Linux"(from tensorflow 2.16.1) pdm.termui: Candidate rejected: tensorflow@2.16.1 because it introduces a new requirement tensorflow-cpu-aws==2.16.1; (platform_machine == "arm64" or platform_machine == "aarch64") and platform_system == "Linux" that conflicts with other requirements: pdm.termui: Adding requirement tensorflow==2.16.1; platform_system == "Darwin" and platform_machine == "arm64"(from tensorflow 2.16.1) pdm.termui: Adding requirement tensorflow-cpu-aws==2.16.1; (platform_machine == "arm64" or platform_machine == "aarch64") and platform_system == "Linux"(from tensorflow 2.16.1) pdm.termui: Candidate rejected: tensorflow@2.16.1 because it introduces a new requirement tensorflow-cpu-aws==2.16.1; (platform_machine == "arm64" or platform_machine == "aarch64") and platform_system == "Linux" that conflicts with other requirements: pdm.termui: Conflicts detected: tensorflow-cpu-aws==2.16.1; (platform_machine == "arm64" or platform_machine == "aarch64") and platform_system == "Linux" (from tensorflow@2.16.1) ```

verbose traceback

``` Traceback (most recent call last): File "", line 198, in _run_module_as_main File "", line 88, in _run_code File "$USERPROFILE\.local\bin\pdm.exe\__main__.py", line 7, in File "D:\UserFiles\pipx\venvs\pdm\Lib\site-packages\pdm\core.py", line 358, in main return core.main(args or sys.argv[1:]) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\UserFiles\pipx\venvs\pdm\Lib\site-packages\pdm\core.py", line 276, in main raise cast(Exception, err).with_traceback(traceback) from None File "D:\UserFiles\pipx\venvs\pdm\Lib\site-packages\pdm\core.py", line 271, in main self.handle(project, options) File "D:\UserFiles\pipx\venvs\pdm\Lib\site-packages\pdm\core.py", line 207, in handle command.handle(project, options) File "D:\...\.pdm-plugins\Lib\site-packages\pdm_conda\cli\commands\lock.py", line 31, in handle super().handle(project=project, options=options) File "D:\UserFiles\pipx\venvs\pdm\Lib\site-packages\pdm\cli\commands\lock.py", line 92, in handle actions.do_lock( File "D:\UserFiles\pipx\venvs\pdm\Lib\site-packages\pdm\cli\actions.py", line 125, in do_lock raise ResolutionImpossible("Unable to find a resolution") from None resolvelib.resolvers.ResolutionImpossible: Unable to find a resolution ```

Expected behavior

if pip is able to install tensorflow>=2.16 then why can't pdm fetch that?
pip does so with wheels: tensorflow_intel-2.16.1-cp310-cp310-win_amd64.whl (376.9 MB)
... which corresponds to package tensorflow-intel but with automatic resolution from package tensorflow
this automatic resolution can be seen from output of pip install below: tensorflow-intel->tensorflow

> pip install tensorflow
Collecting tensorflow
  Downloading tensorflow-2.16.1-cp310-cp310-win_amd64.whl.metadata (3.5 kB)
Collecting tensorflow-intel==2.16.1 (from tensorflow)
  Downloading tensorflow_intel-2.16.1-cp310-cp310-win_amd64.whl.metadata (5.0 kB)
Collecting absl-py>=1.0.0 (from tensorflow-intel==2.16.1->tensorflow)
  ...
Collecting mdurl~=0.1 (from markdown-it-py>=2.2.0->rich->keras>=3.0.0->tensorflow-intel==2.16.1->tensorflow)
  Using cached mdurl-0.1.2-py3-none-any.whl.metadata (1.6 kB)
Downloading tensorflow-2.16.1-cp310-cp310-win_amd64.whl (2.1 kB)
Downloading tensorflow_intel-2.16.1-cp310-cp310-win_amd64.whl (376.9 MB)
  ...
Using cached mdurl-0.1.2-py3-none-any.whl (10.0 kB)
Installing collected packages: namex, libclang, flatbuffers, wrapt, typing-extensions, termcolor, tensorflow-io-gcs-filesystem, tensorboard-data-server, six, pygments, protobuf, numpy, mdurl, MarkupSafe, markdown, grpcio, gast, absl-py, werkzeug, optree, opt-einsum, ml-dtypes, markdown-it-py, h5py, google-pasta, astunparse, tensorboard, rich, keras, tensorflow-intel, tensorflow
Successfully installed MarkupSafe-2.1.5 absl-py-2.1.0 astunparse-1.6.3 flatbuffers-24.3.25 gast-0.5.4 google-pasta-0.2.0 grpcio-1.63.0 h5py-3.11.0 keras-3.3.3 libclang-18.1.1 markdown-3.6 markdown-it-py-3.0.0 mdurl-0.1.2 ml-dtypes-0.3.2 namex-0.0.8 numpy-1.26.4 opt-einsum-3.3.0 optree-0.11.0 protobuf-4.25.3 pygments-2.18.0 rich-13.7.1 six-1.16.0 tensorboard-2.16.2 tensorboard-data-server-0.7.2 tensorflow-2.16.1 tensorflow-intel-2.16.1 tensorflow-io-gcs-filesystem-0.31.0 termcolor-2.4.0 typing-extensions-4.11.0 werkzeug-3.0.2 wrapt-1.16.0

Environment Information

> which pip
/d/UserFiles/scoop/apps/mambaforge/current/scripts/pip

> pip --version
pip 24.0 from D:\UserFiles\scoop\apps\mambaforge\current\lib\site-packages\pip (python 3.10)

> # Paste the output of `pdm info && pdm info --env` below:
(project_1-3.12)
> pdm info
PDM version:
  2.15.1
Python Interpreter:
  D:\...\.venv\Scripts\python.exe (3.12)
Project Root:
  D:/...
Local Packages:

(project_1-3.12)
> pdm info --env
{
  "implementation_name": "cpython",
  "implementation_version": "3.12.2",
  "os_name": "nt",
  "platform_machine": "AMD64",
  "platform_release": "10",
  "platform_system": "Windows",
  "platform_version": "10.0.19045",
  "python_full_version": "3.12.2",
  "platform_python_implementation": "CPython",
  "python_version": "3.12",
  "sys_platform": "win32"
}

frostming commented 3 weeks ago

if pip is able to install tensorflow>=2.16 then why can't pdm fetch that?

pip is resolving for the current platform but PDM is resolving for multiple platforms. Read more: https://frostming.com/en/2024/pdm-lockfile/

You can do pdm lock -S no_cross_platform to replicate pip's behavior.

goyalyashpal commented 3 weeks ago

thanks for "closing as not planned" 😃

frostming commented 3 weeks ago

thanks for "closing as not planned" 😃

you are welcome. or what should I do otherwise?

goyalyashpal commented 3 weeks ago

i have seen many projects arrogantly close even non planned issues as completed.

so i always thanks those who dont do such weird things.

pawamoy commented 3 weeks ago

I am myself often unsure if I should close an issue as "completed" or "not planned" :thinking: Sometimes won't fix, can't repro, duplicate, or stale do not really apply :thinking: What's your reasoning when choosing not planned or completed @goyalyashpal :slightly_smiling_face:? Curious to see how others handle that.

goyalyashpal commented 3 weeks ago

if a proper solution exists, and author agrees (indicated by yes, this is what i needed etc) then "completed"

if workaround etc is required (like here), out of scope, out of plan, not enough resources (will, time, power, knowledge), technical or code debt, etc etc etc then "not planned"

goyalyashpal commented 3 weeks ago

but PDM is resolving for multiple platforms

just sharing for reference the following para of how micromamba addresses this:

Using conda-lock, you can generate lockfiles which, - ..., pin precisely and include a checksum for each package for reproducibility. - [... also] , these “unified” lock files are multi-platform.

- #conda-lock-yaml-spec-files | Micromamba User Guide

frostming commented 3 weeks ago

Not planned means no offense, if so, GitHub should remove this option.

I close as completed when there is any action taken(docs/code) on the maintainers' side, otherwise not planned, including Q&A

goyalyashpal commented 1 week ago

You can do pdm lock -S no_cross_platform to replicate pip's behavior. - @ frostming at https://github.com/pdm-project/pdm/issues/2867#issuecomment-2097979784

can that be specified in pyproject.toml ?

frostming commented 1 week ago

can that be specified in pyproject.toml ?

[tool.pdm.options]
lock = ["-S", "no_cross_platform"]

pdm-project / pdm