PDM doesn't benefit significantly from its cache

[deronnax]

• [x] I have searched the issue tracker and believe that this is not a duplicate.

Actual behavior

We did several tries to cache PDM cache on our Gitlab CI and we were surprised of how little it improved pdm sync performances. Until PDM 2.13, PDM was taking 1m57 to install our long list of dependencies, and 1m43 with a filled cache. Cache restauration time on our Gitlab is ~10 secondes so it was not worth it. With PDM 2.13+, timing moved to 6m20 for an installation with empty cache to 4m35, so yes caching became finally worth it but you see the problem.

Expected behavior

I don't know if this can be fixed/improved. If you think you it can, that's awesome. If you think it's all normal, I'll close the issue.

Steps to reproduce

I did a small demonstration repo, pdm-cache-demonstrator , with a very simple reproducible case and long dependencies list, similar to the our production dependencies. There is a pdm cache info command before each pdm sync, and pdm sync is run with -v. We get somewhat similar results that what we experience, but less exacerbated: pdm 2.12.4, empty cache: 1m38 pdm 2.12.4, filled cache: 0m57 pdm 2.13.3, empty cache: 2m37 pdm 2.13.3, filled cache: 2m9

Environment Information

For PDM 2.12:

root@02d0a41dda1d:/app/backend# pdm info && pdm info --env
PDM version:
  2.12.4
Python Interpreter:
  /app/backend/.venv/bin/python (3.11)
Project Root:
  /app/backend
Local Packages:

INFO: PDM 2.12.4 is installed, while 2.13.3 is available.
Please run `pdm self update` to upgrade.
Run `pdm config check_update false` to disable the check.
{
  "implementation_name": "cpython",
  "implementation_version": "3.11.9",
  "os_name": "posix",
  "platform_machine": "x86_64",
  "platform_release": "6.7.11-orbstack-00143-ge6b82e26cd22",
  "platform_system": "Linux",
  "platform_version": "#1 SMP Sat Mar 30 12:20:36 UTC 2024",
  "python_full_version": "3.11.9",
  "platform_python_implementation": "CPython",
  "python_version": "3.11",
  "sys_platform": "linux"
}

It would be the same for PDM 2.13

[sr-murthy]

@deronnax Just out of curiosity, have your steps been evaluated against the latest PDM release 2.14.0?

[deronnax]

Regarding caching, PDM 2.14.0 just reverts what was done in PDM 2.13, so I expect cache performance to be strictly the same than PDM 2.12.

[polarathene]

Might be related to the explanation here: https://github.com/pdm-project/pdm/issues/2964#issuecomment-2184486093

In contrast to uv, we still consult the index(in this case, pytorch source), download the wheel(use the http cache if available) and link to the local package cache for installation.

while uv skips the http entirely as it assumes the local copy MUST be identical.

My reproduction is far simpler with just a few explicit torch deps but 5GB of cache that only reduces the install time by 25 sec, which similar to yours still required over a minute to install with that cache. While uv could restore from cache in a mere 4 seconds.

[nkhitrov]

I have similar issue with Gitlab CI cache. This method works for me with poetry

$ pdm use -f $PYTHON_VENV_DIR
INFO: Using the first matched interpreter.
Using Python interpreter: /builds/.../master/.venv/bin/python (3.11)
INFO: PDM 2.17.1 is installed, while 2.19.1 is available.
Please run `pdm self update` to upgrade.
Run `pdm config check_update false` to disable the check.
$ pdm sync --dev --fail-fast
STATUS: Resolving packages from lockfile...
All packages are synced to date, nothing to do.
Installing the project as an editable package...
  ✔ Update myproject 0.154.0 -> 0.154.0 successful
🎉 All complete!
INFO: PDM 2.17.1 is installed, while 2.19.1 is available.
Please run `pdm self update` to upgrade.
Run `pdm config check_update false` to disable the check.

------

$ mypy app tests
pyproject.toml:1: error: Error importing plugin "sqlalchemy.ext.mypy.plugin": No module named 'sqlalchemy'  [misc]