search

pdm-project / pdm

A modern Python package and dependency manager supporting the latest PEP standards
https://pdm-project.org
MIT License
7.97k stars 409 forks source link

`pyproject.toml` adapter for `uv` writes to the filesystem within the project root, breaks down in read-only situations #3292

Open sanmai-NL opened 1 week ago

sanmai-NL commented 1 week ago

Describe the bug

To support uv (indeed, a feature not yet production ready in the current implementation), the pyproject.toml file gets moved away and the contents at its path mutate. After uv completes, pdm restores the original pyproject.toml.

This doesn't work when the filesystem is read-only (in some place). This design also creates some noise in IDEs that watch the filesystem and thing the VCS working copy is undergoing changes.

I haven't studied the current implementation in detail, but creating a temporary directory and/or files using e.g., the tempfile module is more likely to get past this filesystem constraint.

To reproduce

024-11-14T21:50:48.973170+0100 - DEBUG - aimtools.cli - execute - [stdout] + pdm install --fail-fast --frozen-lockfile --no-editable --no-isolation --production --verbose

2024-11-14T21:50:48.973527+0100 - DEBUG - aimtools.cli - execute - [stdout] STATUS: Resolving packages from lockfile...

2024-11-14T21:50:48.973912+0100 - DEBUG - aimtools.cli - execute - [stdout] WARNING: inherit_metadata strategy is not supported by uv resolver, it will be ignored

2024-11-14T21:50:48.974240+0100 - DEBUG - aimtools.cli - execute - [stdout] Traceback (most recent call last):

2024-11-14T21:50:48.974540+0100 - DEBUG - aimtools.cli - execute - [stdout]   File "/home/unprivileged/.local/bin/pdm", line 8, in <module>

2024-11-14T21:50:48.974899+0100 - DEBUG - aimtools.cli - execute - [stdout]     sys.exit(main())

2024-11-14T21:50:48.975196+0100 - DEBUG - aimtools.cli - execute - [stdout]              ^^^^^^

2024-11-14T21:50:48.975485+0100 - DEBUG - aimtools.cli - execute - [stdout]   File "/home/unprivileged/.local/share/pdm/venv/lib/python3.12/site-packages/pdm/core.py", line 379, in main

2024-11-14T21:50:48.975756+0100 - DEBUG - aimtools.cli - execute - [stdout]     return core.main(args or sys.argv[1:])

2024-11-14T21:50:48.976070+0100 - DEBUG - aimtools.cli - execute - [stdout]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

2024-11-14T21:50:48.976367+0100 - DEBUG - aimtools.cli - execute - [stdout]   File "/home/unprivileged/.local/share/pdm/venv/lib/python3.12/site-packages/pdm/core.py", line 267, in main

2024-11-14T21:50:48.976664+0100 - DEBUG - aimtools.cli - execute - [stdout]     raise cast(Exception, err).with_traceback(traceback) from None

2024-11-14T21:50:48.976963+0100 - DEBUG - aimtools.cli - execute - [stdout]   File "/home/unprivileged/.local/share/pdm/venv/lib/python3.12/site-packages/pdm/core.py", line 262, in main

2024-11-14T21:50:48.977263+0100 - DEBUG - aimtools.cli - execute - [stdout]     self.handle(project, options)

2024-11-14T21:50:48.977591+0100 - DEBUG - aimtools.cli - execute - [stdout]   File "/home/unprivileged/.local/share/pdm/venv/lib/python3.12/site-packages/pdm/core.py", line 194, in handle

2024-11-14T21:50:48.977969+0100 - DEBUG - aimtools.cli - execute - [stdout]     command.handle(project, options)

2024-11-14T21:50:48.978298+0100 - DEBUG - aimtools.cli - execute - [stdout]   File "/home/unprivileged/.local/share/pdm/venv/lib/python3.12/site-packages/pdm/cli/commands/install.py", line 103, in handle

2024-11-14T21:50:48.978603+0100 - DEBUG - aimtools.cli - execute - [stdout]     actions.do_sync(

2024-11-14T21:50:48.978947+0100 - DEBUG - aimtools.cli - execute - [stdout]   File "/home/unprivileged/.local/share/pdm/venv/lib/python3.12/site-packages/pdm/cli/actions.py", line 284, in do_sync

2024-11-14T21:50:48.979219+0100 - DEBUG - aimtools.cli - execute - [stdout]     packages = list(resolve_from_lockfile(project, requirements, groups=list(selection)))

2024-11-14T21:50:48.979505+0100 - DEBUG - aimtools.cli - execute - [stdout]                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

2024-11-14T21:50:48.979794+0100 - DEBUG - aimtools.cli - execute - [stdout]   File "/home/unprivileged/.local/share/pdm/venv/lib/python3.12/site-packages/pdm/cli/actions.py", line 218, in resolve_from_lockfile

2024-11-14T21:50:48.980090+0100 - DEBUG - aimtools.cli - execute - [stdout]     return resolver.resolve().packages

2024-11-14T21:50:48.980446+0100 - DEBUG - aimtools.cli - execute - [stdout]            ^^^^^^^^^^^^^^^^^^

2024-11-14T21:50:48.980818+0100 - DEBUG - aimtools.cli - execute - [stdout]   File "/home/unprivileged/.local/share/pdm/venv/lib/python3.12/site-packages/pdm/resolver/uv.py", line 173, in resolve

2024-11-14T21:50:48.981154+0100 - DEBUG - aimtools.cli - execute - [stdout]     builder.build_pyproject_toml()

2024-11-14T21:50:48.981447+0100 - DEBUG - aimtools.cli - execute - [stdout]   File "/home/unprivileged/.local/share/pdm/venv/lib/python3.12/site-packages/pdm/formats/uv.py", line 64, in build_pyproject_toml

2024-11-14T21:50:48.981775+0100 - DEBUG - aimtools.cli - execute - [stdout]     path = self._enter_path(self.project.root / "pyproject.toml")

2024-11-14T21:50:48.982066+0100 - DEBUG - aimtools.cli - execute - [stdout]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

2024-11-14T21:50:48.982373+0100 - DEBUG - aimtools.cli - execute - [stdout]   File "/home/unprivileged/.local/share/pdm/venv/lib/python3.12/site-packages/pdm/formats/uv.py", line 72, in _enter_path

2024-11-14T21:50:48.982694+0100 - DEBUG - aimtools.cli - execute - [stdout]     backup = path.rename(name)

2024-11-14T21:50:48.983082+0100 - DEBUG - aimtools.cli - execute - [stdout]              ^^^^^^^^^^^^^^^^^

2024-11-14T21:50:48.983370+0100 - DEBUG - aimtools.cli - execute - [stdout]   File "/usr/lib/python3.12/pathlib.py", line 1363, in rename

2024-11-14T21:50:48.983645+0100 - DEBUG - aimtools.cli - execute - [stdout]     os.rename(self, target)

2024-11-14T21:50:48.983981+0100 - DEBUG - aimtools.cli - execute - [stdout] OSError: [Errno 16] Resource busy: '/opt/app/pyproject.toml' -> '/opt/app/pyproject.toml.w1buacou'

Expected Behavior

Don't assume renames etc. are possible and avoid filesystem mutations as much as possible.

Environment Information

2024-11-14T21:50:48.867099+0100 - DEBUG - aimtools.cli - execute - [stdout] + pdm info

2024-11-14T21:50:48.867460+0100 - DEBUG - aimtools.cli - execute - [stdout] PDM version:

2024-11-14T21:50:48.867931+0100 - DEBUG - aimtools.cli - execute - [stdout]   2.20.1

2024-11-14T21:50:48.868263+0100 - DEBUG - aimtools.cli - execute - [stdout] Python Interpreter:

2024-11-14T21:50:48.868593+0100 - DEBUG - aimtools.cli - execute - [stdout]   /opt/app/.venv/bin/python (3.12)

2024-11-14T21:50:48.868933+0100 - DEBUG - aimtools.cli - execute - [stdout] Project Root:

2024-11-14T21:50:48.869294+0100 - DEBUG - aimtools.cli - execute - [stdout]   /opt/app

2024-11-14T21:50:48.869603+0100 - DEBUG - aimtools.cli - execute - [stdout] Local Packages:

2024-11-14T21:50:48.869939+0100 - DEBUG - aimtools.cli - execute - [stdout]   

2024-11-14T21:50:48.870314+0100 - DEBUG - aimtools.cli - execute - [stdout] + pdm info --env

2024-11-14T21:50:48.870597+0100 - DEBUG - aimtools.cli - execute - [stdout] {

2024-11-14T21:50:48.870940+0100 - DEBUG - aimtools.cli - execute - [stdout]   "implementation_name": "cpython",

2024-11-14T21:50:48.871360+0100 - DEBUG - aimtools.cli - execute - [stdout]   "implementation_version": "3.12.7",

2024-11-14T21:50:48.871707+0100 - DEBUG - aimtools.cli - execute - [stdout]   "os_name": "posix",

2024-11-14T21:50:48.872006+0100 - DEBUG - aimtools.cli - execute - [stdout]   "platform_machine": "aarch64",

2024-11-14T21:50:48.872331+0100 - DEBUG - aimtools.cli - execute - [stdout]   "platform_release": "6.11.3-200.fc40.aarch64",

2024-11-14T21:50:48.872724+0100 - DEBUG - aimtools.cli - execute - [stdout]   "platform_system": "Linux",

2024-11-14T21:50:48.873115+0100 - DEBUG - aimtools.cli - execute - [stdout]   "platform_version": "#1 SMP PREEMPT_DYNAMIC Thu Oct 10 22:53:48 UTC 2024",

2024-11-14T21:50:48.873409+0100 - DEBUG - aimtools.cli - execute - [stdout]   "python_full_version": "3.12.7",

2024-11-14T21:50:48.873669+0100 - DEBUG - aimtools.cli - execute - [stdout]   "platform_python_implementation": "CPython",

2024-11-14T21:50:48.873960+0100 - DEBUG - aimtools.cli - execute - [stdout]   "python_version": "3.12",

2024-11-14T21:50:48.874211+0100 - DEBUG - aimtools.cli - execute - [stdout]   "sys_platform": "linux"

2024-11-14T21:50:48.874470+0100 - DEBUG - aimtools.cli - execute - [stdout] }

pdm -v output

No response

Additional Context

When pyproject.toml is a read-only bind mount, it's simply impossible to rename as far as I know.

Are you willing to submit a PR to fix this bug?

frostming commented 1 week ago

Usually, building or installing the current project needs to be done in place, otherwise, you would need to copy the entire project to a temporary directory, causing unnecessary overhead. Moreover, if your project references packages from the parent directory as dependencies, copying becomes infeasible either.

sanmai-NL commented 1 week ago

I think if we can define more strictly what mutations/write transactions are needed where, we can in practice work around this without significant disadvantages. In my case, only pyproject.toml, pdm.toml and pdm.lock are bind mounts and the rest of the filesystem tree isn't, avoiding some of the essential limitations you point out.

Arguments in favor of such a more refined design, are that read-only filesystems or parts thereof (such as some mount points) are common in Linux-containerized scenarios. These environments allow for more performance-efficient and hermetic builds. See source identity on https://bazel.build/basics/hermeticity.