Open lambda opened 2 months ago
I've reproduce this a few times in ~1000 attempts. I have reproduced it on a couple of different versions of Python, ranging from 3.10 to 3.12, and both inside a container that derived from the official Python Docker container and on my host system, an Ubuntu 22.04 system running Python 3.12 from the deadsnakes ppa.
Reproduced by running the following in a project that has exhibited the issue:
trap 'exit_code=$?; echo "exit code: $exit_code"; echo; if [ $exit_code -ne 143 ]; then break; fi' ERR
for i in {1..500}; do (set -o pipefail; rm -rf .venv; python3.12 -m venv .venv; pdm use --ignore-remembered --first .venv/bin/python; pdm sync --no-editable -vv 2>&1 | (while read l; do echo "$l"; if (echo "$l" | grep -q "Install .* successful"); then pkill -s0 pdm; echo; break; fi; done)); done
This eventually produced:
Using Python interpreter: /home/brcampbell/tmp/<REDACTED>/.venv/bin/python (3.12)
STATUS: Resolving packages from lockfile...
Synchronizing working set with resolved packages: 122 to add, 0 to update, 0 to remove
unearth.collector: Collecting links from https://gitlab.<REDACTED>/api/v4/projects/<REDACTED>/packages/pypi/simple/aiobotocore/
unearth.auth: Found index url https://gitlab.<REDACTED>/api/v4/projects/<REDACTED>/packages/pypi/simple/
unearth.collector: Collecting links from https://gitlab.<REDACTED>/api/v4/projects/<REDACTED>/packages/pypi/simple/aiohttp/
unearth.auth: Found index url https://gitlab.<REDACTED>/api/v4/projects/<REDACTED>/packages/pypi/simple/
unearth.auth: Found credentials in netrc for gitlab.<REDACTED>
unearth.auth: Found credentials in netrc for gitlab.<REDACTED>
unearth.collector: Collecting links from https://gitlab.<REDACTED>/api/v4/projects/<REDACTED>/packages/pypi/simple/aiosignal/
unearth.collector: Collecting links from https://gitlab.<REDACTED>/api/v4/projects/<REDACTED>/packages/pypi/simple/aioitertools/
unearth.auth: Found index url https://gitlab.<REDACTED>/api/v4/projects/<REDACTED>/packages/pypi/simple/
unearth.collector: Collecting links from https://gitlab.<REDACTED>/api/v4/projects/<REDACTED>/packages/pypi/simple/annotated-types/
unearth.auth: Found index url https://gitlab.<REDACTED>/api/v4/projects/<REDACTED>/packages/pypi/simple/
unearth.collector: Collecting links from https://gitlab.<REDACTED>/api/v4/projects/<REDACTED>/packages/pypi/simple/argparse-addons/
unearth.collector: Collecting links from https://gitlab.<REDACTED>/api/v4/projects/<REDACTED>/packages/pypi/simple/attrs/
unearth.auth: Found credentials in netrc for gitlab.<REDACTED>
unearth.auth: Found index url https://gitlab.<REDACTED>/api/v4/projects/<REDACTED>/packages/pypi/simple/
unearth.collector: Collecting links from https://gitlab.<REDACTED>/api/v4/projects/<REDACTED>/packages/pypi/simple/babel/
unearth.collector: Collecting links from https://gitlab.<REDACTED>/api/v4/projects/<REDACTED>/packages/pypi/simple/<REDACTED>/
unearth.auth: Found index url https://gitlab.<REDACTED>/api/v4/projects/<REDACTED>/packages/pypi/simple/
unearth.collector: Collecting links from https://gitlab.<REDACTED>/api/v4/projects/<REDACTED>/packages/pypi/simple/<REDACTED>/
unearth.auth: Found index url https://gitlab.<REDACTED>/api/v4/projects/<REDACTED>/packages/pypi/simple/
unearth.collector: Collecting links from https://gitlab.<REDACTED>/api/v4/projects/<REDACTED>/packages/pypi/simple/bitstruct/
unearth.collector: Collecting links from https://gitlab.<REDACTED>/api/v4/projects/<REDACTED>/packages/pypi/simple/boto3/
unearth.collector: Collecting links from https://gitlab.<REDACTED>/api/v4/projects/<REDACTED>/packages/pypi/simple/botocore/
unearth.auth: Found credentials in netrc for gitlab.<REDACTED>
unearth.collector: Collecting links from https://gitlab.<REDACTED>/api/v4/projects/<REDACTED>/packages/pypi/simple/bracex/
unearth.auth: Found index url https://gitlab.<REDACTED>/api/v4/projects/<REDACTED>/packages/pypi/simple/
unearth.collector: Collecting links from https://gitlab.<REDACTED>/api/v4/projects/<REDACTED>/packages/pypi/simple/cachetools/
unearth.auth: Found index url https://gitlab.<REDACTED>/api/v4/projects/<REDACTED>/packages/pypi/simple/
unearth.collector: Collecting links from https://gitlab.<REDACTED>/api/v4/projects/<REDACTED>/packages/pypi/simple/cerberus/
unearth.auth: Found credentials in netrc for gitlab.<REDACTED>
unearth.auth: Found index url https://gitlab.<REDACTED>/api/v4/projects/<REDACTED>/packages/pypi/simple/
unearth.collector: Collecting links from https://gitlab.<REDACTED>/api/v4/projects/<REDACTED>/packages/pypi/simple/certifi/
unearth.collector: Collecting links from https://gitlab.<REDACTED>/api/v4/projects/<REDACTED>/packages/pypi/simple/charset-normalizer/
unearth.collector: Collecting links from https://gitlab.<REDACTED>/api/v4/projects/<REDACTED>/packages/pypi/simple/click/
unearth.auth: Found credentials in netrc for gitlab.<REDACTED>
unearth.collector: Collecting links from https://gitlab.<REDACTED>/api/v4/projects/<REDACTED>/packages/pypi/simple/colorama/
unearth.auth: Found index url https://gitlab.<REDACTED>/api/v4/projects/<REDACTED>/packages/pypi/simple/
unearth.auth: Found index url https://gitlab.<REDACTED>/api/v4/projects/<REDACTED>/packages/pypi/simple/
unearth.auth: Found index url https://gitlab.<REDACTED>/api/v4/projects/<REDACTED>/packages/pypi/simple/
unearth.auth: Found credentials in netrc for gitlab.<REDACTED>
unearth.auth: Found index url https://gitlab.<REDACTED>/api/v4/projects/<REDACTED>/packages/pypi/simple/
unearth.auth: Found index url https://gitlab.<REDACTED>/api/v4/projects/<REDACTED>/packages/pypi/simple/
unearth.auth: Found index url https://gitlab.<REDACTED>/api/v4/projects/<REDACTED>/packages/pypi/simple/
unearth.auth: Found credentials in netrc for gitlab.<REDACTED>
unearth.auth: Found credentials in netrc for gitlab.<REDACTED>
unearth.auth: Found index url https://gitlab.<REDACTED>/api/v4/projects/<REDACTED>/packages/pypi/simple/
unearth.auth: Found index url https://gitlab.<REDACTED>/api/v4/projects/<REDACTED>/packages/pypi/simple/
unearth.auth: Found credentials in netrc for gitlab.<REDACTED>
unearth.auth: Found index url https://gitlab.<REDACTED>/api/v4/projects/<REDACTED>/packages/pypi/simple/
unearth.auth: Found index url https://gitlab.<REDACTED>/api/v4/projects/<REDACTED>/packages/pypi/simple/
unearth.auth: Found credentials in netrc for gitlab.<REDACTED>
unearth.auth: Found credentials in netrc for gitlab.<REDACTED>
unearth.auth: Found credentials in netrc for gitlab.<REDACTED>
unearth.auth: Found credentials in netrc for gitlab.<REDACTED>
unearth.auth: Found credentials in netrc for gitlab.<REDACTED>
unearth.auth: Found credentials in netrc for gitlab.<REDACTED>
unearth.auth: Found credentials in netrc for gitlab.<REDACTED>
unearth.auth: Found credentials in netrc for gitlab.<REDACTED>
unearth.auth: Found credentials in netrc for gitlab.<REDACTED>
unearth.auth: Found credentials in netrc for gitlab.<REDACTED>
exit code: 139
The project is proprietary, and pointing at our internal pypi repository, hosted on a GitLab package registry. Working on minimizing the repro example to see if I can do it without those factors.
Great reproduction @lambda
We have experienced a very similar symtom.
pdm lock --check
) returns no errorspdm install
[project]
name = "redacted"
version = "redacted"
description = "redacted"
authors = [
{name = "redacted", email = "redacted"},
]
dependencies = [
"pydantic>=2.5.2",
"PyYAML~=6.0.1",
"requests>=2.31.0",
]
requires-python = "==3.12.*"
readme = "README.md"
license = {text = "redacted"}
[project.optional-dependencies]
dev = [
"pytest>=8.2.1",
"ruff>=0.2.2",
"pydantic>=2.5.2",
"pytest-cov>=5.0.0",
"requests-mock>=1.12.1",
"ipython>=8.25.0",
]
[tool.pdm]
distribution = false
I am attempting @lambda 's brute force attempt at local recreation and will update if I can get it.
Edit - no luck in reproduction. I'll leave it running over lunch.
Ah, nice to know that we're not the only ones hitting this.
Seeing this new comment prompted me to try again, and I just reproduced it again, this time with no reference to internal packages or our internal package registry.
Here is my most recent pyproject.toml
with which I was able to reproduce it (not really fully minimal, going to continue trying to cut it down but given how long it takes to repro that can take a while):
[build-system]
requires = ["pdm-pep517"]
build-backend = "pdm.pep517.api"
[tool.pdm.dev-dependencies]
dev = [
"flake8>=4.0.1",
"pep8-naming>=0.13.1",
"flake8-docstrings>=1.6.0",
"pytest>=7.1.2",
"pytest-cov>=3.0.0",
"pytest-xdist>=2.5.0",
"mypy>=0.971",
"types-requests>=2.28.2",
"types-mock>=4.0.15",
"isort>=5.10.1",
"black>=22.6.0",
"docformatter>=1.5.0, !=1.5.1",
]
docs = [
# base tool
"mkdocs>=1.3.1",
# theme
"mkdocs-material>=8.3.9",
# api reference
"mkdocstrings[python]>=0.19.0",
"mkdocs-gen-files>=0.3.5",
"mkdocs-literate-nav>=0.4.1",
"mkdocs-section-index>=0.3.4",
# include files
"mkdocs-include-markdown-plugin>=3.6.1",
# cli reference
"mkdocs-click>=0.8.0",
]
[project]
name = "pdm-memory-corruption"
requires-python = ">=3.8"
version = "1.0.0"
dependencies = [
"python-gitlab>=3.6.0",
"click>=8.1.3",
"more-itertools>=8.13.0",
"packaging>=21.3",
"markdown-it-py>=2.1.0",
"backoff>=2.1.2",
]
And here's the result; amazingly, I hit it on the first iteration of this test, but it frequently takes hundreds or thousands of runs before I hit it.
brcampbell@BETA-0661:~/tmp/pdm-memory-corruption$ for i in {1..500}; do (set -o pipefail; rm -rf .venv; python3.12 -m venv .venv; pdm use --ignore-remembered --first .venv/bin/python; pdm sync --no-editable -vv 2>&1 | (while read l; do echo "$l"; if (echo "$l" | grep -q "Install .* successful"); then pkill -s0 pdm; echo; break; fi; done)); done
Using Python interpreter: /home/brcampbell/tmp/pdm-memory-corruption/.venv/bin/python (3.12)
INFO: PDM 2.15.4 is installed, while 2.17.1 is available.
Please run `pipx upgrade pdm` to upgrade.
Run `pdm config check_update false` to disable the check.
WARNING: Lockfile is generated on an older version of PDM
STATUS: Resolving packages from lockfile...
STATUS: Fetching hashes for resolved packages...
Synchronizing working set with resolved packages: 68 to add, 0 to update, 0 to remove
unearth.collector: Collecting links from https://pypi.org/simple/attrs/
unearth.auth: Found index url https://pypi.org/simple/
unearth.collector: Collecting links from https://pypi.org/simple/certifi/
unearth.auth: Found index url https://pypi.org/simple/
unearth.collector: Collecting links from https://pypi.org/simple/black/
unearth.auth: Found index url https://pypi.org/simple/
unearth.collector: Collecting links from https://pypi.org/simple/charset-normalizer/
unearth.auth: Found index url https://pypi.org/simple/
unearth.collector: Collecting links from https://pypi.org/simple/click/
unearth.auth: Found index url https://pypi.org/simple/
unearth.collector: Collecting links from https://pypi.org/simple/backoff/
unearth.auth: Found index url https://pypi.org/simple/
unearth.collector: Collecting links from https://pypi.org/simple/colorama/
unearth.auth: Found index url https://pypi.org/simple/
unearth.collector: Collecting links from https://pypi.org/simple/coverage/
unearth.auth: Found index url https://pypi.org/simple/
unearth.collector: Collecting links from https://pypi.org/simple/docformatter/
unearth.auth: Found index url https://pypi.org/simple/
unearth.collector: Collecting links from https://pypi.org/simple/execnet/
unearth.auth: Found index url https://pypi.org/simple/
unearth.collector: Collecting links from https://pypi.org/simple/flake8/
unearth.auth: Found index url https://pypi.org/simple/
unearth.collector: Collecting links from https://pypi.org/simple/flake8-docstrings/
unearth.collector: Collecting links from https://pypi.org/simple/ghp-import/
unearth.auth: Found index url https://pypi.org/simple/
unearth.auth: Found index url https://pypi.org/simple/
unearth.collector: Collecting links from https://pypi.org/simple/griffe/
unearth.collector: Collecting links from https://pypi.org/simple/idna/
unearth.auth: Found index url https://pypi.org/simple/
unearth.collector: Collecting links from https://pypi.org/simple/iniconfig/
unearth.collector: Collecting links from https://pypi.org/simple/isort/
unearth.collector: Collecting links from https://pypi.org/simple/jinja2/
unearth.auth: Found index url https://pypi.org/simple/
unearth.collector: Collecting links from https://pypi.org/simple/markdown/
unearth.collector: Collecting links from https://pypi.org/simple/markdown-it-py/
unearth.auth: Found index url https://pypi.org/simple/
unearth.auth: Found index url https://pypi.org/simple/
unearth.auth: Found index url https://pypi.org/simple/
unearth.auth: Found index url https://pypi.org/simple/
unearth.auth: Found index url https://pypi.org/simple/
exit code: 139
Aye, I had no luck on reproduction, but that's a good job @lambda .
So establishing
Synchronizing working set
Being such a low-level error and PDM apparently being pure-python - somehow it's triggering something quite low down (seemingly in python itself?). That takes it a bit out of my skillset to debug more deeply.
@lambda I cannot even install your project because it fails with pyyaml :/ Apparently there is a dependency issue between pyyaml and cython. I will try python 3.10 instead of 3.12 which has wheels for pyyaml…
$ pdm sync --no-editable WARNING: Lockfile is generated on an older version of PDM
Synchronizing working set with resolved packages: 1 to add, 0 to update, 0 to remove
✖ Install pyyaml 6.0 failed
✖ Install pyyaml 6.0 failed
ERRORS:
add pyyaml failed:
Traceback (most recent call last):
File "/usr/lib64/python3.12/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/florian/.local/share/pipx/venvs/pdm/lib64/python3.12/site-packages/pdm/installers/synchronizers.py", line 228, in install_candidate
self.manager.install(can)
File "/home/florian/.local/share/pipx/venvs/pdm/lib64/python3.12/site-packages/pdm/installers/manager.py", line 33, in install
prepared.build(),
^^^^^^^^^^^^^^^^
File "/home/florian/.local/share/pipx/venvs/pdm/lib64/python3.12/site-packages/pdm/models/candidates.py", line 411, in build
self._cached = Path(builder.build(build_dir, metadata_directory=self._metadata_dir))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/florian/.local/share/pipx/venvs/pdm/lib64/python3.12/site-packages/pdm/builders/base.py", line 84, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/florian/.local/share/pipx/venvs/pdm/lib64/python3.12/site-packages/pdm/builders/wheel.py", line 24, in build
requires = self._hook.get_requires_for_build_wheel(self.config_settings)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/florian/.local/share/pipx/venvs/pdm/lib64/python3.12/site-packages/pyproject_hooks/_impl.py", line 196, in get_requires_for_build_wheel
return self._call_hook(
^^^^^^^^^^^^^^^^
File "/home/florian/.local/share/pipx/venvs/pdm/lib64/python3.12/site-packages/pyproject_hooks/_impl.py", line 392, in _call_hook
self._subprocess_runner(
File "/home/florian/.local/share/pipx/venvs/pdm/lib64/python3.12/site-packages/pdm/builders/base.py", line 281, in subprocess_runner
return log_subprocessor(cmd, cwd, extra_environ=env)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/florian/.local/share/pipx/venvs/pdm/lib64/python3.12/site-packages/pdm/builders/base.py", line 128, in log_subprocessor
raise build_error(e) from None
pdm.exceptions.BuildError: Build backend raised error: Showing the last 10 lines of the build output:
super().add_defaults()
File "/tmp/pdm-build-env-i9wa_tsb-shared/lib/python3.12/site-packages/setuptools/_distutils/command/sdist.py", line 250, in add_defaults
self._add_defaults_ext()
File "/tmp/pdm-build-env-i9wa_tsb-shared/lib/python3.12/site-packages/setuptools/_distutils/command/sdist.py", line 335, in _add_defaults_ext
self.filelist.extend(build_ext.get_source_files())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "<string>", line 204, in get_source_files
File "/tmp/pdm-build-env-i9wa_tsb-shared/lib/python3.12/site-packages/setuptools/_distutils/cmd.py", line 107, in __getattr__
raise AttributeError(attr)
AttributeError: cython_sources
So here are some further thoughts while I try to reproduce:
Further things that would be helpful:
pdm sync -r
would reinstall the files, does this work as well, would probably make reproduction also fasterIf the segfault is still triggered afterwards we are already in a better position… Although if it is a memory corruption the backtrace is most likely useless. But maybe it is always at the same location and gives some hints?
@lambda @mikecowie-seequent, our team is experiencing this issue as well. Were you guys able to figure out the cause?
@FaitAccompli , no sadly, it's something we've been tolerating as contributing to CI flakiness . It's rare enough to not be a priority...
I might well be imagining it, but it feels like I might have seen it less in the past few weeks?
Make sure you run commands with
-v
flag before pasting the output.Steps to reproduce
Twice in the past 24 hours, I have seen PDM exhibit a memory corruption related crash in my CI jobs.
I do not yet know how to reproduce this issue reliably. Re-running the same job a second time can work successfully; this problem is not deterministic. Filing it in case there might be something suspicious that you can think of, and to provide a place to keep track of further instances to collect more information.
In a Python 3.11 CI job yesterday:
In a Python 3.12 CI job today (for a different project):
Actual behavior
PDM sometimes dumps core due to memory corruption issues.
Expected behavior
PDM doesn't crash and installs the files.
Environment Information
Here is how PDM is installed and set up in these CI jobs:
``` $ PIP_INDEX_URL= pip install --upgrade pip setuptools wheel Requirement already satisfied: pip in /usr/local/lib/python3.11/site-packages (23.2.1) Collecting pip Obtaining dependency information for pip from https://files.pythonhosted.org/packages/8a/6a/19e9fe04fca059ccf770861c7d5721ab4c2aebc539889e97c7977528a53b/pip-24.0-py3-none-any.whl.metadata Downloading pip-24.0-py3-none-any.whl.metadata (3.6 kB) Requirement already satisfied: setuptools in /usr/local/lib/python3.11/site-packages (65.5.1) Collecting setuptools Obtaining dependency information for setuptools from https://files.pythonhosted.org/packages/de/88/70c5767a0e43eb4451c2200f07d042a4bcd7639276003a9c54a68cfcc1f8/setuptools-70.0.0-py3-none-any.whl.metadata Downloading setuptools-70.0.0-py3-none-any.whl.metadata (5.9 kB) Requirement already satisfied: wheel in /usr/local/lib/python3.11/site-packages (0.42.0) Collecting wheel Obtaining dependency information for wheel from https://files.pythonhosted.org/packages/7d/cd/d7460c9a869b16c3dd4e1e403cce337df165368c71d6af229a74699622ce/wheel-0.43.0-py3-none-any.whl.metadata Downloading wheel-0.43.0-py3-none-any.whl.metadata (2.2 kB) Downloading pip-24.0-py3-none-any.whl (2.1 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.1/2.1 MB 31.0 MB/s eta 0:00:00 Downloading setuptools-70.0.0-py3-none-any.whl (863 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 863.4/863.4 kB 31.2 MB/s eta 0:00:00 Downloading wheel-0.43.0-py3-none-any.whl (65 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 65.8/65.8 kB 5.8 MB/s eta 0:00:00 Installing collected packages: wheel, setuptools, pip Attempting uninstall: wheel Found existing installation: wheel 0.42.0 Uninstalling wheel-0.42.0: Successfully uninstalled wheel-0.42.0 Attempting uninstall: setuptools Found existing installation: setuptools 65.5.1 Uninstalling setuptools-65.5.1: Successfully uninstalled setuptools-65.5.1 Attempting uninstall: pip Found existing installation: pip 23.2.1 Uninstalling pip-23.2.1: Successfully uninstalled pip-23.2.1 Successfully installed pip-24.0 setuptools-70.0.0 wheel-0.43.0 WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv $ if [ -f pdm.lock ]; then pip install pdm~=$PDM_VERSION; fi Looking in indexes: https://gitlab-ci-token:****@gitlab.