search

MagicStack / uvloop

Ultra fast asyncio event loop.
Apache License 2.0
10.45k stars 548 forks source link

[0.18.0] Running black causes core dump: src/unix/core.c:633: uv__close: Assertion `fd > 2' failed. #580

Closed khneal closed 7 months ago

khneal commented 1 year ago

While building a Docker image with podman in Jenkins, running black causes a core dump:

STEP 14/15: RUN venv/bin/black --verbose --check --diff . || true
Identified `/` as project root containing a file system root.
Found input source directory: "/approot"
/approot/venv ignored: matches the --exclude regular expression
python: src/unix/core.c:633: uv__close: Assertion `fd > 2' failed.
Aborted (core dumped)

The error seems to refer to: https://github.com/libuv/libuv/blob/v1.46.0/src/unix/core.c#L633

I was able to reproduce with python 3.8-slim, 3.11-slim, and 3.12-slim. I did not try others. The core dump does not happen when I switch back to 0.17.0, so we have a workaround for now. See attached Dockerfile.

fantix commented 1 year ago

This is probably happening in a situation not reflected in the provided Dockerfile because I cannot reproduce this with either Podman, Docker or a local run. The link to the libuv code (thanks!) is also a very generic fd close call used in many places; thus, I'm unable to locate the issue atm.

It looks like black is using asyncio (uvloop if present) for its concurrency, so it would be great if you could try debugging the crash (a call stack would be nice!) or even better - find a reproducible Python script.

khneal commented 1 year ago

Producing a stacktrace from the core dump is on my to-do list... thank you for your response, and for your patience with me following up.

khneal commented 1 year ago

Does this help?

STEP 16/16: RUN venv/bin/pystack core --native --locals core.*
Using executable found in the core file: venv/bin/python

Core file information:
state: R zombie: True niceness: 0
pid: 1436 ppid: 1435 sid: 7
uid: 0 gid: 0 pgrp: 7
executable: python arguments: venv/bin/python -m black --verbose --check --diff . 

The process died due receiving signal SIGABRT
Traceback for thread 1436 [Has the GIL] (most recent call last):
    (C) File "???", line 0, in _start (python)
    (C) File "???", line 0, in __libc_start_main (libc.so.6)
    (C) File "???", line 0, in Py_BytesMain (libpython3.8.so.1.0)
    (C) File "???", line 0, in Py_RunMain (libpython3.8.so.1.0)
    (Python) File "/usr/local/lib/python3.8/runpy.py", line 194, in _run_module_as_main
        return _run_code(code, main_globals, None,
      Arguments:
        alter_argv: 1
        mod_name: "black.__main__"
      Locals:
        main_globals: {"__name__": "__main__", "__doc__": None, "__package__": "black", ...}
        mod_spec: <ModuleSpec at 0x7f996f33a610>
        code: <code at 0x7f996f678be0>
    (Python) File "/usr/local/lib/python3.8/runpy.py", line 87, in _run_code
        exec(code, run_globals)
      Arguments:
        script_name: None
        pkg_name: "black"
        run_globals: {"__name__": "__main__", "__doc__": None, "__package__": "black", ...}
        code: <code at 0x7f996f678be0>
        init_globals: None
        mod_name: "__main__"
        mod_spec: <ModuleSpec at 0x7f996f33a610>
      Locals:
        cached: "/approot/venv/lib/python3.8/site-packages/black/__pycache__/__main__.cpython...
        loader: <SourceFileLoader at 0x7f99726ae730>
        fname: "/approot/venv/lib/python3.8/site-packages/black/__main__.py"
    (Python) File "/approot/venv/lib/python3.8/site-packages/black/__main__.py", line 3, in <module>
        patched_main()
    (C) File "???", line 0, in CPyPy_black___patched_main (2ec0e72aa72355e6eccf__mypyc.cpython-38-x86_64-linux-gnu.so)
    (C) File "???", line 0, in CPyDef_black___patched_main (2ec0e72aa72355e6eccf__mypyc.cpython-38-x86_64-linux-gnu.so)
    (Python) File "/approot/venv/lib/python3.8/site-packages/click/core.py", line 1157, in __call__
        return self.main(*args, **kwargs)
      Arguments:
        self: <Command at 0x7f996f33a6d0>
      Locals:
        kwargs: {}
        args: ()
    (Python) File "/approot/venv/lib/python3.8/site-packages/click/core.py", line 1078, in main
        rv = self.invoke(ctx)
      Arguments:
        complete_var: None
        windows_expand_args: True
        standalone_mode: True
        prog_name: "python -m black"
        self: <Command at 0x7f996f33a6d0>
        args: []
      Locals:
        ctx: <Context at 0x7f99725f8310>
        extra: {}
    (Python) File "/approot/venv/lib/python3.8/site-packages/click/core.py", line 1434, in invoke
        return ctx.invoke(self.callback, **ctx.params)
      Arguments:
        ctx: <Context at 0x7f99725f8310>
        self: <Command at 0x7f996f33a6d0>
    (Python) File "/approot/venv/lib/python3.8/site-packages/click/core.py", line 783, in invoke
        return __callback(*args, **kwargs)
      Arguments:
        _Context__callback: <function at 0x7f99725a9430>
        _Context__self: <Context at 0x7f99725f8310>
      Locals:
        ctx: <Context at 0x7f99725f8310>
        args: ()
        kwargs: {"src": ("."), "stdin_filename": None, "config": None, "verbose": True, ...}
    (Python) File "/approot/venv/lib/python3.8/site-packages/click/decorators.py", line 33, in new_func
        return f(get_current_context(), *args, **kwargs)
      Locals:
        kwargs: {"src": ("."), "stdin_filename": None, "config": None, "verbose": True, ...}
        args: ()
    (C) File "???", line 0, in CPyPy_black___main (2ec0e72aa72355e6eccf__mypyc.cpython-38-x86_64-linux-gnu.so)
    (C) File "???", line 0, in CPyDef_black___main (2ec0e72aa72355e6eccf__mypyc.cpython-38-x86_64-linux-gnu.so)
    (Python) File "/approot/venv/lib/python3.8/site-packages/black/concurrency.py", line 97, in reformat_many
        loop = asyncio.new_event_loop()
      Arguments:
        workers: 32
        report: <Report at 0x7f99726ae280>
        sources: <set at 0x7f996f591ac0>
        mode: < at 0x7f996f338ef0>
        write_back: <WriteBack at 0x7f996f6dd8b0>
        fast: False
      Locals:
        executor: <ProcessPoolExecutor at 0x7f996e4c3370>
    (Python) File "/usr/local/lib/python3.8/asyncio/events.py", line 758, in new_event_loop
        return get_event_loop_policy().new_event_loop()
    (Python) File "/usr/local/lib/python3.8/asyncio/events.py", line 656, in new_event_loop
        return self._loop_factory()
      Arguments:
        self: <EventLoopPolicy at 0x7f996f33a970>
    (Python) File "/approot/venv/lib/python3.8/site-packages/uvloop/__init__.py", line 137, in _loop_factory
        return new_event_loop()
      Arguments:
        self: <EventLoopPolicy at 0x7f996f33a970>
    (Python) File "/approot/venv/lib/python3.8/site-packages/uvloop/__init__.py", line 22, in new_event_loop
        return Loop()
    (C) File "uvloop/loop.c", line 152942, in __pyx_tp_new_6uvloop_4loop_Loop (loop.cpython-38-x86_64-linux-gnu.so)
    (C) File "uvloop/loop.c", line 10385, in __pyx_pw_6uvloop_4loop_4Loop_1__cinit__ (inlined) (loop.cpython-38-x86_64-linux-gnu.so)
    (C) File "uvloop/loop.c", line 10642, in __pyx_pf_6uvloop_4loop_4Loop___cinit__ (inlined) (loop.cpython-38-x86_64-linux-gnu.so)
    (C) File "src/unix/loop.c", line 80, in uv_loop_init (loop.cpython-38-x86_64-linux-gnu.so)
    (C) File "src/unix/linux.c", line 590, in uv__platform_loop_init (loop.cpython-38-x86_64-linux-gnu.so)
    (C) File "src/unix/linux.c", line 562, in uv__iou_init (loop.cpython-38-x86_64-linux-gnu.so)
    (C) File "???", line 0, in abort (libc.so.6)
    (C) File "???", line 0, in raise (libc.so.6)
nayanmanojgupta commented 8 months ago

Any update on this?

khneal commented 7 months ago

Closing... I'm now unable to reproduce the original error.