search

platformio / platformio-vscode-ide

PlatformIO IDE for VSCode: The next generation integrated development environment for IoT
https://marketplace.visualstudio.com/items?itemName=platformio.platformio-ide
Apache License 2.0
1.2k stars 192 forks source link

If process unexpectedly quits while debugging, Python will use 100% CPU until force quit #2891

Open theficus opened 2 years ago

theficus commented 2 years ago

When debugging a program in PlatformIO when using the integrated VSCode debugging experience, I've found that if the process unexpectedly quits (for example because of a seg fault), the underlying Python process gets stuck in a tight loop where it uses 100% CPU and doesn't quit until being force killed. This is not visible in VSCode's Process Explorer view.

This is the command line of the process from ps:

/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/Resources/Python.app/Contents/MacOS/Python /Users/XXX/.platformio/penv/bin/platformio debug -e native --interface gdb --interpreter=mi2 -q

I originally noticed this when after trying to debug a code issue I found that my laptop fan was going crazy and my laptop battery was draining very quickly.

This is very easy for me to reproduce.

Here's a simple bit of repro code:

int main(int argc, char** argv)
{
    char* c;
    *c = 'a';
    return 0;
}

When I debug into this code, once the process exits due to the seg fault, I can see a Python process using 100% CPU. (It may also be necessary to set a breakpoint before the segfault, but I'm not 100% certain.)

This does not reproduce when using the CLI debugger with pio, this only reproduces when using the VSCode integrated debugger.

I'm assuming that the gdb shim to VSCode doesn't gracefully handle the process unexpectedly quitting and gets stuck into a bad state when this happens.

This is what my platformio.ini looks like:

[env:native]
platform = native
test_ignore = test_embedded
build_flags = -std=gnu++11
debug_test = test_native
test_build_project_src = true
Environment details - MacOS 11.6.2 - VSCode: 1.63.2 - PlatformIO extension: 2.4.0 - PlatformIO Core: 5.2.4
theficus commented 2 years ago

Discussion on community forum: https://community.platformio.org/t/debugger-using-huge-amounts-of-cpu-and-battery-until-python-is-force-quit/25512

theficus commented 2 years ago

I wonder if this is at all related to #2683 and the extension using the Xcode version of Python vs the ~/.platformio version.

Note in my repro above its using /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/Resources/Python.app/Contents/MacOS/Python versus ~/.platformio/python3.

It wouldn't be the first time Apple's bundled versions of certain tools have different behaviors from mainstream versions.

In haven't had a chance to try to repro this issue on Windows to confirm this hunch, though.

theficus commented 2 years ago

Just a quick update -- I'm seeing this when performing both native debugging on my local machine as well as when doing JTAG debugging against a board. It seems to repro easily by simply stopping a debug session while it's in progress (doesn't repro 100% of the time but enough that I feel it necessary to check running processes every time I stop debugging to kill the runaway Python process if needed). This is making debugging a pretty miserable experience. :(

If there's any more information I can provide that may assist in getting this fixed, please let me know.

michikite commented 1 year ago

Same problem when debugging an esp32 with JTAG. I am using Python 3.10 using PyEnv. Happy to test, if someone has a hint on how to fix this.

britannio commented 4 months ago

I'm facing this issue too.

McKayRansom commented 2 months ago

We are running into this issue on the latest MacOS. I will update if we find any steps to reproduce.

m-mcgowan commented 3 weeks ago

I see this routinely. Yesterday I had dozens of python processes all pinned (at 50%..probably because there were so many?), to the point that my Mac claimed it was out of memory.

When I opened my laptop this morning, I found 4 python processes running at %100 cpu.

I used sudo killall python, at 9:56, which didn't do anything, so then tried sudo killall -9 python 3 times in succession.

Back in VS Code, I see a notification "The Black Formatter server crashed 5 times in the last 3 minutes. The server will not be restarted. See the output for more information."

Logs are below.

This could be completely unrelated to this github issue, and in hindsight I should have at least looked at all the python processes in the system in more detail, rather than using a blunt tool to kill them all. I will report in more detail when this happens again.

4-08-23 00:08:28.431 [info] Server run command: /Users/mat/.pyenv/versions/3.8.16/bin/python /Users/mat/.vscode/extensions/ms-python.black-formatter-2024.2.0/bundled/tool/lsp_server.py
2024-08-23 00:08:28.431 [info] Server: Start requested.
2024-08-23 00:08:30.991 [info] CWD Server: /Users/mat/e/firmware/iot_thing
2024-08-23 00:08:31.130 [info] Settings received on server:
[
    {
        "cwd": "/Users/mat/e/firmware/iot_thing",
        "workspace": "file:///Users/mat/e/firmware/iot_thing",
        "args": [],
        "path": [],
        "interpreter": [
            "/Users/mat/.pyenv/versions/3.8.16/bin/python"
        ],
        "importStrategy": "useBundled",
        "showNotifications": "off"
    }
]

2024-08-23 00:08:31.314 [info] Global settings received on server:
{
    "cwd": "/",
    "workspace": "/",
    "args": [],
    "path": [],
    "interpreter": [],
    "importStrategy": "useBundled",
    "showNotifications": "off"
}

2024-08-23 00:08:31.346 [info] sys.path used to run Server:
   /Users/mat/.vscode/extensions/ms-python.black-formatter-2024.2.0/bundled/libs
   /Users/mat/.vscode/extensions/ms-python.black-formatter-2024.2.0/bundled/tool
   /Users/mat/.pyenv/versions/3.8.16/lib/python38.zip
   /Users/mat/.pyenv/versions/3.8.16/lib/python3.8
   /Users/mat/.pyenv/versions/3.8.16/lib/python3.8/lib-dynload
   /Users/mat/.pyenv/versions/3.8.16/lib/python3.8/site-packages
2024-08-23 00:08:31.356 [info] /Users/mat/.pyenv/versions/3.8.16/bin/python -m black --version
2024-08-23 00:08:31.358 [info] CWD formatter: /Users/mat/e/firmware/iot_thing
2024-08-23 00:08:31.359 [info] Version info for formatter running for /Users/mat/e/firmware/iot_thing:
black, 24.3.0 (compiled: no)
Python (CPython) 3.8.16

2024-08-23 00:08:31.381 [info] SUPPORTED black>=22.3.0
FOUND black==24.3.0

2024-08-23 09:56:29.682 [info] [Error - 9:56:29 AM] Server process exited with signal SIGTERM.
2024-08-23 09:56:29.693 [info] [Info  - 9:56:29 AM] Connection to server got closed. Server will restart.
2024-08-23 09:56:29.693 [info] true
2024-08-23 09:56:30.470 [info] CWD Server: /Users/mat/e/firmware/iot_thing
2024-08-23 09:56:30.470 [info] Settings received on server:
[
    {
        "cwd": "/Users/mat/e/firmware/iot_thing",
        "workspace": "file:///Users/mat/e/firmware/iot_thing",
        "args": [],
        "path": [],
        "interpreter": [
            "/Users/mat/.pyenv/versions/3.8.16/bin/python"
        ],
        "importStrategy": "useBundled",
        "showNotifications": "off"
    }
]

2024-08-23 09:56:30.470 [info] Global settings received on server:
{
    "cwd": "/",
    "workspace": "/",
    "args": [],
    "path": [],
    "interpreter": [],
    "importStrategy": "useBundled",
    "showNotifications": "off"
}

2024-08-23 09:56:30.470 [info] sys.path used to run Server:
   /Users/mat/.vscode/extensions/ms-python.black-formatter-2024.2.0/bundled/libs
   /Users/mat/.vscode/extensions/ms-python.black-formatter-2024.2.0/bundled/tool
   /Users/mat/.pyenv/versions/3.8.16/lib/python38.zip
   /Users/mat/.pyenv/versions/3.8.16/lib/python3.8
   /Users/mat/.pyenv/versions/3.8.16/lib/python3.8/lib-dynload
   /Users/mat/.pyenv/versions/3.8.16/lib/python3.8/site-packages
2024-08-23 09:56:30.476 [info] /Users/mat/.pyenv/versions/3.8.16/bin/python -m black --version
2024-08-23 09:56:30.476 [info] CWD formatter: /Users/mat/e/firmware/iot_thing
2024-08-23 09:56:30.627 [info] Version info for formatter running for /Users/mat/e/firmware/iot_thing:
black, 24.3.0 (compiled: no)
Python (CPython) 3.8.16

2024-08-23 09:56:30.627 [info] SUPPORTED black>=22.3.0
FOUND black==24.3.0

2024-08-23 09:56:30.915 [info] [Error - 9:56:30 AM] Server process exited with signal SIGTERM.
2024-08-23 09:56:30.916 [info] [Info  - 9:56:30 AM] Connection to server got closed. Server will restart.
2024-08-23 09:56:30.916 [info] true
2024-08-23 09:56:31.535 [info] CWD Server: /Users/mat/e/firmware/iot_thing
2024-08-23 09:56:31.535 [info] Settings received on server:
[
    {
        "cwd": "/Users/mat/e/firmware/iot_thing",
        "workspace": "file:///Users/mat/e/firmware/iot_thing",
        "args": [],
        "path": [],
        "interpreter": [
            "/Users/mat/.pyenv/versions/3.8.16/bin/python"
        ],
        "importStrategy": "useBundled",
        "showNotifications": "off"
    }
]

2024-08-23 09:56:31.535 [info] Global settings received on server:
{
    "cwd": "/",
    "workspace": "/",
    "args": [],
    "path": [],
    "interpreter": [],
    "importStrategy": "useBundled",
    "showNotifications": "off"
}

2024-08-23 09:56:31.536 [info] sys.path used to run Server:
   /Users/mat/.vscode/extensions/ms-python.black-formatter-2024.2.0/bundled/libs
   /Users/mat/.vscode/extensions/ms-python.black-formatter-2024.2.0/bundled/tool
   /Users/mat/.pyenv/versions/3.8.16/lib/python38.zip
   /Users/mat/.pyenv/versions/3.8.16/lib/python3.8
   /Users/mat/.pyenv/versions/3.8.16/lib/python3.8/lib-dynload
   /Users/mat/.pyenv/versions/3.8.16/lib/python3.8/site-packages
2024-08-23 09:56:31.539 [info] /Users/mat/.pyenv/versions/3.8.16/bin/python -m black --version
2024-08-23 09:56:31.539 [info] CWD formatter: /Users/mat/e/firmware/iot_thing
2024-08-23 09:56:31.646 [info] Version info for formatter running for /Users/mat/e/firmware/iot_thing:
black, 24.3.0 (compiled: no)
Python (CPython) 3.8.16

2024-08-23 09:56:31.646 [info] SUPPORTED black>=22.3.0
FOUND black==24.3.0

2024-08-23 09:56:32.399 [info] [Error - 9:56:32 AM] Server process exited with signal SIGTERM.
2024-08-23 09:56:32.399 [info] [Info  - 9:56:32 AM] Connection to server got closed. Server will restart.
2024-08-23 09:56:32.399 [info] true
2024-08-23 09:56:33.021 [info] CWD Server: /Users/mat/e/firmware/iot_thing
2024-08-23 09:56:33.021 [info] Settings received on server:
[
    {
        "cwd": "/Users/mat/e/firmware/iot_thing",
        "workspace": "file:///Users/mat/e/firmware/iot_thing",
        "args": [],
        "path": [],
        "interpreter": [
            "/Users/mat/.pyenv/versions/3.8.16/bin/python"
        ],
        "importStrategy": "useBundled",
        "showNotifications": "off"
    }
]

2024-08-23 09:56:33.021 [info] Global settings received on server:
{
    "cwd": "/",
    "workspace": "/",
    "args": [],
    "path": [],
    "interpreter": [],
    "importStrategy": "useBundled",
    "showNotifications": "off"
}

2024-08-23 09:56:33.021 [info] sys.path used to run Server:
   /Users/mat/.vscode/extensions/ms-python.black-formatter-2024.2.0/bundled/libs
   /Users/mat/.vscode/extensions/ms-python.black-formatter-2024.2.0/bundled/tool
   /Users/mat/.pyenv/versions/3.8.16/lib/python38.zip
   /Users/mat/.pyenv/versions/3.8.16/lib/python3.8
   /Users/mat/.pyenv/versions/3.8.16/lib/python3.8/lib-dynload
   /Users/mat/.pyenv/versions/3.8.16/lib/python3.8/site-packages
2024-08-23 09:56:33.024 [info] /Users/mat/.pyenv/versions/3.8.16/bin/python -m black --version
2024-08-23 09:56:33.025 [info] CWD formatter: /Users/mat/e/firmware/iot_thing
2024-08-23 09:56:33.133 [info] Version info for formatter running for /Users/mat/e/firmware/iot_thing:
black, 24.3.0 (compiled: no)
Python (CPython) 3.8.16

2024-08-23 09:56:33.133 [info] SUPPORTED black>=22.3.0
FOUND black==24.3.0

2024-08-23 09:56:36.599 [info] [Error - 9:56:36 AM] Server process exited with signal SIGKILL.
2024-08-23 09:56:36.599 [info] [Info  - 9:56:36 AM] Connection to server got closed. Server will restart.
2024-08-23 09:56:36.599 [info] true
2024-08-23 09:56:37.289 [info] CWD Server: /Users/mat/e/firmware/iot_thing
2024-08-23 09:56:37.289 [info] Settings received on server:
[
    {
        "cwd": "/Users/mat/e/firmware/iot_thing",
        "workspace": "file:///Users/mat/e/firmware/iot_thing",
        "args": [],
        "path": [],
        "interpreter": [
            "/Users/mat/.pyenv/versions/3.8.16/bin/python"
        ],
        "importStrategy": "useBundled",
        "showNotifications": "off"
    }
]

2024-08-23 09:56:37.289 [info] Global settings received on server:
{
    "cwd": "/",
    "workspace": "/",
    "args": [],
    "path": [],
    "interpreter": [],
    "importStrategy": "useBundled",
    "showNotifications": "off"
}

2024-08-23 09:56:37.289 [info] sys.path used to run Server:
   /Users/mat/.vscode/extensions/ms-python.black-formatter-2024.2.0/bundled/libs
   /Users/mat/.vscode/extensions/ms-python.black-formatter-2024.2.0/bundled/tool
   /Users/mat/.pyenv/versions/3.8.16/lib/python38.zip
   /Users/mat/.pyenv/versions/3.8.16/lib/python3.8
   /Users/mat/.pyenv/versions/3.8.16/lib/python3.8/lib-dynload
   /Users/mat/.pyenv/versions/3.8.16/lib/python3.8/site-packages
2024-08-23 09:56:37.293 [info] /Users/mat/.pyenv/versions/3.8.16/bin/python -m black --version
2024-08-23 09:56:37.293 [info] CWD formatter: /Users/mat/e/firmware/iot_thing
2024-08-23 09:56:37.416 [info] Version info for formatter running for /Users/mat/e/firmware/iot_thing:
black, 24.3.0 (compiled: no)
Python (CPython) 3.8.16

2024-08-23 09:56:37.416 [info] SUPPORTED black>=22.3.0
FOUND black==24.3.0

2024-08-23 09:56:37.635 [info] [Error - 9:56:37 AM] Server process exited with signal SIGKILL.
2024-08-23 09:56:37.637 [info] [Error - 9:56:37 AM] The Black Formatter server crashed 5 times in the last 3 minutes. The server will not be restarted. See the output for more information.