Very long delay when a breakpoint is hit and many (150+) threads are created by the program

[jusid]

Environment

• OS and version: Ubuntu 20.04.5 LTS
• VS Code: 1.72.1
• C/C++ extension: v1.12.4
• OS and version of remote machine (if applicable): gdbserver is running on an embedded ARMv7 Linux
• GDB version: 7.4

Bug Summary and Steps to Reproduce

Bug Summary: I am developing a C application with 150+ threads and debug it remotely via gdbserver. When the first breakpoint is hit, vscode requests detailed information about every thread in the application and it takes about 3 minutes to complete. The debugging is very painful due to this. The communication between gdb and gdbserver is very sensitive to the network latency. It is important to make as few requests to gdb/gdbserver as possible.

Steps to reproduce: Create an app which starts 150+ threads. Debug it remotely using gdbserver. Set a breakpoint to be hit after all threads are created already. Debug the program. When the breakpoint is hit you will get a significant delay before vscode UI reports the breakpoint hit.

Debugger Configurations

{
    // Use IntelliSense to learn about possible attributes.
    // Hover to view descriptions of existing attributes.
    // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
    "version": "0.2.0",
    "configurations": [
        {
            "name": "(gdb) Debug app",
            "type": "cppdbg",
            "request": "launch",
            "program": "link.out",
            "stopAtEntry": false,
            "cwd": "${workspaceRoot}",
            "environment": [],
            "externalConsole": true,
            "MIMode": "gdb",
            "miDebuggerPath": "arm-linux-gnueabi-gdb",
            "miDebuggerServerAddress": "10.0.1.10:9099",
            "targetArchitecture": "arm",
            "symbolLoadInfo": {
                "loadAll": false
            },
            "logging": {
                "engineLogging": false
            },
            "setupCommands": [
                {
                    "description": "Enable pretty-printing for gdb",
                    "text": "-enable-pretty-printing",
                    "ignoreFailures": true
                },
                {
                    "text": "-interpreter-exec console \"set sysroot build_dir/sysroot\"",
                    "ignoreFailures": true
                }
            ]
        }
    ]
}

Debugger Logs

Here is what happens when a break point is hit and why it takes so long. The break point is hit:

1: (20384) ->=thread-created,id="2",group-id="i1"
1: (20384) ->~"[New Thread 1800]\n"
1: (20385) <-1023-thread-info 2
[New Thread 1800]
1: (20389) ->=breakpoint-modified,bkpt={number="3",type="breakpoint",disp="keep",enabled="y",addr="0x00b5efb4",func="x",file="x.c",fullname="x.c",line="568",times="1",original-location="x.c:568"}
1: (20390) ->~"[Switching to Thread 1800]\n"
[Switching to Thread 1800]
1: (20466) ->*stopped,reason="breakpoint-hit",disp="keep",bkptno="3",frame={addr="0x00b5efb4",func="x",args=[],file="x.c",fullname="x.c",line="568"},thread-id="2",stopped-threads="all",core="0"

Information about all(!) threads is requested:

1: (20467) <-1024-thread-info

While GDB executes the request above, it reports availability of new threads (158 in my case):

1: (20509) ->=thread-created,id="3",group-id="i1"
1: (20509) ->~"[New Thread 1639]\n"
1: (20509) ->=thread-created,id="4",group-id="i1"
1: (20509) ->~"[New Thread 1640]\n"
1: (20509) ->=thread-created,id="5",group-id="i1"
1: (20510) ->~"[New Thread 1641]\n"

vscode starts to request info about each reported thread. Even though information about all threads has been requested earlier:

1: (20510) <-1025-thread-info 3
1: (20510) ->=thread-created,id="6",group-id="i1"
1: (20510) ->~"[New Thread 1642]\n"
[New Thread 1639]
1: (20510) ->=thread-created,id="7",group-id="i1"
1: (20510) ->~"[New Thread 1649]\n"
1: (20510) ->=thread-created,id="8",group-id="i1"
...

...
1: (20528) <-1178-thread-info 156
[New Thread 1798]
1: (20528) <-1179-thread-info 157
[New Thread 1799]
1: (20528) <-1180-thread-info 158
[New Thread 1801]
1: (20528) 1023: elapsed time 143
1: (20528) Send Event AD7ThreadCreateEvent
...

Information about all 158 threads has been requested. After 11 seconds GDB returns response to request -thread-info``` and starts to slowly return responses for individual thread info requests:

...
1: (31603) ->1024^done,threads=[{id="158",target-id="Thread 1801",frame={level="0",addr="0xb6e97798",func="??",args=[]},state="stopped",core="0"},{id="157",target-id="Thread 1799",frame={level="0",addr="0xb6fa7ce4",func="??",args=[]},state="stopped",core="0"}, ...
1: (31604) ->(gdb)
1: (31606) 1024: elapsed time 11138
1: (32457) ->1025^done,threads=[{id="3",target-id="Thread 1639",frame={level="0",addr="0xb6e91e7c",func="??",args=[]},state="stopped",core="0"}]
1: (32458) ->(gdb)
1: (32458) 1025: elapsed time 11948
1: (33343) ->1026^done,threads=[{id="4",target-id="Thread 1640",frame={level="0",addr="0xb6e97798",func="??",args=[]},state="stopped",core="0"}]
...

After 151 (!) seconds GDB responds to the last thread info request and vscode is almost ready to inform a user about the break point:

...
1: (171843) ->1180^done,threads=[{id="158",target-id="Thread 1801",frame={level="0",addr="0xb6e97798",func="??",args=[]},state="stopped",core="0"}]
1: (171843) ->(gdb)
1: (171844) 1180: elapsed time 151315
1: (171845) <-1181-stack-list-frames 0 1000

Conclusion: it is not needed to request individual thread info since all thread info is requested anyway and such combined request is executed much much faster compared to many individual requests.

[reneparis]

You got my vote - got similar problems on our side.

[olgapshen]

My vote

[jaemzfleming]

you got my vote too, this is a horrible issue that makes vscode much less useful than it should be. We should be able to debug in a timely manner. I'm new to this group. Is there some formal mechanism I can use to show my vote?

[rlivings39]

+1 the C++ debugging performance is the major workflow pain our team and colleagues experience using VSCode as a C++ IDE. It gets so bad for me that I sometimes switch to command-line GDB or use GDB in Emacs. Everything else for C++ development in VSCode scales much better in my experience.

[rcontinuum]

+1

[gewa24]

+1

[muddle216]

+1

[martymcbro]

+1

[krysgor]

+1

[shaharh9110]

+1

[crazydef]

This has nothing to do with remote debugging. Performance when debugging locally is equally awful.

I have to wait minutes to get local variables to appear (during which time I can do nothing). Closing "locals" in the variables window does nothing because it's automatically reopened when execution next stops.

This effectively makes VSCode unfit for purpose and needs to be fixed.

[timmeh87]

+1 why should we have to disable everything just to step normally? Not very useful to me with everything hidden

[er-mo]

+1 from me as well. Very important to have fast gdb on VSCode.

[marcomeyerVFX]

+1 Just recently started debugging a project again after having it sit around for 2 years and this used to work quite decent and snappy. Now it's almost impossible to debug it at all, I wonder what was introduced in the meantime that caused this

[Ubachris]

+1 I'm currently debugging a multi-threaded program (80+ threads) that takes a long time to step through the lines after hitting a breakpoint. It makes debugging with VS code a rather painful experience. I have no such performance issues when debugging with gdb on the command line

[phillipmself]

+1 I am debugging on a platform with many threads (~250) and VSCode is now unusable to for this purpose.

[timmeh87]

Hi all, Is this a related issue? I updated to gdb12 a few months ago and I gotta say the performance seems "nearly reasonable" as a opposed to "it actually literally crashes sometimes". it seems like certain versions of gdb, at least some 7 and some 11, have "bugs" when used with vscode. Check this thread and the post by LeLuc4 for instructions to upgrade to gdb12.

https://github.com/microsoft/vscode-cpptools/issues/103

[crazydef]

@timmeh87 I don't think it's related. I'm using Linux Mint at work, and the version of GDB that ships with is totally broken with VSCode. I had to update to version 12 within a day of getting my machine up and running.

[NikBomb]

+1

[rjoomen]

For reference: debugging ROS 2 code with Ubuntu 22.04 and the default GDB 12.1 was painfully slow, but with GBD 13.1 performance is quite good.

[kotborealis]

Just wanted to address that this issue still persists today on version 1.16.3.

I think that the main problem is an aggressive amount of info threads requests for most events. Maybe there could be some caching mechanism or, at least, a toggle to revert back to a """primitive""" debugger?

[Celthi]

I have given up using vscode to debug any serious C++ program, but I still use it as my code editor for C++ programs.

A similar issue has previously been reported in https://github.com/microsoft/vscode-cpptools/issues/2901 4+ years ago, and it is passed to https://github.com/microsoft/vscode/issues/132025 and then pass back here https://github.com/microsoft/vscode-cpptools/issues/9988.

I'm curious about the root cause of this issue. I'm also a programmer, I should have dug deep to find the root cause for myself.

[krysgor]

I am cross debugging an application with 350+ threads and VSCode is because of this issue not the first choice doing this for me.

I understand that there are more important things than that, but an issue that has been open for so long needs to be addressed at some point. Especially if there is no workaround other than not using VSCode.

So pls fix it.

[sb362]

+1

[solomax]

I'm also affected trying to get familiar with new QT based project DEBUG is useless :( about a minute of waiting if using any of breakpoint, step, continue to line

Version 1.81.1 for Ubuntu

[yi-shui]

+1

[egrosspietsch-fds]

+1

[orlandini]

+1

[YongBig]

+1

[grantyuc]

Vote. Debugging a large fortran project is infeasible since every step takes several minutes to show thousands of variables. There should be an option to toggle the behavior.

[JGLAHE]

+1

[kyselejsyrecek]

+1

[Jorgen-VikingGod]

+1

[nano8259]

+1

[borjamunozf]

Large production codebase debugging experience is really bad.

+1

[kamil-holubicki]

+1

[reneetse]

+1

[nathan-duddles]

+1

[dvirtz]

@sean-mcmanus can we can a status of the plan regarding this issue? It's one of the highest voted issues here (2nd highest if you drop the feature request, the 1st one being also about slowness).

[kotborealis]

FYI: It works significantly better (~2-3x) with a UNIX-socket instead of a TCP one.

[dyollb]

my vote 👍

[Altyrost]

+1

The performances are indeed a lot better with GDB 13 (I tried with 13.2).

[fmadero]

And it works even faster when on the target using gdb or gdbtui. The issue is why is it probing each thread so deep when the user has not requested it.

[solomax]

The performances are indeed a lot better with GDB 13 (I tried with 13.2).

I can confirm! Just have installed GDB 14.2 (from sources) debugging is flawless!

[dvirtz]

I don't feel much improvement with gdb 14.2. Target is ARM 32

[sean-mcmanus]

@dvirtz I don't work on the debugger component -- that's a different team and they're busy doing other debugger work for Visual Studio unless their priorities change in the future (you'll know if that happens if this gets assigned someone and put in a milestone/project). We're aware that this is a highly upvoted issue and I've hit this myself while debugging on Linux and I sent an email to people on the debugger team about it.

[jusid]

The performances are indeed a lot better with GDB 13 (I tried with 13.2).

I can confirm! Just have installed GDB 14.2 (from sources) debugging is flawless!

I've tried remote debugging using GDB 14.2 and the performance is the same. Unfortunately. :(

[sampathrg]

Upvoting. This debugger is really slow when debugging large codebases like llvm.

[tgeffroyModuleus]

+1

[georglauterbach]

+1