Slow listing of threads when hitting breakpoint

[MulattoKid]

Hi,

I'm working on a project on the i.MX RT1060 using ThreadX, and using MCULink + LinkServer + cortex-debug for debugging.

During development the debugging experience has become worse, with hitting breakpoints taking more than 30s to properly stop and letting me interact with the debugger. Inspecting the debug console I see the breakpoint being hit and then all the threads being listed with the GDB commands thread-list-ids and thread-info <thread ID>. I've found that this happens here. Interestingly, if the same breakpoint is hit early on during system initialization it's a lot quicker to gather the information, as there are fewer threads in the system at that point. It makes sense that having more threads would lead to the process of gathering information about all them taking longer, but the increase in time doesn't seem linear (not even close).

I will investigate if there are any specific threads that are causing the process to take so much more time, but I'm wondering if there's anything that can be done to speed this up, besides actually limiting the number of threads?

Here's my system information:

• Ubuntu 22.04 (x86_64)
• VSCode v1.91.1
• cortex-debug v1.12.1
• GDB v12.1 (gcc-arm-none-eabi/now 15:10.3-2021.07-4)
• LinkServer v1.5.30
• MCU-LINK (r0FF) CMSIS-DAP V3.140
• NXP SDK v2.15.0
• i.MX RT1060

Here's the debug console output from the smallest case I can reproduce (go to the line with 27-thread-list-ids to see where it takes a long time to list each thread's information): debug_console.txt

And launch.json:

{
    "configurations": [
        {
            "name": "App Debug",
            "type": "cortex-debug",
            "request": "launch",
            "servertype": "external",
            "gdbTarget": "localhost:3334",
            "cwd": "${workspaceFolder}",
            "executable": "app/build/app_signed.elf",
            "runToEntryPoint": "main",
            "showDevDebugOutput": "parsed",
            "showDevDebugTimestamps": true,
        }
}

Any comments or insight is appreciated!

Thanks, Daniel

[MulattoKid]

I've tested a bit with various number of threads, and going from 14 -> 15 threads is where the behavior changes, and becomes really slow.

When running with 14 threads the Ignoring packet error, continuing... messages in the debug console are gone, and now looks like this: debug_console_14_threads.txt.

Seems to maybe indicate that there is some communication issue somewhere in the chain, and not really related to cortex-debug (besides that it's the thing issuing the GDB commands).

[haneefdm]

We don't do anything special for threads. It is all done by gdb and your gdb-server. We just wait for the results. There have been instances where we had to change strategy as to what commands we send to gdb but it still needs to be identified. You have just as much info as we do, plus you have the HW.

I prefer the output from the "showDevDebugOutput": "raw" method better (more readable) if timestamps still work with that.

You can also duplicate the problem without involving cortex-debug by running gdb on the command line. My suspicion is that even gdb is fine. It is your server I would look into first. Sometimes, an invalid stack can also cause problems (both in gdb and the gdb-server) where they never find the end of the stack. This is common with FreeRTOS where it creates bad stacks when creating tasks that are not properly terminated according to the ARM spec.

[MulattoKid]

Thanks for your reply and suggestions @haneefdm :)

I'll continue investigating, and close this.

[sullivanmj]

This is common with FreeRTOS where it creates bad stacks when creating tasks that are not properly terminated according to the ARM spec.

@haneefdm do you know where I could find more information about this? Is it something that could be found in the arm architecture reference manual for my specific processor core?

[haneefdm]

If you google freertos, arm and stack unwind, you will find many threads

https://forums.freertos.org/t/gdb-stack-unfolding-in-cortex-m4f-port/2176 https://forums.freertos.org/t/gdb-call-stack-unwinding-issue-with-freertos/19441/11 https://stackoverflow.com/questions/14959663/freertos-stack-corruption-on-stm32f4-with-gcc https://community.nxp.com/t5/MCUXpresso-IDE/FreeRTOS-gdb-plugin-seems-to-slow-down-as-the-number-of-active/td-p/1532194

The ARM spec says which registers are callee/caller saved and how the stack is organized. It also specifies somewhere how the LR is used to mark the end of stack unwind which I thought the FreeRTOS does not set or even fake it. I has been a while since I looked at it and can't remember details

[sullivanmj]

I wasn't sure which search terms to use, that makes total sense though. Thank you!