Black Magic Probe's `run` causes race condition with CDT leading to debug commands not working

[jief666]

Hi, In the startup page of my GDB hardware debug configuration, I didn't click "resume". Instead I put "-exec-run" in the text area. ("continue" skip the softdevice initialization of my nrf52833, so no good). If there is no breakpoint at all, it's working well. I can stop and resume the process and do variable exploration and everything. If there is some breakpoint, for example in the main function, the debug commands (continue, stop, step, etc.) doesn't work anymore. Eclipse is confused. Sometime the debug view show that the process is running but the debugger console says it is stopped. Sometime Eclipse become totally unresponsive (killing gdb process gives back control of Eclipse). I've noticed that if I put a delay (TimeUnit.SECONDS.sleep(2)) at the beginning of org.eclipse.cdt.dsf.mi.service.MIBreakpointsManager.doUninstallBreakpoint(), the control problem disappear. So I think there is a race condition somewhere. I don't think I can find that by myself. Any help ?

PS : I can explain more, make a screencast or anything needed to help finding that bug.

[jonahgraham]

cc: @jld01 @T-Svensson as they may have an idea. This bug has been transferred from Bug 580544

[Torbjorn-Svensson]

Nothing obvious comes to mind, but doing run control things in the free text area is generally a bad idea. The reason for this statement is that CDT expects to be in charge of run control until the launch sequence completes, but the free text box is just one of many steps that are included in the launch sequence.

[jief666]

Can we have a check box "run" instead of "resume", so CDT is still in charge ?

[Torbjorn-Svensson]

@jief666: Can you please describe why "resume" does not work for you and why you need "run" instead? "run" is usually used when you want to create a new process and debug it - more or less only usable when there is a real OS. For bare metal targets, you usually only have one process and there is no way to restart the application other than writing some target specific memory to reset the chip or move $pc to the entry point of the application.

[jief666]

"continue" skip the softdevice initialization of my nrf52833 and dfu bootloader. Run works well. The chip restart its boot process. So softdevice is initialized. Softdevice then call bootloader. Bootloader then starts the application.

[jief666]

If I don't use the Nordic softdevice, resume works.

[Torbjorn-Svensson]

Are you using OpenOCD or some other GDB server? Does it have any means to do a reset? OpenOCD for example supports a few monitor commands, including one that will reset the target. Maybe this is what you are looking for? Another possibility is to simply set $pc to where your "softdevice" entry point is located and then run "resume".

[jief666]

I use Black Magic Probe. Yes, setting PC might work. But a reset of the chip is better because it is the same as when the chip starts without GDB. But because it works on my computer with a delay in the java code, I still think this is the occasion to find a bug...

[jld01]

@jief666 what does the GDB command "show remote exec-file" indicate when added to the launch configuration Run Commands text box before you attempt to "-exec-run"?

[jief666]

I'm on another project of the same workspace. This time, launch works, with or without a breakpoint. show remote exec-file shows nothing in gdb traces. Also shows nothing after I manually interrupted the program and enter show remote exec-file in debugger console.

[jief666]

Here is an example of gdb traces where eclipse "thought" gdb was started, but gdb was stopped and waiting for commands. I had to enter 'continue' in the debugger console. So it's like the -exec-run was understood by eclipse by not by gdb.

196,132 2-gdb-version
196,132 ~"GNU gdb (GDB) 12.1\n"
196,133 ~"Copyright (C) 2022 Free Software Foundation, Inc.\n"
196,133 ~"License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>\nThis is fre\
e software: you are free to change and redistribute it.\nThere is NO WARRANTY, to the extent permitt\
ed by law."
196,133 ~"\nType \"show copying\" and \"show warranty\" for details.\n"
196,133 ~"This GDB was configured as \"x86_64-apple-darwin18.7.0\".\n"
196,133 ~"Type \"show configuration\" for configuration details.\n"
196,133 ~"For bug reporting instructions, please see:\n"
196,133 ~"<https://www.gnu.org/software/gdb/bugs/>.\n"
196,133 ~"Find the GDB manual and other documentation resources online at:\n    <http://www.gnu.org/\
software/gdb/documentation/>."
196,133 ~"\n\n"
196,133 ~"For help, type \"help\".\n"
196,133 ~"Type \"apropos word\" to search for commands related to \"word\".\n"
196,133 2^done
196,133 (gdb) 
196,133 3-environment-cd /JiefLand/5.Devel/Embedded/nrf52833/04.Usbd_MIDI
196,134 3^done
196,134 (gdb) 
196,134 4-gdb-set breakpoint pending on
196,134 4^done
196,134 (gdb) 
196,135 5-enable-pretty-printing
196,135 5^done
196,135 (gdb) 
196,135 6-gdb-set python print-stack none
196,135 6^done
196,135 (gdb) 
196,136 7-gdb-set print object on
196,136 7^done
196,136 (gdb) 
196,136 8-gdb-set print sevenbit-strings on
196,137 8^done
196,137 (gdb) 
196,137 9-gdb-set charset ISO-8859-1
196,137 9^done
196,137 (gdb) 
196,137 10-gdb-set dprintf-style gdb
196,138 10^done
196,138 (gdb) 
196,138 11source .gdbinit
196,138 &"source .gdbinit\n"
196,139 &".gdbinit: No such file or directory.\n"
196,139 11^error,msg=".gdbinit: No such file or directory."
196,139 (gdb) 
196,139 12-gdb-set target-async off
196,139 ~"Warning: 'set target-async', an alias for the command 'set mi-async', is deprecated.\n"
196,140 ~"Use 'set mi-async'.\n\n"
196,140 12^done
196,140 (gdb) 
196,140 13-gdb-set auto-solib-add on
196,141 13^done
196,141 (gdb) 
196,155 14-file-exec-file /JiefLand/5.Devel/Embedded/nrf52833/04.Usbd_MIDI/Debug/04.Usbd_MIDI.elf
196,157 14^done
196,157 (gdb) 
196,158 15-gdb-show --thread-group i1 language
196,158 15^done,value="auto"
196,158 (gdb) 
196,158 16-gdb-set --thread-group i1 language c
196,159 16^done
196,159 (gdb) 
196,159 17-interpreter-exec --thread-group i1 console "p/x (char)-1"
196,160 ~"$1 = 0xff\n"
196,160 17^done
196,160 (gdb) 
196,160 18-data-evaluate-expression --thread-group i1 "sizeof (void*)"
196,161 18^done,value="4"
196,161 (gdb) 
196,161 19-gdb-set --thread-group i1 language auto
196,162 19^done
196,162 (gdb) 
196,162 20-interpreter-exec --thread-group i1 console "show endian"
196,162 ~"The target endianness is set automatically (currently little endian).\n"
196,163 20^done
196,163 (gdb) 
196,163 21-target-select extended-remote /dev/cu.usbmodemC3DEABD41

196,171 21^connected
196,171 (gdb) 
196,171 22set mem inaccessible-by-default off
196,171 23monitor swdp_scan
196,171 24attach 1
196,171 25file Debug/04.Usbd_MIDI.elf
196,171 &"\n"
196,171 ^done
196,171 (gdb) 
196,171 &"set mem inaccessible-by-default off\n"
196,171 =cmd-param-changed,param="mem inaccessible-by-default",value="off"
196,171 22^done
196,172 (gdb) 
196,172 &"monitor swdp_scan\n"
196,172 @"Target voltage: 3.31V\n"
196,391 @"Available Targets:\n"
196,391 @"No. Att Driver\n"
196,392 @" 1      Nordic nRF52 M4\n"
196,392 @" 2      Nordic nRF52 Access Port \n"
196,393 23^done
196,393 (gdb) 
196,393 &"attach 1\n"
196,394 26load
196,394 27starti
196,394 28-list-thread-groups
196,396 ~"Attaching to program: /JiefLand/5.Devel/Embedded/nrf52833/04.Usbd_MIDI/Debug/04.Usbd_MIDI.\
elf, Remote target\n"
196,397 =thread-group-started,id="i1",pid="1"
196,398 =thread-created,id="1",group-id="i1"
196,398 *running,thread-id="1"
196,419 24^done
196,419 ~"0x0002ebbe in ?? ()\n"
196,419 *stopped,frame={addr="0x0002ebbe",func="??",args=[],arch="armv7e-m"},thread-id="1",stopped-t\
hreads="all"
196,419 (gdb) 
196,419 &"file Debug/04.Usbd_MIDI.elf\n"
196,419 ~"A program is being debugged already.\nAre you sure you want to change the file? "
196,419 ~"(y or n) [answered Y; input not from terminal]\n"
196,420 ~"Reading symbols from Debug/04.Usbd_MIDI.elf...\n"
196,538 25^done
196,538 (gdb) 
196,539 &"load\n"
198,217 ~"Loading section .text, size 0x12614 lma 0x27000\n"
198,219 26+download,{section=".text",section-size="75284",total-size="1473905"}
198,219 26+download,{section=".text",section-sent="992",section-size="75284",total-sent="992",total-\
size="1473905"}
198,721 26+download,{section=".text",section-sent="57808",section-size="75284",total-sent="57808",to\
tal-size="1473905"}
198,873 ~"Loading section .log_const_data, size 0x60 lma 0x39614\n"
198,873 26+download,{section=".log_const_data",section-size="96",total-size="1473905"}
198,874 ~"Loading section .log_backends, size 0x10 lma 0x39674\n"
198,874 26+download,{section=".log_backends",section-size="16",total-size="1473905"}
198,874 ~"Loading section .cli_command, size 0x50 lma 0x39684\n"
198,874 26+download,{section=".cli_command",section-size="80",total-size="1473905"}
198,875 ~"Loading section .nrf_balloc, size 0x14 lma 0x396d4\n"
198,875 26+download,{section=".nrf_balloc",section-size="20",total-size="1473905"}
198,875 ~"Loading section .ARM.exidx, size 0x8 lma 0x396e8\n"
198,875 26+download,{section=".ARM.exidx",section-size="8",total-size="1473905"}
198,875 ~"Loading section .data, size 0x32c lma 0x396f0\n"
198,890 26+download,{section=".data",section-size="812",total-size="1473905"}
198,890 ~"Loading section .cli_sorted_cmd_ptrs, size 0x28 lma 0x39a1c\n"
198,890 26+download,{section=".cli_sorted_cmd_ptrs",section-size="40",total-size="1473905"}
198,896 ~"Start address 0x00029a80, load size 76356\n"
198,897 ~"Transfer rate: 31 KB/sec, 909 bytes/write.\n"
198,897 26^done
198,897 (gdb) 
198,897 &"starti\n"
198,897 ~"The program being debugged has been started already.\nStart it from the beginning? "
198,897 ~"(y or n) [answered Y; input not from terminal]\n"
198,917 =thread-exited,id="1",group-id="i1"
198,918 =thread-group-exited,id="i1"
198,919 ~"Starting program: /JiefLand/5.Devel/Embedded/nrf52833/04.Usbd_MIDI/Debug/04.Usbd_MIDI.elf \
\n"
198,958 =thread-group-started,id="i1",pid="42000"
198,958 =thread-created,id="1",group-id="i1"
198,972 27^running
198,972 *running,thread-id="all"
198,972 (gdb) 
198,973 ~"\nProgram"
198,973 ~" stopped.\n"
198,979 ~"0x00000a80 in ?? ()\n"
198,981 *stopped,reason="signal-received",signal-name="0",signal-meaning="Signal 0",frame={addr="0x0\
0000a80",func="??",args=[],arch="armv7e-m"},thread-id="1",stopped-threads="all"
198,981 (gdb) 
198,988 28^done,groups=[{id="i1",type="process",pid="42000",executable="/JiefLand/5.Devel/Embedded/n\
rf52833/04.Usbd_MIDI/Debug/04.Usbd_MIDI.elf"}]
198,988 (gdb) 
199,024 29-list-thread-groups i1
199,027 29^done,threads=[{id="1",target-id="Thread 1",frame={level="0",addr="0x00000a80",func="??",a\
rgs=[],arch="armv7e-m"},state="stopped"}]
199,027 (gdb) 
199,030 30-stack-info-depth --thread 1 11
199,033 30^done,depth="2"
199,034 (gdb) 
199,059 31-stack-select-frame --thread 1 0
199,060 31^done
199,060 (gdb) 
199,076 32-break-insert -f /JiefLand/5.Devel/Embedded/nrf52833/04.Usbd_MIDI/src/usb.c:611
199,076 33-break-insert -f /JiefLand/5.Devel/Embedded/nrf52833/04.Usbd_MIDI/nRF5_SDK_17.1.0_ddde560/\
modules/nrfx/hal/nrf_clock.h:439
199,089 34-list-thread-groups
199,089 35-list-thread-groups i1
199,212 ~"Note: automatically using hardware breakpoints for read-only addresses.\n"
199,212 32^done,bkpt={number="1",type="breakpoint",disp="keep",enabled="y",addr="0x00028e9a",func="e\
p_configuration",file="../src/usb.c",fullname="/JiefLand/5.Devel/Embedded/nrf52833/04.Usbd_MIDI/src/\
usb.c",line="611",thread-groups=["i1"],times="0",original-location="/JiefLand/5.Devel/Embedded/nrf52\
833/04.Usbd_MIDI/src/usb.c:611"}
199,212 (gdb) 
199,495 33^done,bkpt={number="2",type="breakpoint",disp="keep",enabled="y",addr="0x0002a178",func="n\
rf_clock_lf_src_set",file="../nRF5_SDK_17.1.0_ddde560/modules/nrfx/hal/nrf_clock.h",fullname="/JiefL\
and/5.Devel/Embedded/nrf52833/04.Usbd_MIDI/nRF5_SDK_17.1.0_ddde560/modules/nrfx/hal/nrf_clock.h",lin\
e="439",thread-groups=["i1"],times="0",original-location="/JiefLand/5.Devel/Embedded/nrf52833/04.Usb\
d_MIDI/nRF5_SDK_17.1.0_ddde560/modules/nrfx/hal/nrf_clock.h:439"}
199,495 (gdb) 
199,496 34^done,groups=[{id="i1",type="process",pid="42000",executable="/JiefLand/5.Devel/Embedded/n\
rf52833/04.Usbd_MIDI/Debug/04.Usbd_MIDI.elf"}]
199,496 (gdb) 
199,496 36show remote exec-file
199,496 37-exec-run
199,497 35^done,threads=[{id="1",target-id="Thread 1",frame={level="0",addr="0x00000a80",func="??",a\
rgs=[],arch="armv7e-m"},state="stopped"}]
199,497 (gdb) 
199,497 &"show remote exec-file\n"
199,497 ~"\n"
199,497 36^done
199,497 (gdb) 
199,521 =thread-exited,id="1",group-id="i1"
199,522 =thread-group-exited,id="i1"
199,532 38-list-thread-groups
199,537 =thread-group-started,id="i1",pid="42000"
199,537 =thread-created,id="1",group-id="i1"
199,559 37^running
199,559 *running,thread-id="all"
199,559 (gdb) 
199,560 39-break-list
199,561 40-break-list
199,590 =breakpoint-modified,bkpt={number="2",type="breakpoint",disp="keep",enabled="y",addr="0x0002\
a178",func="nrf_clock_lf_src_set",file="../nRF5_SDK_17.1.0_ddde560/modules/nrfx/hal/nrf_clock.h",ful\
lname="/JiefLand/5.Devel/Embedded/nrf52833/04.Usbd_MIDI/nRF5_SDK_17.1.0_ddde560/modules/nrfx/hal/nrf\
_clock.h",line="439",thread-groups=["i1"],times="1",original-location="/JiefLand/5.Devel/Embedded/nr\
f52833/04.Usbd_MIDI/nRF5_SDK_17.1.0_ddde560/modules/nrfx/hal/nrf_clock.h:439"}
199,609 &"Quit\n"
199,609 (gdb) 
199,611 38^done,reason="breakpoint-hit",disp="keep",bkptno="2",groups=[{id="i1",type="process",pid="\
42000",executable="/JiefLand/5.Devel/Embedded/nrf52833/04.Usbd_MIDI/Debug/04.Usbd_MIDI.elf"}]
199,611 (gdb) 
199,611 39^done,BreakpointTable={nr_rows="2",nr_cols="6",hdr=[{width="7",alignment="-1",col_name="nu\
mber",colhdr="Num"},{width="14",alignment="-1",col_name="type",colhdr="Type"},{width="4",alignment="\
-1",col_name="disp",colhdr="Disp"},{width="3",alignment="-1",col_name="enabled",colhdr="Enb"},{width\
="10",alignment="-1",col_name="addr",colhdr="Address"},{width="40",alignment="2",col_name="what",col\
hdr="What"}],body=[bkpt={number="1",type="breakpoint",disp="keep",enabled="y",addr="0x00028e9a",func\
="ep_configuration",file="../src/usb.c",fullname="/JiefLand/5.Devel/Embedded/nrf52833/04.Usbd_MIDI/s\
rc/usb.c",line="611",thread-groups=["i1"],times="0",original-location="/JiefLand/5.Devel/Embedded/nr\
f52833/04.Usbd_MIDI/src/usb.c:611"},bkpt={number="2",type="breakpoint",disp="keep",enabled="y",addr=\
"0x0002a178",func="nrf_clock_lf_src_set",file="../nRF5_SDK_17.1.0_ddde560/modules/nrfx/hal/nrf_clock\
.h",fullname="/JiefLand/5.Devel/Embedded/nrf52833/04.Usbd_MIDI/nRF5_SDK_17.1.0_ddde560/modules/nrfx/\
hal/nrf_clock.h",line="439",thread-groups=["i1"],times="1",original-location="/JiefLand/5.Devel/Embe\
dded/nrf52833/04.Usbd_MIDI/nRF5_SDK_17.1.0_ddde560/modules/nrfx/hal/nrf_clock.h:439"}]}
199,611 (gdb) 
199,611 40^done,BreakpointTable={nr_rows="2",nr_cols="6",hdr=[{width="7",alignment="-1",col_name="nu\
mber",colhdr="Num"},{width="14",alignment="-1",col_name="type",colhdr="Type"},{width="4",alignment="\
-1",col_name="disp",colhdr="Disp"},{width="3",alignment="-1",col_name="enabled",colhdr="Enb"},{width\
="10",alignment="-1",col_name="addr",colhdr="Address"},{width="40",alignment="2",col_name="what",col\
hdr="What"}],body=[bkpt={number="1",type="breakpoint",disp="keep",enabled="y",addr="0x00028e9a",func\
="ep_configuration",file="../src/usb.c",fullname="/JiefLand/5.Devel/Embedded/nrf52833/04.Usbd_MIDI/s\
rc/usb.c",line="611",thread-groups=["i1"],times="0",original-location="/JiefLand/5.Devel/Embedded/nr\
f52833/04.Usbd_MIDI/src/usb.c:611"},bkpt={number="2",type="breakpoint",disp="keep",enabled="y",addr=\
"0x0002a178",func="nrf_clock_lf_src_set",file="../nRF5_SDK_17.1.0_ddde560/modules/nrfx/hal/nrf_clock\
.h",fullname="/JiefLand/5.Devel/Embedded/nrf52833/04.Usbd_MIDI/nRF5_SDK_17.1.0_ddde560/modules/nrfx/\
hal/nrf_clock.h",line="439",thread-groups=["i1"],times="1",original-location="/JiefLand/5.Devel/Embe\
dded/nrf52833/04.Usbd_MIDI/nRF5_SDK_17.1.0_ddde560/modules/nrfx/hal/nrf_clock.h:439"}]}
199,612 (gdb) 

[jonahgraham]

Something unusual about this trace is that there is this:

14-file-exec-file /JiefLand/5.Devel/Embedded/nrf52833/04.Usbd_MIDI/Debug/04.Usbd_MIDI.elf

followed later by

25file Debug/04.Usbd_MIDI.elf

but it is strange to re-specify the file and indeed GDB prompts with this:

196,419 ~"A program is being debugged already.\nAre you sure you want to change the file? "
196,419 ~"(y or n) [answered Y; input not from terminal]\n"

[jief666]

Here's what I have in "Initialization commands" :

set mem inaccessible-by-default off
monitor swdp_scan
attach 1
file Debug/04.Usbd_MIDI.elf
load
starti

and "Run commands" :

show remote exec-file
-exec-run

Looks like I made the mistake to leave "starti". It's because I tried to check "resume" instead of "--exec-run". But it shouldn't be the reason there is "file-exec-file" and "file" commands. NOTE : with "starti" and "resume" it's working a bit better, but not always. If it's a race condition, it seems normal to work sometimes but not always... I can make some more tests.

[jld01]

@jief666 the GDB Hardware Debugging launch sequence is not intended to accommodate running/suspending the target within the initialization commands section. Try removing your file, load and starti commands from this section. Load both the image and the symbols from the project binary by checking the "Load image" and "Load symbols" boxes. Start the target either by using a command in the run commands section or by checking the "Resume" box (but not both).

[jief666]

If I put file in "initialization commands" and don't check "Load image" and "Load symbols", what's the difference ?

Anyway, I can tell that it was what I was doing at the beginning : "Load file" + "Load symbols" and then "run" in run commands. It doesn't work. But I can re-do that test if it's needed.