elf not refreshed when restarting debugger after build

[JayToltTech]

Describe the bug

When restarting a debug session after rebuilding my .elf file, the previous build is started by the debugger unless I clean the build folder first.

To Reproduce

Build file use VSCode CMake Build or terminal cmake -S . -B build -G Ninja; ninja -C build -j 10

VSCode 'Run and Debug' with below config:

        {
            "name": "Cortex Debug",
            "cwd": "${workspaceFolder}",
            "executable": "./build/adr-prototype-2307.elf",
            "request": "launch",
            "type": "cortex-debug",
            "runToEntryPoint": "main",
            "servertype": "jlink",
            "device": "ATSAME54P20A",
            "showDevDebugOutput": "none"
        }

View output using JLinkRTTViewer which is show build date from printf("Built on %s at %s\r\n", __DATE__, __TIME__);

I have confirmed that the ELF is modified by viewing the file system. For whatever odd reason, it will not flash until I clean rm -rf build and then rebuild first.

Expected behavior

After build, new ELF is flashed before debugger session is started.

Environment (please complete the following information):

VSCode 1.85.0 Cortex-Debug v1.12.1 Segger JLink Plus Compact Microchip SAME54Xplained MacOS 14.1.2 gcc version 13.2.1 20231009 (Arm GNU Toolchain 13.2.rel1 (Build arm-13.7)) GNU gdb (Arm GNU Toolchain 13.2.rel1 (Build arm-13.7)) 13.2.90.20231008-git

launch.json

{
    // Use IntelliSense to learn about possible attributes.
    // Hover to view descriptions of existing attributes.
    // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
    "version": "0.2.0",
    "configurations": [

        {
            "name": "Cortex Debug",
            "cwd": "${workspaceFolder}",
            "executable": "./build/adr-prototype-2307.elf",
            "request": "launch",
            "type": "cortex-debug",
            "runToEntryPoint": "main",
            "servertype": "jlink",
            "device": "ATSAME54P20A",
            "showDevDebugOutput": "none"
        }
    ]
}

Attach text from Debug Console

Cortex-Debug: VSCode debugger extension version 1.12.1 git(652d042). Usage info: https://github.com/Marus/cortex-debug#usage
"configuration": {
    "name": "Cortex Debug",
    "cwd": "/Users/jay/Code/adr_prototype_2307",
    "executable": "/Users/jay/Code/adr_prototype_2307/build/adr-prototype-2307.elf",
    "request": "launch",
    "type": "cortex-debug",
    "runToEntryPoint": "main",
    "servertype": "jlink",
    "device": "ATSAME54P20A",
    "showDevDebugOutput": "raw",
    "__configurationTarget": 6,
    "gdbServerConsolePort": 55878,
    "pvtAvoidPorts": [],
    "chainedConfigurations": {
        "enabled": false
    },
    "debuggerArgs": [],
    "swoConfig": {
        "enabled": false,
        "decoders": [],
        "cpuFrequency": 0,
        "swoFrequency": 0,
        "source": "probe"
    },
    "rttConfig": {
        "enabled": false,
        "decoders": []
    },
    "graphConfig": [],
    "preLaunchCommands": [],
    "postLaunchCommands": [],
    "preAttachCommands": [],
    "postAttachCommands": [],
    "preRestartCommands": [],
    "postRestartCommands": [],
    "preResetCommands": [],
    "postResetCommands": [],
    "interface": "swd",
    "toolchainPrefix": "arm-none-eabi",
    "extensionPath": "/Users/jay/.vscode/extensions/marus25.cortex-debug-1.12.1",
    "registerUseNaturalFormat": true,
    "variableUseNaturalFormat": true,
    "pvtVersion": "1.12.1",
    "__sessionId": "73bb8753-abdf-4396-980d-1260ddcdbd38",
    "pvtShowDevDebugOutput": "raw"
}
Reading symbols from arm-none-eabi-objdump --syms -C -h -w /Users/jay/Code/adr_prototype_2307/build/adr-prototype-2307.elf
Reading symbols from arm-none-eabi-nm --defined-only -S -l -C -p /Users/jay/Code/adr_prototype_2307/build/adr-prototype-2307.elf
Launching GDB: arm-none-eabi-gdb -q --interpreter=mi2
1-gdb-version
Launching gdb-server: JLinkGDBServer -singlerun -nogui -if swd -port 50000 -swoport 50001 -telnetport 50002 -device ATSAME54P20A
    Please check TERMINAL tab (gdb-server) for output from JLinkGDBServer
Finished reading symbols from objdump: Time: 37 ms
Finished reading symbols from nm: Time: 49 ms
-> =thread-group-added,id="i1"
-> ~"GNU gdb (Arm GNU Toolchain 13.2.rel1 (Build arm-13.7)) 13.2.90.20231008-git\n"
-> ~"Copyright (C) 2023 Free Software Foundation, Inc.\n"
-> ~"License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>\nThis is free software: you are free to change and redistribute it.\nThere is NO WARRANTY, to the extent permitted by law."
-> ~"\nType \"show copying\" and \"show warranty\" for details.\n"
-> ~"This GDB was configured as \"--host=aarch64-apple-darwin20.6.0 --target=arm-none-eabi\".\n"
-> ~"Type \"show configuration\" for configuration details.\n"
-> ~"For bug reporting instructions, please see:\n"
-> ~"<https://bugs.linaro.org/>.\n"
-> ~"Find the GDB manual and other documentation resources online at:\n    <"
-> ~"http://www.gnu.org/software/gdb/documentation/>.\n\n"
-> ~"For help, type \"help\".\n"
-> ~"Type \"apropos word\" to search for commands related to \"word\".\n"
-> 1^done
2-gdb-set mi-async on
-> 2^done
3-interpreter-exec console "set print demangle on"
-> 3^done
4-interpreter-exec console "set print asm-demangle on"
-> =cmd-param-changed,param="print asm-demangle",value="on"
-> 4^done
5-enable-pretty-printing
-> 5^done
6-interpreter-exec console "source /Users/jay/.vscode/extensions/marus25.cortex-debug-1.12.1/support/gdbsupport.init"
-> 6^done
7-interpreter-exec console "source /Users/jay/.vscode/extensions/marus25.cortex-debug-1.12.1/support/gdb-swo.init"
-> =cmd-param-changed,param="language",value="c"
-> =cmd-param-changed,param="language",value="auto"
-> 7^done
8-interpreter-exec console "set output-radix 0xa"
-> ~"Output radix now set to decimal 10, hex a, octal 12.\n"
Output radix now set to decimal 10, hex a, octal 12.
-> 8^done
9-interpreter-exec console "set input-radix 0xa"
-> ~"Input radix now set to decimal 10, hex a, octal 12.\n"
Input radix now set to decimal 10, hex a, octal 12.
-> 9^done
10-file-exec-and-symbols "/Users/jay/Code/adr_prototype_2307/build/adr-prototype-2307.elf"
-> 10^done
11-target-select extended-remote localhost:50000
-> =thread-group-started,id="i1",pid="42000"
-> =thread-created,id="1",group-id="i1"
-> ~"OS_Idle () at /Users/jay/Code/adr_prototype_2307/libraries/embos/Start/BoardSupport/Microchip/SAME54P20A_SAME54_XPlainedPro/Setup/RTOSInit_SAME54.c:252\n"
OS_Idle () at /Users/jay/Code/adr_prototype_2307/libraries/embos/Start/BoardSupport/Microchip/SAME54P20A_SAME54_XPlainedPro/Setup/RTOSInit_SAME54.c:252
-> ~"252\t  while (1) {         // Nothing to do ... wait for interrupt\n"
252   while (1) {         // Nothing to do ... wait for interrupt
-> *stopped,frame={addr="0x00001ac8",func="OS_Idle",args=[],file="/Users/jay/Code/adr_prototype_2307/libraries/embos/Start/BoardSupport/Microchip/SAME54P20A_SAME54_XPlainedPro/Setup/RTOSInit_SAME54.c",fullname="/Users/jay/Code/adr_prototype_2307/libraries/embos/Start/BoardSupport/Microchip/SAME54P20A_SAME54_XPlainedPro/Setup/RTOSInit_SAME54.c",line="252",arch="armv7e-m"},thread-id="1",stopped-threads="all"
mi2.status = stopped
Program stopped, probably due to a reset and/or halt issued by debugger
-> 11^connected
12-interpreter-exec console "monitor halt"
-> 12^done
13-interpreter-exec console "monitor reset"
-> @"Resetting target\r\n"
Resetting target
-> 13^done
14-target-download
-> 14+download,{section=".text",section-size="115916",total-size="4356475"}
-> 14+download,{section=".text",section-sent="16112",section-size="115916",total-sent="16112",total-size="4356475"}
-> 14+download,{section=".ARM.exidx",section-size="8",total-size="4356475"}
-> 14+download,{section=".relocate",section-size="328",total-size="4356475"}
-> 14^done,address="0x00000000",load-size="116252",transfer-rate="465008000",write-rate="11625"
15-interpreter-exec console "monitor reset"
-> @"Resetting target\r\n"
Resetting target
-> 15^done
16-break-insert "/Users/jay/Code/adr_prototype_2307/libraries/embos/Start/BoardSupport/Microchip/SAME54P20A_SAME54_XPlainedPro/Setup/OS_Error.c:289"
-> 16^done,bkpt={number="1",type="breakpoint",disp="keep",enabled="y",addr="0x00001884",func="OS_Error",file="/Users/jay/Code/adr_prototype_2307/libraries/embos/Start/BoardSupport/Microchip/SAME54P20A_SAME54_XPlainedPro/Setup/OS_Error.c",fullname="/Users/jay/Code/adr_prototype_2307/libraries/embos/Start/BoardSupport/Microchip/SAME54P20A_SAME54_XPlainedPro/Setup/OS_Error.c",line="289",thread-groups=["i1"],times="0",original-location="/Users/jay/Code/adr_prototype_2307/libraries/embos/Start/BoardSupport/Microchip/SAME54P20A_SAME54_XPlainedPro/Setup/OS_Error.c:289"}
Returning dummy thread-id to workaround VSCode issue with pause button not working
Returning dummy stack frame to workaround VSCode issue with pause button not working: {"threadId":1,"startFrame":0,"levels":20}
17-break-insert -t --function main
-> 17^done,bkpt={number="2",type="breakpoint",disp="del",enabled="y",addr="0x000002f4",func="main",file="/Users/jay/Code/adr_prototype_2307/main.c",fullname="/Users/jay/Code/adr_prototype_2307/main.c",line="6",thread-groups=["i1"],times="0",original-location="-function main"}
18-exec-continue --all
-> 18^running
-> *running,thread-id="all"
mi2.status = running
-> =breakpoint-modified,bkpt={number="2",type="breakpoint",disp="del",enabled="y",addr="0x000002f4",func="main",file="/Users/jay/Code/adr_prototype_2307/main.c",fullname="/Users/jay/Code/adr_prototype_2307/main.c",line="6",thread-groups=["i1"],times="1",original-location="-function main"}
-> ~"\n"

-> ~"Temporary breakpoint 2, main () at /Users/jay/Code/adr_prototype_2307/main.c:6\n"
Temporary breakpoint 2, main () at /Users/jay/Code/adr_prototype_2307/main.c:6
-> ~"6\t    SYS_Initialize(NULL);\n"
6       SYS_Initialize(NULL);
-> *stopped,reason="breakpoint-hit",disp="del",bkptno="2",frame={addr="0x000002f4",func="main",args=[],file="/Users/jay/Code/adr_prototype_2307/main.c",fullname="/Users/jay/Code/adr_prototype_2307/main.c",line="6",arch="armv7e-m"},thread-id="1",stopped-threads="all"
mi2.status = stopped
-> =breakpoint-deleted,id="2"
19-thread-list-ids
-> 19^done,thread-ids={thread-id="1"},current-thread-id="1",number-of-threads="1"
20-thread-info 1
-> 20^done,threads=[{id="1",target-id="Thread 57005",frame={level="0",addr="0x000002f4",func="main",args=[],file="/Users/jay/Code/adr_prototype_2307/main.c",fullname="/Users/jay/Code/adr_prototype_2307/main.c",line="6",arch="armv7e-m"},state="stopped"}]
21-stack-list-frames --thread 1 0 19
-> 21^done,stack=[frame={level="0",addr="0x000002f4",func="main",file="/Users/jay/Code/adr_prototype_2307/main.c",fullname="/Users/jay/Code/adr_prototype_2307/main.c",line="6",arch="armv7e-m"}]

gdb-server console spew:

[2023-12-07T23:32:20.312Z] SERVER CONSOLE DEBUG: onBackendConnect: gdb-server session connected. You can switch to "DEBUG CONSOLE" to see GDB interactions.
JLinkGDBServer -singlerun -nogui -if swd -port 50000 -swoport 50001 -telnetport 50002 -device ATSAME54P20A
SEGGER J-Link GDB Server V7.86g Command Line Version

JLinkARM.dll V7.86g (DLL compiled Apr  6 2023 16:25:24)

Command line: -singlerun -nogui -if swd -port 50000 -swoport 50001 -telnetport 50002 -device ATSAME54P20A
-----GDB Server start settings-----
GDBInit file:                  none
GDB Server Listening port:     50000
SWO raw output listening port: 50001
Terminal I/O port:             50002
Accept remote connection:      yes
Generate logfile:              off
Verify download:               off
Init regs on start:            off
Silent mode:                   off
Single run mode:               on
Target connection timeout:     0 ms
------J-Link related settings------
J-Link Host interface:         USB
J-Link script:                 none
J-Link settings file:          none
------Target related settings------
Target device:                 ATSAME54P20A
Target device parameters:      none
Target interface:              SWD
Target interface speed:        4000kHz
Target endian:                 little

Connecting to J-Link...
J-Link is connected.
Firmware: J-Link V12 compiled Mar 28 2023 17:00:20
Hardware: V12.00
S/N: 852000911
Feature(s): RDI, FlashBP, FlashDL, JFlash, GDB
Checking target voltage...
Target voltage: 3.30 V
Listening on TCP/IP port 50000
Connecting to target...
Halting core...
Connected to target
Waiting for GDB connection...Connected to 0000:0000:0000:0000:0000:0000:0000:0001
GDB client (conn. 10) requested target.xml from GDB Server
Reading common registers: Read register 'r0' (4 bytes) from hardware: 0x01000000
Read register 'r1' (4 bytes) from hardware: 0xA8AA0020
Read register 'r2' (4 bytes) from hardware: 0x00000000
Read register 'r3' (4 bytes) from hardware: 0x0600CCCC
Read register 'r4' (4 bytes) from hardware: 0xA8AA0020
Read register 'r5' (4 bytes) from hardware: 0x00000000
Read register 'r6' (4 bytes) from hardware: 0x0600CCCC
Read register 'r7' (4 bytes) from hardware: 0x0700CCCC
Read register 'r8' (4 bytes) from hardware: 0x0800CCCC
Read register 'r9' (4 bytes) from hardware: 0x0900CCCC
Read register 'r10' (4 bytes) from hardware: 0x0A00CCCC
Read register 'r11' (4 bytes) from hardware: 0x0B00CCCC
Read register 'r12' (4 bytes) from hardware: 0x0700CCCC
Read register 'sp' (4 bytes) from hardware: 0x70B10120
Read register 'lr' (4 bytes) from hardware: 0x0800CCCC
Read register 'pc' (4 bytes) from hardware: 0xC81A0000
Read register 'xpsr' (4 bytes) from hardware: 0x00000001
Read 4 bytes @ address 0x00001AC8 (Data = 0xB508E7FE)
Received monitor command: halt
Halting target CPU...
...Target halted (PC = 0x00001AC8)
Received monitor command: reset
Resetting target
Downloading 16112 bytes @ address 0x00000000
Downloading 16048 bytes @ address 0x00003EF0
Downloading 16096 bytes @ address 0x00007DA0
Downloading 16016 bytes @ address 0x0000BC80
Downloading 16160 bytes @ address 0x0000FB10
Downloading 16128 bytes @ address 0x00013A30
Downloading 16144 bytes @ address 0x00017930
Downloading 3212 bytes @ address 0x0001B840
Downloading 8 bytes @ address 0x0001C4CC
Downloading 328 bytes @ address 0x0001C4D4
Writing register 'pc' = 0x00000000
Comparing flash   [....................] Done.
Received monitor command: reset
Resetting target
Reading 64 bytes @ address 0x00001880
Read 2 bytes @ address 0x00001884 (Data = 0xF014)
Reading 64 bytes @ address 0x000002C0
Read 2 bytes @ address 0x000002F4 (Data = 0x2000)
Setting breakpoint @ address 0x000002F4, Kind = 2, Type = THUMB, BPHandle = 0x0001
Setting breakpoint @ address 0x00001884, Kind = 2, Type = THUMB, BPHandle = 0x0002
Starting target CPU...
...Breakpoint reached @ address 0x000002F4
Reading common registers: Read register 'r0' (4 bytes) from hardware: 0xD4C40100
Read register 'r1' (4 bytes) from hardware: 0x00000000
Read register 'r2' (4 bytes) from hardware: 0x0000F000
Read register 'r3' (4 bytes) from hardware: 0x00000000
Read register 'r4' (4 bytes) from hardware: 0x04080020
Read register 'r5' (4 bytes) from hardware: 0x24080020
Read register 'r6' (4 bytes) from hardware: 0x24080020
Read register 'r7' (4 bytes) from hardware: 0x0700CCCC
Read register 'r8' (4 bytes) from hardware: 0x0800CCCC
Read register 'r9' (4 bytes) from hardware: 0x0900CCCC
Read register 'r10' (4 bytes) from hardware: 0x0A00CCCC
Read register 'r11' (4 bytes) from hardware: 0x0B00CCCC
Read register 'r12' (4 bytes) from hardware: 0x88070020
Read register 'sp' (4 bytes) from hardware: 0x68B10120
Read register 'lr' (4 bytes) from hardware: 0x1D090000
Read register 'pc' (4 bytes) from hardware: 0xF4020000
Read register 'xpsr' (4 bytes) from hardware: 0x00000061
Removing breakpoint @ address 0x000002F4, Size = 2
Removing breakpoint @ address 0x00001884, Size = 2
Read 4 bytes @ address 0x000002F4 (Data = 0xF0002000)

Output from JLinkRTTViewer with build date of 15:02

00> 
00> adr-prototype-2307 v0.0.1
00> Built on Dec  7 2023 at 15:02:58

Output from ls -l build showing the new (but unflashed) .elf file from 15:15

rw-r--r--  1 jay  staff    15095 Dec  7 15:02 CMakeCache.txt
drwxr-xr-x  9 jay  staff      288 Dec  7 15:15 CMakeFiles
-rwxr-xr-x  1 jay  staff  4341936 Dec  7 15:15 adr-prototype-2307.elf
-rw-r--r--  1 jay  staff  2451471 Dec  7 15:15 adr-prototype-2307.map
-rw-r--r--  1 jay  staff   395418 Dec  7 15:15 build.ninja
-rw-r--r--  1 jay  staff     1527 Dec  7 15:02 cmake_install.cmake

[haneefdm]

When restarting a debug session after rebuilding my .elf file, the previous build is started by the debugger unless I clean the build folder first.

Not sure how you 'restarted'. Did you stop and start a new debug session? Or did you just use the re-start button? If it is the later we don't reload the executable. That was not a desirable feature for most users and will not change. You can however force a reload it by using preRestartCommands... but the symbols may not match any more because that is also done once at the start.

We also are not involved in how/why you build your executables.

[JayToltTech]

Restart is Stop/Start. I have confirmed that the filesystem shows new timestamps on the ELF, so it's not an 'oops I didn't actually build' or 'I built to the wrong place' issue, AFAIK. I've confirmed there isn't a second 'mystery ELF' hanging around in the file system that I can find under the build directory or the source tree. Cortex-Debug is being passed the correct full path to the ELF file which has the updated build timestamp.

The log shows that the elf is being flashed, but the code from the older elf is what is running after the Start.

[haneefdm]

We don't control that. We don't even cache ANYTHING across a Stop and Start. In fact, our debugger is killed and restarted. So, I am not sure what is wrong with your OS -- are you using a network drive?

Also, if we are reloading, where did we get the older elf from?

[JayToltTech]

It's a mystery to me too! Using a local drive on MacOS with a single disk/volume, no mystery mounts or links or such...

[JayToltTech]

The logs don't show more detail about how Cortex-Debug goes from:

"executable": "/Users/jay/Code/adr_prototype_2307/build/adr-prototype-2307.elf",

to

Resetting target Downloading 16112 bytes @ address 0x00000000 ...

Perhaps something in that path is using a stale open file handle to read bytes?!?

[JayToltTech]

So looking at the logs, it looks like Cortex-Debug passes a file path, not object bytes, to JLinkGDBServer:

Launching gdb-server: JLinkGDBServer -singlerun -nogui -if swd -port 50000 -swoport 50001 -telnetport 50002 -device ATSAME54P20A Please check TERMINAL tab (gdb-server) for output from JLinkGDBServer ... 10-file-exec-and-symbols "/Users/jay/Code/adr_prototype_2307/build/adr-prototype-2307.elf"

So perhaps this is a bug in JLinkGDBServer rather than Cortex-Debug itself?

[JayToltTech]

OK, I moved to Segger Tools V7.94a from V7.86g and the issue seems to have gone away after a few light tests...

[haneefdm]

No, it can't even be a bug in JLinkGDBServer. Something else is at play.

For instance, on a Mac, if you move a file to trash, it is not really gone. If you use the same path name, it will serve things from the trash without you even knowing it.

I don't have a clue what is exactly happening on your computer. I am going to close this because I am 99.9% sure it is not our problem.

[tim-hilt]

@JayToltTech did you ever find out why this is happening for you? Sounds a lot like a problem I have. We are using Windows though.