Eclipse "terminate button" halts target with the latest gcc-arm-none-eabi-6-2017-q1-update

[bakerngan]

Steps to reproduce: 1) start debug session with J-Link 2) run the program 3) press the red "terminate" button 4) program stays halted

Not sure if this is expected behavior, but in gcc-arm-none-eabi-5_4-2016q3 of the toolchain (everything else the same) if I start the debug session, then press terminate button, the debug session will stop, but the program will continue.

Info: Plug in version: plug in version GNU ARM C/C++ J-Link Debugging 4.1.3.201702251311

Eclipse Version: Version: Neon.3 Release (4.6.3) Build id: 20170314-1500

Java version: java version "1.8.0_121" Java(TM) SE Runtime Environment (build 1.8.0_121-b13) Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)

OS: macOS 10.12.4

Toolchain: gcc-arm-none-eabi-6-2017-q1-update

[ilg-ul]

press the red "terminate" button

which of the many red buttons? each process has it's own, each console, etc.

please be more specific, you need to mention which button and which process/console is active.

I don't know now what is the expected behaviour, there were many problems when transitioning from GDB 7.10 to GDB 7.12; for a reference, please run the same sequence of operations with the GDB Hardware Debugging plug-in.

[bakerngan]

Console output: SEGGER J-Link GDB Server V6.14c - Terminal output channel Connection closed by the GDB server.

For reference the exact configuration the same, using gcc-arm-none-eabi-5_4-2016q3 does not seem to cause the the program to halt after the debug session terminates.

[ilg-ul]

you mean the red button from the toolbar? ok, but that button refers to one of the selections in the debug window; what entry was selected there? the application thread or one of the other processes? (I know it is very confusing, but the red button coupled with the selection is equivalent with terminating that process with kill or equivalent).

you can also capture the gdb trace console in both cases and compare them; on Neon, the gdb trace console must be explicitly activated in preferences.

[bakerngan]

So I compared both, terminating at Thread #1 see screenshots:

Using gcc-arm-none-eabi-5_4-2016q3:

Using gcc-arm-none-eabi-6-2017-q1-update:

I found a workaround gcc-arm-none-eabi-6-2017-q1-update, if I press terminate in the context of JLinkGDBServer ( not Thread #1) then the program does not halt after termination.

I'm not sure what is the "correct" behavior but they are different as can be seen (returns different exit value)

[ilg-ul]

if I press terminate in the context of JLinkGDBServer ( not Thread #1) then the program does not halt after termination

as I already said, pressing terminate in the context of JLinkGDBServer simply sends the terminate signal to the JLinkGDBServer. similarly pressing terminate in the context of GSB sends the terminate signal to the GDB process.

each time the termination of one process drags the other processes down.

it is very hard to diagnose such a problem, where multiple processes terminate in a more or less orderly way.

[ilg-ul]

I did some tests with a friend, and the problems seems to be somewhere between GDB and J-Link GDB server, for unknown reasons the GDB server decides to reset the target.

I informed SEGGER, maybe they have an explanation for the unwanted reset.

For the moment it seems there is nothing to do in the plug-ins to prevent this.

As a workaround, I confirm that explicitly terminating the J-Link GDB Server does not reset the target.

[bakerngan]

Thanks for your response,

To be clear, when you talk about the "unwanted reset" do you mean for gcc-arm-none-eabi-6-2017-q1-update or gcc-arm-none-eabi-5_4-2016q3 (or older for that matter). Did you mean halt or reset?

For gcc-arm-none-eabi-6-2017-q1-update it seems to actually HALT the program when terminating on the Thread and for gcc-arm-none-eabi-5_4-2016q3 (and older) it seems to just continue the program (not reset) after terminating on the thread.

And yes, both continue when terminating the GDB sever.

For context I am running the latest JLink software/firmware V6.14c

[ilg-ul]

if you compare two sessions, one with gdb 7.10 an one with 7.12, you'll see in the J-Link output that for 7.12 the J-Link GDB Server, instead of gracefully shutting down, it issues a "Reset target CPU...", which freezes the device:

7.10

Starting target CPU...
Debugger requested to halt target...
...Target halted (PC = 0x080098EE)
Reading all registers
Read 4 bytes @ address 0x080098EE (Data = 0x3720BF00)
GDB closed TCP/IP connection
Restoring target state and closing J-Link connection...
Shutting down...

7.12

Starting target CPU...
...Breakpoint reached @ address 0x08009EBA
Reading all registers
Read 4 bytes @ address 0x08009EBA (Data = 0xF0034834)
Removing breakpoint @ address 0x08009EBA, Size = 2
Reading 64 bytes @ address 0x2004FF80
Read 4 bytes @ address 0x0800052C (Data = 0x6B786378)
Reset target CPU...

it is quite easy to make a small change in the setup so you can use both gdb versions:

• go to the gcc 6 bin folder and rename arm-none-eabi-gdb to arm-none-eabi-gdb-7.12
• make a symlink from gcc 5 arm-none-eabi-gdb to gcc 6 arm-none-eabi-gdb

when you need to use gdb 7.12, edit the GDB client executable field in the debug configuration and add -7.12 at the end

[bakerngan]

Great observation with the reset issue. Hopefully the SEGGER folks will have a fix for this.

[ilg-ul]

for completeness, I attached full gdb trace and j-link logs. gdb-7.10-7.12.zip

[ilg-ul]

From Segger Team:

We are pleased to announce the availability of a new version of the J-Link software and documentation pack V6.14f with the following changes: ...

• GDBServer: When using GDB Server with GDB V7.12 and later versions, unintentional resets were issued (problem caused by change inside GDB, not GDB Server). Fixed.

The software can be downloaded from the following location: http://www.segger.com/jlink-software.html

@bakerngan, could you try again with V6.14f?

[bakerngan]

I tried with the latest segger and the behavior appears to be the same. (still resetting with the same exit value as shown above). Doesn't appear to be fixed.

[SEGGER-Alex]

@bakerngan Could you please let me know your setup? I am the one at SEGGER who implemented the fix and I verified it with the GDB V7.12 command line version, no Eclipse etc. involved. The issue was reproducible before the fix and gone after the fix was applied (as expected).

[ilg-ul]

@SEGGER-Alex, I guess the setup is the same as explained in the first part of this issue.

[bakerngan]

Yes, setup is the same. I compiled with arm-none-eabi-6-2017-q1 and using the gdb from the same toolchain.

This is on macOS 10.12.4 with eclipse + gdb.

Additional settings that might be helpful that configures gdb from eclipse:

[SEGGER-Alex]

@bakerngan I have setup everything under Mac to give it a try.

Result: Works perfect.... V6.14e when hitting the red "Terminate" button:

V6.14f when hitting the red "Terminate" button:

So it is behaving exactly as expected. Please find below my debug settings. Note that I have 4.1.4.xxxxx of the plugin, as this is the version I got when setting up the environment today. Without a reproduction scenario, there is not much I can do here...

[bakerngan]

Hm this might be a different issue all together then. With the 2 different GDB versions, I still get different exit values from Eclipse (see screen shot above).

[ilg-ul]

@SEGGER-Alex, I'm afraid the problem was not completely solved.

with GDB 7.10 the GDB server seems to terminate gracefully, and the target state is restored:

SEGGER J-Link GDB Server V6.14g Command Line Version

JLinkARM.dll V6.14g (DLL compiled May  9 2017 17:19:32)

...
Starting target CPU...
Debugger requested to halt target...
...Target halted (PC = 0x080013EA)
Reading all registers
Read 4 bytes @ address 0x080013EA (Data = 0x2B00681B)
Read 4 bytes @ address 0x080015C2 (Data = 0x46313601)
GDB closed TCP/IP connection
Restoring target state and closing J-Link connection...
Shutting down...

with GDB 7.12 the target remains halted:

SEGGER J-Link GDB Server V6.14g Command Line Version

JLinkARM.dll V6.14g (DLL compiled May  9 2017 17:19:32)

...
Starting target CPU...
Debugger requested to halt target...
...Target halted (PC = 0x080013EE)
Reading all registers
Read 4 bytes @ address 0x080013EE (Data = 0x4770D1FB)
Read 4 bytes @ address 0x080015D8 (Data = 0x46313601)
GDB closed TCP/IP connection

[SEGGER-Alex]

@ilg-ul I am confused... The part after "GDB closed TCP/IP connection is absolutely unrelated to the GDB version... In both cases, the last thing GDB did was halting the CPU + reading the CPU registers (this happens with 7.10 and 7.12. None of them starts the CPU as last action). After that, it disconnected and the rest is up to J-Link GDB Server which should behave exactly the same regarding "Restoring target state and closing J-Link connection" as soon as GDB Server is closed gracefully. Is it possible that you kill the GDB Server process in the latter case?

[ilg-ul]

Is it possible that you kill the GDB Server process in the latter case?

I ran the debug session again, terminating it with the top red button while the debugged thread was selected; this should terminate the session normally.

[mstoilov]

What is the difference between Red button (Stop) and Red button (Terminate). I would assume one should stop and hold the device in reset state, the other one should close the debug session, but let the device run. Right now they both behave exactly the same way, i.e. let the device continue to run. Here is the output:

SEGGER J-Link GDB Server V6.44e Command Line Version

JLinkARM.dll V6.44e (DLL compiled Apr 5 2019 16:16:13)

Command line: -if swd -device STM32F745VE -endian little -speed 1000 -port 2331 -swoport 2332 -telnetport 2333 -vd -ir -localhostonly 1 -singlerun -strict -timeout 0

---------> Red button (Terminate)

Reading all registers Starting target CPU... Debugger requested to halt target... ...Target halted (PC = 0x0800B434) Reading all registers Read 4 bytes @ address 0x0800B434 (Data = 0xFF84F7FF) Reading 64 bytes @ address 0x200063C0 Read 4 bytes @ address 0x0800A678 (Data = 0x2300B082) GDB closed TCP/IP connection Restoring target state and closing J-Link connection... Shutting down...

SEGGER J-Link GDB Server V6.44e Command Line Version

JLinkARM.dll V6.44e (DLL compiled Apr 5 2019 16:16:13)

Command line: -if swd -device STM32F745VE -endian little -speed 1000 -port 2331 -swoport 2332 -telnetport 2333 -vd -ir -localhostonly 1 -singlerun -strict -timeout 0

---------> Red button (Stop)

Reading all registers Starting target CPU... Debugger requested to halt target... ...Target halted (PC = 0x0800B43C) Reading all registers Read 4 bytes @ address 0x0800B43C (Data = 0xD9F72B01) Reading 64 bytes @ address 0x200063C0 Read 4 bytes @ address 0x0800A678 (Data = 0x2300B082) GDB closed TCP/IP connection Restoring target state and closing J-Link connection... Shutting down...

[ilg-ul]

What is the difference between Red button (Stop) and Red button (Terminate)

I would say that the difference is only in the inconsistent naming convention.

The Stop button in the new toolbar was probably inspired by the Arduino environment, the Terminate button is from the traditional Eclipse debug plug-ins.

I'm not sure about the new button, but the old one was intended to terminate the selected process (the debugged target, the gdb client, the gdb server, etc). So the behaviour is different if you select different processes and then push the Terminate button.

[mstoilov]

Sorry for the confusion, but what else than the "Thread 1" on the side pane in Debug View can I select? I tried selecting all the items in the hierarchy and it always results in debug session terminated, but the device continues to run. This behavior is really useful, but sometimes you might need to reset the target. Is that possible?

[ilg-ul]

the behaviour is similar, but not identical, the order of killing processes is different. the red button sends the 'terminate' signal to the selected process, which intercepts it and a controlled shutdown is expected, including dependencies.

for example if you kill the gdb server, the client will also die, and the target will follow soon.

if you want to also reset the target, you need to check the j-link documentation, perhaps there is an option to set.

[baba-zul]

@SEGGER-Alex I am having the same problem that @ilg-ul mentions here. Neither the red button on top left of the Debug/Run drop down menu nor the red button on the controls that stops steps over etc.. works on my case. All the processes are getting terminated except Semihosting and SWV. Terminating the thread first helps sometimes but not always on my end.

I tried starting the eclipse with console and query the threads. Although inititally being there, after this incidence, I cannot find the Semihosting and SWV there. That makes me suspect that, it is terminated but its state is not catched. This prevents restarting the debugging session. Restarting eclipse solves the problem but incredibly annoying.

[gemschl]

@ilg-ul: is this issue in progress?

We are using eclipse 2018-12 with cdt v9.6.0 and the gnu mcu plugin v4.7.2 running on Windows OS. We see equal behavior as reported in this issue. Using the arm-none-eabi-gdb and JLinkGDBServerCL.exe in console commands closing the debug session does not halt the target, it works. But using it within Eclipse it is as described in this issue

What we have found out with log files so far is, that with use in console the gdb server from segger reports the following after TCP connect has closed/terminated by GDB: 00-0000043C-00-00184659-000E: $vKill;a410#33 01-0000043C-00-00184659-0004: $#00 00-0000043C-00-00184659-0005: $k#6b 03-00000000-00-00184659-002B: GDB closed TCP/IP connection (Socket 1084)

03-00000000-00-00184834-0038: Restoring target state and closing J-Link connection...

02-00000000-00-00184837-0020: T0898 184:847.162 JLINK_IsOpen()
02-00000000-00-00184837-0028: T0898 184:847.239 - 0.078ms returns 0x01
02-00000000-00-00184840-001F: T0898 184:850.104 JLINK_Close()
02-00000000-00-00184840-0035: T0898 184:850.657   CPU_ReadMem(4 bytes @ 0xE0001000)
03-00000000-00-00184894-0011: Shutting down...

The log in case it is used within Eclipse is: 00-00000438-00-00018804-000E: $vKill;a410#33 01-00000438-00-00018804-0004: $#00 00-00000438-00-00018805-0005: $k#6b 03-00000000-00-00018805-002B: GDB closed TCP/IP connection (Socket 1080)

Could it be, that Eclipse is killing / terminating the JLinkGDBServerCL.exe rather waiting that it has stopped?

[TommyMurphyTM1234]

Could it be, that Eclipse is killing / terminating the JLinkGDBServerCL.exe rather waiting that it has stopped?

I don't know if it's the cause of your problem but Windows CDT's starter.exe is definitely very, and arguably excessively, aggressive in how it terminates processes that it previously started. (And obviously this is not a GNU MCU Eclipse/Eclipse Embedded CDT plugin issue).

• https://git.eclipse.org/c/cdt/org.eclipse.cdt.git/tree/core/org.eclipse.cdt.core.win32/library/starter/starter.cpp?h=cdt_9_6

It basically sends the equivalent of a Ctrl-C to the process (and sometimes multiple Ctrl-C's) and more or less immediately calls TerminateJobObject() on it without giving it much time to do a graceful cleanup and exit. In many cases this does not matter but in some it will.

I had to make a local ugly hack to it to be less aggressive when terminating openocd.exe specifically because our version needed time to clean up and terminate gracefully. I have attached this but, to date, never had a chance to clean it up, generalize it and submit a pull request to see if I could get it upstreamed.

• starter.cpp.txt

Note: I had to rename it from starter.cpp to starter.cpp.txt to attach it. See the "MICROSEMI" comments/#ifdef's.

Hope this helps.

[ilg-ul]

This issue was not closed, but things are quite confusing and I don't know if there is anything to do.

[TommyMurphyTM1234]

@ilg-ul - if the issue is down to how Windows CDT's starter.exe (too) agressively terminates processes then it's a CDT and not a GNU MCU Eclipse/Eclipse Embedded CDT plugin issue. One way to test this theory is to apply my hack but extend it to the relevant J-Link executable (JLinkGDBServerCL.exe?) and see if that resolves the issue. Maybe @gemschl could try that? It does involve changing the starter.cpp code, compiling the modified version of starter.exe and then dropping that version into the CDT installation.

[jonahgraham]

@TommyMurphyTM1234 Can you submit this change to CDT (under an ECA)- I think this should be more universal and be part of CDT for everyone. As I have been editing native parts of CDT I (and others) have been removing JNI and other native parts and rewriting in JNA. This seems like a good next candidate.

[ilg-ul]

The GDB servers are started by the debug plugins, together with monitor threads to detect termination, and the entire logic is in Java.

I have no idea how the plugins behaviour can be influenced by startup.exe.

[jonahgraham]

On Windows when processes are launched it is not direct, the starter.exe is the middle man. Primary reason is so that signals can be sent to the processes.

[TommyMurphyTM1234]

@TommyMurphyTM1234 Can you submit this change to CDT (under an ECA)- I think this should be more universal and be part of CDT for everyone. As I have been editing native parts of CDT I (and others) have been removing JNI and other native parts and rewriting in JNA. This seems like a good next candidate.

Hi Jonah Yes - I agree and that is my intention but I would need to generalize it (e.g. it's specific to openocd.exe in my hack because I was being cautious) and clean it up (e.g. I was never really sure about the wide char/Unicode string parsing but just did what I had to do to get something functional for my own purposes. I will see if I can do something on this in the next week or so.

As I mentioned I suspect that most processes have no problems with being terminated so aggressively but some that require "some time" to clean up and terminate gracefully may end up being terminated "too soon" by existing starter.exe.

@gemschl - if you want to try this then change this line:

• sg_target_process.f_is_openocd = (wcsstr(argv[8], L"openocd") != NULL);

to

• sg_target_process.f_is_openocd = (wcsstr(argv[8], L"JLinkGDBServerCL.exe") != NULL);

(I guess?) and then compile the changes as follows (I used TDM-GCC for Windows):

all:    clean starter.exe

clean:  
    rm -rf starter.exe starter.o

starter.exe:    starter.o
    g++ -m64 -static starter.o -o starter.exe -lpsapi
    strip starter.exe

starter.o:  starter.cpp
    g++ -m64 -DMICROSEMI -DUNICODE -c -municode starter.cpp

And then drop the modified starter.exe into your CDT installation:

E.g. (adjust the path as needed)

• .../eclipse/plugins/org.eclipse.cdt.core.win32.x86_64_5.4.2.201811241055/os/win32/x86_64/starter.exe

[TommyMurphyTM1234]

On Windows when processes are launched it is not direct, the starter.exe is the middle man. Primary reason is so that signals can be sent to the processes.

Yes - look at Process Explorer or similar when a debug session is running and you will see various "intermediary" starter.exe processes.

Most likely this issue here should be closed and a separate one opened against CDT for the starter.exe process termination issue.

But I was and still am not sure that the starter.exe process termination issue is the root of the original issue posted here?

[TommyMurphyTM1234]

BTW @jonahgraham - I suspect that part of the cleanup and generalization should be to

(a) apply the change to all processes - not just openocd.exe as in my hack, and

(b) ensure that only one Ctrl-C is ever sent (c) maybe do a WaitForSingleObject() on the process for a judicious amount of time (whatever that might be!) and only if it does not terminate as a result of the Ctrl-C resort to more extreme measures such as TerminateJobObject()?

For now I guess that the way forward is to (a) log an issue against CDT about this starter.exe process termination issue and (b) submit a (possibly strawman) PR/change that can be discssed/teased out over on the CDT project rather than here?

Regards Tommy

[jonahgraham]

For now I guess that the way forward is to (a) log an issue against CDT about this starter.exe process termination issue and (b) submit a (possibly strawman) PR/change that can be discssed/teased out over on the CDT project rather than here?

Yes on both counts.

My inclination is to rewrite in JNA and get rid of starter.exe altogether, but lets discuss that in CDT land.

For now I am going to close this issue, once a bug is submitted in CDT it would be nice to add a link to it here so that @gemschl and others know where to continue the conversation.

[TommyMurphyTM1234]

Thanks @jonahgraham - I will start by opening a CDT issue to outline the issue and kick off the discussion in the right place. :-)

[gemschl]

Thanks @TommyMurphyTM1234 for your proposal. With slight modification this has worked for me with the JLinkGDBServer on Windows OS. The modified file, based on your proposal is: starter.cpp.txt

Is the CDT issue created?

[TommyMurphyTM1234]

Thanks @TommyMurphyTM1234 for your proposal. With slight modification this has worked for me with the JLinkGDBServer on Windows OS. The modified file, based on your proposal is: starter.cpp.txt

Is the CDT issue created?

Thanks @gemschl for trying my hack and confirming that it also works for you. I have logged the issue here:

• https://bugs.eclipse.org/bugs/show_bug.cgi?id=574131

@jonahgraham - if I am missing any key info please let me know.

[LorandMarton]

I also had this problem. I will post my solution, maybe it will help someone. Created a config for OpenOCD:

# gdb_openocd_config.cfg (in your project root dir)

$_TARGETNAME configure -event gdb-detach {
    resume
}

Added as Debugger/Config options: -f gdb_openocd_config.cfg

[TommyMurphyTM1234]

I also had this problem. I will post my solution, maybe it will help someone. Created a config for OpenOCD:

# gdb_openocd_config.cfg (in your project root dir)

$_TARGETNAME configure -event gdb-detach {
    resume
}

Added as Debugger/Config options: -f gdb_openocd_config.cfg

If the original issue was related to how the CDT for Windows starter.exe was "too aggressive" in how it terminated processes then your suggestion isn't really a fix for the root cause of the problem. See the previous detailed discussion for more on the specifics. However, at least some of this may be moot if, as I suspect, significant changes have been made to starter.exe in the intervening years - in particular fixes/changes to how starter.exe handles process termination.