`v.0.2.4` Now Produces "Failed to launch OpenOCD GDB Server: Timeout"...

[busticated]

hi there :wave:

this morning i started hitting the following error while attempting to debug:

Failed to launch OpenOCD GDB Server: Timeout

i don't see anything interesting in the developer tools console. the adapter output is:

Open On-Chip Debugger 0.10.0 (2019-01-29-17:30)
Licensed under GNU GPL v2
For bug reports, read
    http://openocd.org/doc/doxygen/bugs.html
Warn : Interface already configured, ignoring
Info : auto-selecting first available session transport "swd". To override use 'transport select <transport>'.
adapter speed: 10000 kHz
cortex_m reset_config sysresetreq
Info : CMSIS-DAP: SWD  Supported
Info : CMSIS-DAP: Interface Initialised (SWD)
Info : CMSIS-DAP: FW Version = 1.10
Info : SWCLK/TCK = 1 SWDIO/TMS = 1 TDI = 0 TDO = 0 nTRST = 0 nRESET = 1
Info : CMSIS-DAP: Interface ready
Info : reduce speed request: 10000kHz to 5000kHz maximum
Info : clock speed 10000 kHz
Info : SWD DPIDR 0x2ba01477
Info : nrf52.cpu: hardware has 6 breakpoints, 4 watchpoints

but nothing seems out of order there 🤷‍♂

as of v0.2.3, this was working fine. i see v0.2.4 was released a few hours ago. if i use VSCode's "Install Another Version..." option to revert back to v0.2.3, debugging works again for me.

i'll try to update this issue w/ more technical detail as i find it :+1:

[busticated]

fwiw, running my extension within the extension development host and plunking around src/frontend/configprovider.js trying to understand what's going on, i see:

`config` returned from resolveDebugConfiguration()

```json { "type": "cortex-debug", "request": "attach", "name": "Particle Debugger (argon, boron, xenon)", "servertype": "openocd", "interface": "swd", "device": "nRF52840_xxAA", "cwd": "${workspaceRoot}", "executable": "${command:particle.getDebuggerExecutable}", "preLaunchTask": "Particle: Flash application for debug (local)", "armToolchainPath": "${command:particle.getDebuggerCompilerDir}", "svdFile": "${command:particle.getDebuggerSVDFile}", "searchDir": [ "${command:particle.getDebuggerSearchDir}" ], "configFiles": [ "interface/cmsis-dap.cfg", "target/nrf52-particle.cfg" ], "windows": { "configFiles": [ "interface\\cmsis-dap.cfg", "target\\nrf52-particle.cfg" ] }, "debuggerArgs": [], "swoConfig": { "enabled": false, "decoders": [], "cpuFrequency": 0, "swoFrequency": 0, "source": "probe" }, "graphConfig": [], "preLaunchCommands": [], "postLaunchCommands": [], "preAttachCommands": [], "postAttachCommands": [], "preRestartCommands": [], "postRestartCommands": [], "runToMain": false, "serverpath": "/Users/me/.particle/toolchains/openocd/0.10.0-particle.1/bin/openocd", "toolchainPath": "${command:particle.getDebuggerCompilerDir}", "extensionPath": "/Users/me/Sites/particle/workbench/packages/marus25.cortex-debug-0.2.4" } ```

verifyOpenOCDConfiguration() receives `config`:

```json { "type": "cortex-debug", "request": "attach", "name": "Particle Debugger (argon, boron, xenon)", "servertype": "openocd", "interface": "swd", "device": "nRF52840_xxAA", "cwd": "${workspaceRoot}", "executable": "${command:particle.getDebuggerExecutable}", "preLaunchTask": "Particle: Flash application for debug (local)", "armToolchainPath": "${command:particle.getDebuggerCompilerDir}", "svdFile": "${command:particle.getDebuggerSVDFile}", "searchDir": [ "${command:particle.getDebuggerSearchDir}" ], "configFiles": [ "interface/cmsis-dap.cfg", "target/nrf52-particle.cfg" ], "windows": { "configFiles": [ "interface\\cmsis-dap.cfg", "target\\nrf52-particle.cfg" ] }, "debuggerArgs": [], "swoConfig": { "enabled": false, "decoders": [], "cpuFrequency": 0, "swoFrequency": 0, "source": "probe" }, "graphConfig": [], "preLaunchCommands": [], "postLaunchCommands": [], "preAttachCommands": [], "postAttachCommands": [], "preRestartCommands": [], "postRestartCommands": [], "runToMain": false, "serverpath": "/Users/me/.particle/toolchains/openocd/0.10.0-particle.1/bin/openocd" } ```

i'm able to verify that my command variables - e.g. ${command:particle.getDebuggerExecutable} - (docs) are being called and returning correct values despite their unsubstituted representation appearing here. all the static values (e.g. serverpath) appear to be correct.

my `launch.json`:

```json { "version": "0.2.0", "configurations": [ { "type": "cortex-debug", "request": "attach", "name": "Particle Debugger (argon, boron, xenon)", "servertype": "openocd", "interface": "swd", "device": "nRF52840_xxAA", "cwd": "${workspaceRoot}", "executable": "${command:particle.getDebuggerExecutable}", "preLaunchTask": "Particle: Flash application for debug (local)", "armToolchainPath": "${command:particle.getDebuggerCompilerDir}", "svdFile": "${command:particle.getDebuggerSVDFile}", "searchDir": [ "${command:particle.getDebuggerSearchDir}" ], "configFiles": [ "interface/cmsis-dap.cfg", "target/nrf52-particle.cfg" ], "windows": { "configFiles": [ "interface\\cmsis-dap.cfg", "target\\nrf52-particle.cfg" ] } } ] } ```

doing the same over in src/frontend/extension.js, i noticed that debugSessionStarted() is never called. likewise, initCommands(), serverExecutable(), serverArguments(), serverLaunchStarted(), or serverLaunchCompleted() within src/openocd.js are never called.

if there's specific debug steps or info that would be helpful to try, i'd be happy to do so - just let me know :pray::+1:

[tn-5]

I'm experiencing same issue, environment = macOS. I'd be happy to provide more debug info if it would help to localise the issue.

[diegoherranz]

Same issue on Ubuntu (0.2.3 working, 0.2.4 not working). Happy to help if more info required. Thanks!

[Marus]

The version of openocd from homebrew on macOS (and likely ubuntu) is rather ancient - and doesn't include the correct output for us to detect. If you are on macOS I'd recommend installing it on using either brew install open-ocd --HEAD to get an update to date source build - or alternatively use the packages from gnu-mcu-eclipse (https://github.com/gnu-mcu-eclipse/openocd/releases)

You can likely also use the gnu-mcu-eclipse releases on ubuntu as well - as he does provide linux builds as well I expect.

[Marus]

To expand on this - to fix other possible problems with using OpenOCD which could appear depending on your particular board configs or other OpenOCD options we had to change how we detected that OpenOCD is finished launching and is ready to accept GDB connections. This change unfortunately is not backwards compatible with the very old versions of OpenOCD available in some repos.

[busticated]

i’m building from source - what version of openocd should we be using?

edit: TIL brew install —HEAD installs the latest from the upstream repo (docs)

still, a known good sha would be helpful for posterity

[john-terrell]

I have the same issue on Mac. I've installed from brew using HEAD. No change. Interestingly, if I try to use pyocd, I also get a timeout.

Using openocd, I can connect to the board as well as manually connected gdb.

The board in question is an Atmel SAM4E if that makes any difference.

[john-terrell]

Following up: before the timeout expires, I can connect to openocd using port 50000 so it looks like openocd launches fine - cortex-debug can't talk to it for some reason.

[john-terrell]

Disregard my comment above. I'd reinstalled openocd (non-HEAD) and that was failing because it appears the stdout of openocd doesn't match what Cortex-Debug is expecting in order to detect if it's running. Coincidentally after that, I'd then installed the HEAD version of openocd but at the same time the debugger on the board stopped responding so when openocd launched, it would immediately fail - Cortex Debug didn't detect the failure and instead waited until the timeout elapsed to report a failure.

[tn-5]

I can confirm that brew install open-ocd --HEAD indeed fixes the problem (macOS). @john-terrell Check that openocd actually runs from the command line and parses the openocd config file without any errors. I had to make a few changes to my openocd.cfg to get it to start up without any errors due to syntax changes (chain position, manual dap creation etc.). Once openocd starts without reporting any errors the plugin works well.

I had to add the following (also a SAM4 board):

• transport select swd before the source [find target/swj-dp.tcl] line
• dap create XXX -chain-position $_TARGETNAME before the $_TARGETNAME configure line
• Replace -chain-position $_TARGETNAME in the configure line to -dap XXX

[john-terrell]

Yep, see my previous comment. :-) I wonder if some better error handling from Cortex-Debug would help us diagnose when openocd doesn't like what's going on. :-)

[tn-5]

When running openocd manually from the command line the error feedback is quite descriptive. What is required somewhere is a document detailing the config changes from 0.10 (released) to the latest HEAD version

[john-terrell]

@tn-5- Yep, sorry...I mistyped: I should have written "better error handling from Cortex-Debug" instead of "better error handling from openocd". I've changed my comment above to reflect this.

[tn-5]

Yes, I'm assuming that Cortex-Debug fails if it detects something like "error" in the openocd output. It would be helpful to display this rather than fail with the cryptic timeout. Well, at least it works now.

[haneefdm]

There were two changes I made that went into a recent PR by me... The regular expression looking for server started...

Old:  /Info\s:\s([^\n\.]*)\.cpu([^\n]*)/i
New:  /Info\s:[^\n]*Listening on port \d+ for gdb connection/i

That is the only thing Cortex-Debug is looking for to see if the server has started... As it happens the old expression would have worked? Looks like I broke it?

The second change was the port setting is the first command and not the last command. If you enable showDevDebugOutput in launch.jason, you will see the exact command line to start the gdb-server (OpenOCD) in the Debug Console

@busticated Can we verify, it these are okay? I am not seeing the full output of OpenOCD in the first post. There is probably more?

[haneefdm]

Problem is that you can see a lot of lines that have the word '.cpu' in OpenOCD output but it doesn't actually work until you see the Listening on a tcp port message and gdb would start and not succeed.

The output for starting a gdb port has been there since 2016-12-19. So, are we using something older than that?

[haneefdm]

@tn-5 @diegoherranz @busticated Could you please post the full output of OpenOCD

[busticated]

thanks @haneefdm that’s good info. i’m traveling atm so won’t be able to check full output until next week. dumb question: where can I find that output within vscode? i posted what I found in the “adapter-output” but i guess there’s more? 🤷‍♂️👍

[haneefdm]

@busticated That is it. It is the" adapter output". What I saw above is that it detected a DAP and the cpu but still has not opened a gdb port yet. A lot can happen (and go wrong) between those two steps. Maybe very old OpenOCD never output a message about the gdb port?

[diegoherranz]

Maybe very old OpenOCD never output a message about the gdb port?

Maybe. Because I get something similar to @busticated here on Ubuntu using OpenOCD version 77189db from early 2017:

Open On-Chip Debugger 0.10.0+dev-00092-g77189db (2017-03-04-09:45)
Licensed under GNU GPL v2
For bug reports, read
    http://openocd.org/doc/doxygen/bugs.html
Info : FTDI SWD mode enabled
srst_only separate srst_gates_jtag srst_open_drain connect_deassert_srst
adapter speed: 1000 kHz
adapter_nsrst_delay: 100
srst_only separate srst_nogate srst_open_drain connect_deassert_srst
cortex_m reset_config sysresetreq
Info : clock speed 1000 kHz
Info : SWD DPIDR 0x1ba01477
Info : stm32f1x.cpu: hardware has 6 breakpoints, 4 watchpoints

I've just compiled today's HEAD (04ebb43) and it's clearly different:

Open On-Chip Debugger 0.10.0+dev-00865-g04ebb43 (2019-05-15-20:23)
Licensed under GNU GPL v2
For bug reports, read
    http://openocd.org/doc/doxygen/bugs.html
Info : FTDI SWD mode enabled
srst_only separate srst_gates_jtag srst_open_drain connect_deassert_srst

Info : Listening on port 6666 for tcl connections
Info : Listening on port 4444 for telnet connections
Info : clock speed 1000 kHz
Info : SWD DPIDR 0x1ba01477
Info : stm32f1x.cpu: hardware has 6 breakpoints, 4 watchpoints
Info : Listening on port 50000 for gdb connections
Info : accepting 'gdb' connection on tcp/50000
Info : device id = 0x20036410
Info : flash size = 64kbytes
undefined debug reason 8 - target needs reset
target halted due to debug-request, current mode: Thread 
xPSR: 0x01000000 pc: 0x08000470 msp: 0x20005000
target halted due to debug-request, current mode: Thread 
[...]

It does work now.

Slightly related: it's been years since the latest OpenOCD stable release. Does anyone know why? It's not because of the lack of development, I would say, since there have been nearly 900 commits since v0.10.0.

Thanks!

[haneefdm]

First, I want to apologize to @Marus and everyone else for screwing up your debugging experience and time. I researched it a bit more.

I don't understand OpenOCD methodology and how vendors are using it. Seems like vendors are forking some old version and making private releases, custom mods, etc. OpenOCD folks don't seem very responsive either. Found a serious bug in OpenOCD there and asked for help, provided fix, etc. and zero response. And, not publishing their latest stable to homebrew, apt-get, etc. I could be wrong.

The version number don't mean a thing either. Every OpenOCD output I saw has the same version but different behavior because of the 26 forks I suspect.

Thing is you may get the latest from OpenOCD but may not have the proper/full implementations from MCU vendors. Aaaaargh.

Again, sorry.

[haneefdm]

Oh btw, that date you see in the first line of OpenOCD output, I think it is the date it was built. Not when the date of the source code snapshot was made.

If you are building yourself, could anyone confirm my suspicion?

[diegoherranz]

Yes, it is the compilation date. In my case it was close to the commit date, but not necessarily.

[Marus]

@busticated That is it. It is the" adapter output". What I saw above is that it detected a DAP and the cpu but still has not opened a gdb port yet. A lot can happen (and go wrong) between those two steps. Maybe very old OpenOCD never output a message about the gdb port?

Yes - old versions didn't have the line about listening for a connection - which is why I had initially watched for the the /Info\s:\s([^\n\.]*)\.cpu([^\n]*)/I bit. In the default configurations (most bundled board configs I tried) of the older builds of OpenOCD that would usually indicate that it was ready to work (but there could be a bit of a race condition). Unfortunately as @haneefdm mentioned - there are a lot of OpenOCD configuration options and versions of OpenOCD floating around as the main branch isn't really maintained much if at all; all these variations can cause other lines that match that to be output and cause other problems, or fall victim to the race condition. So in this case it's something we simply have to live with and people will need to update to a newer build that has this output.

Longer term I might see if I can come up with some other possible thing to detect it being live - but I'm not hopeful because of all the variations of OpenOCD floating around. So for now I think our recommendation would be to either build from --HEAD yourself, or use the gnu-arm-eclipse builds which also seem to work.

[Marus]

I have the same issue on Mac. I've installed from brew using HEAD. No change. Interestingly, if I try to use pyocd, I also get a timeout.

Using openocd, I can connect to the board as well as manually connected gdb.

The board in question is an Atmel SAM4E if that makes any difference.

@john-terrell - The PyOCD timeout issue (caused by some updated output in PyOCD, they changed the word at to on in their output indicating the GDB server was live) should have been fixed with V0.2.5 - so that should hopefully be working again now.

[Marus]

Yes, I'm assuming that Cortex-Debug fails if it detects something like "error" in the openocd output. It would be helpful to display this rather than fail with the cryptic timeout. Well, at least it works now.

I'll look at better error detection instead of waiting for the timeout as it currently does; the main reason I hadn't tried to detect errors yet is because I've seen certain cases where it output things marked as "error" - but those weren't a fatal error and the debug process could still continue (for example things like misconfigured output options may do that). That made it hard to determine between a minor error, and a fatal one that would prevent debugging.

[haneefdm]

Little update. I emailed the OpenOCD maintainers are Cypress. Yes, most vendors are forking it creating/bundling their own versions. Vendors are also unable to put changes/fixes back into OpenOCD. It takes 6 months to year or more before anyone even looks at a PR and schedule a review. The review process itself can take months once it starts. I looked at their list of issues and PRs and it is not very pretty. Commits are happening at a furious rate (several a day) though, but no releases since approx Jan 22, 2017

And once a vendor forks, and gets the thing working for their device, they rarely go back and pull from the original master, because of the testing overhead. Kinda like the Android situation :-)

I am just trying to explain the state of things at OpenOCD. OpenOCD !== OpenOCD. If you pick stuff from the HEAD, it may or may not work for your chip.

@Marus and I are hopeful that we will find a solution that is a bit more resilient, very soon. See issue #167

[louisrubet]

Same problem with Fedora 30 and standard openocd package marked as 0.10.0-11.fc30 for x86_64

openocd --version
Open On-Chip Debugger 0.10.0
Licensed under GNU GPL v2
For bug reports, read
    http://openocd.org/doc/doxygen/bugs.html

Compiling and installing openocd from https://sourceforge.net/p/openocd/code/ci/master/tree/ can solve the problem

From tag v0.10.0 (2019-05-19 13:51) -> does not work From commit 2b78f6 (2019-05-15 16:27) -> works

btw I had to modify the standard build in order to remove -Werror because of the remaining warnings treated as errors

Thank you @busticated @haneefdm @diegoherranz and @Marus for your enlightments

[gigaj0ule]

Looks like I have this problem on the latest Ubuntu LTS release too. OpenOCD runs, exposes a port on 3333, and I can manually connect to it from GDB just fine. But it never says "Info: Listening on port 3333 for gdb connections" so cortex-debug doesn't know it's time to connect.

Edit: I rolled back to the extension version from 4 months ago and now it works perfectly.

@haneefdm Maybe we should just change the regular expression check in the latest build back to /Info\s:\s([^\n\.]*)\.cpu([^\n]*)/i until a proper fix is built?

Edit2: See this PR: https://github.com/Marus/cortex-debug/pull/172

[haneefdm]

@adammunich if we do another release, we may not need to roll back. See PR #169, and @Marus also made changes to make the regexp customizable from launch.json. I am thinking those changes will be in the next release soon.

If you are so inclined, could you try out my copy of the extension? Basically, the contents of PR #169

https://github.com/haneefdm/cortex-debug

It would help us all if people can beta test this.

[Marus]

@adammunich - if you don't want to upgrade your OpenOCD you should be able to now configure a regex through the overrideGDBServerStartedRegex launch.json parameter - something along the lines of "overrideGDBServerStartedRegex": "Info\\s:\\s([^\\n\\.]*)\\.cpu([^\\n]*)" should work (hopefully escaped that all correctly