Gdb returns stack level of -1 in some cases

[AndrewBrianHall]

I suggest we open a bug and do the fix. Treat the -1 as a zero I suppose; maybe just ignore the thread? Will have to try with the sample project to see if we can repro the behavior though.

(8240) ->1009^done,threads=[{id="10",target-id="Thread 936",frame={level="0",addr="0x40036fc0",func="ioctl",args=[],from="libc.so"},state="stopped",core="0"},{id="9",target-id="Thread 935",frame={level="0",addr="0x40036fc0",func="ioctl",args=[],from="libc.so"},state="stopped",core="0"},{id="8",target-id="Thread 931",frame={level="0",addr="0x400380e0",func="futex_syscall3",args=[],from="libc.so"},state="stopped",core="0"},{id="7",target-id="Thread 930",frame={level="0",addr="0x400380e0",func="__futex_syscall3",args=[],from="libc.so"},state="stopped",core="0"},{id="6",target-id="Thread 929",frame={level="0",addr="0x400380e0",func="futex_syscall3",args=[],from="libc.so"},state="stopped",core="0"},{id="5",target-id="Thread 928",frame={level="0",addr="0x400380e0",func="futex_syscall3",args=[],from="libc.so"},state="stopped",core="0"},{id="4",target-id="Thread 927",frame={level="0",addr="0x40037ba8",func="recvmsg",args=[],from="libc.so"},state="stopped",core="0"},{id="3",target-id="Thread 926",frame={level="0",addr="0x400379a0",func="__rt_sigtimedwait",args=[],from="libc.so"},state="stopped",core="0"},{id="2",target-id="Thread 924",frame={level="0",addr="0x400380e0",func="futex_syscall3",args=[],from="libc.so"},state="stopped",core="0"},{id="1",target-id="Thread 921",frame={level="-1",addr="0x40757610",func="??",args=[]},state="stopped",core="0"}],current-thread-id="1"

[suprstefan]

I'm pretty sure I hit this bug when trying to debug an application on a raspberry pi. It has to communicate with the camera and therefore calls bcm_host_init() which spawns four threads (according to the output tab) and if I then hit a breakpoint I get an error saying:

Error while trying to enter break state. Debugging will now stop. Unrecognized format of field "level" in result: {level=-1,addr=0xb6cc902c,func=??,args=[]}

I guess this is because I'm linking to the installed version of these libraries and they're probably stripped?

Edit I enabled logging and it indeed seems like I'm hitting this issue:

threads=[{id="5",target-id="Thread 8186",frame={level="0",addr="0xb6cd0648",func="??",args=[]},state="stopped",core="0"},{id="4",target-id="Thread 8185",frame={level="0",addr="0xb6be4a40",func="??",args=[]},state="stopped",core="0"},{id="3",target-id="Thread 8184",frame={level="0",addr="0xb6be4a40",func="??",args=[]},state="stopped",core="0"},{id="2",target-id="Thread 8183",frame={level="-1",addr="0xb6cc902c",func="??",args=[]},state="stopped",core="0"},{id="1",target-id="Thread 8180",frame={level="0",addr="0x000119b0",func="CCamera::Init",args=[{name="this",value="0x273c8"},{name="width",value="512"},{name="height",value="512"},{name="framerate",value="30"},{name="num_levels",value="1"},{name="do_argb_conversion",value="true"}],file="/home/pi/projects/test01/camera.cpp",fullname="/home/pi/projects/test01/camera.cpp",line="257"},state="stopped",core="0"}],current-thread-id="1"

Here's a longer log: milog.txt

[ghost]

Is there a known fix/cause for this yet? I'm having the same issue with Cygwin GDB -> gdbserver -> JLink -> ARM Cortex-M3. My application is being flashed to external flash, and the address below is the beginning of ext-flash, so that's incorrect as well.

1: (33999) ->1016^done,threads=[{id="1",target-id="Thread 57005",frame={level="-1",addr="0x1c004000",func="myClass::myFunc",args=[{name="this",value="0x0"}],file="myClass.cpp",fullname="/cygdrive/c/workspace/myClass.cpp",line="1515"},state="stopped"}],current-thread-id="1" ... ERROR: Unable to start debugging. Unrecognized format of field "level" in result: {level=-1,addr=0x1c004000,func=fmyClass::myFunc,args=[{name=this,value=0x0}],file=myClass.cpp,fullname=/cygdrive/c/workspace/myClass.cpp,line=1515}

[ihalip]

I also have this same issue. I'm trying to remote debug on an arm device with this configuration:

{ 
    "name": "(gdb) Attach to process",
    "type": "cppdbg",
    "request": "launch",
    "program": "${workspaceFolder}/some/binary",
    "cwd": ".",
    "MIMode": "gdb",
    "miDebuggerPath": "/usr/bin/arm-none-eabi-gdb",
    "targetArchitecture": "arm",
    "setupCommands": [
        { "text": "target remote 192.168.1.150:9000" }
    ]
}

The gdb process is terminated after running into the "level" value, for which I think it expects to find an UInt (see DebuggedThread.cs#L317):

Copyright (C) 2015 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "--host=x86_64-linux-gnu --target=arm-none-eabi".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word".
=cmd-param-changed,param="pagination",value="off"
1: (1074) <-1045-interpreter-exec console "info sharedlibrary"
1: (1074) ->~"No shared libraries loaded at this time.\n"
1: (1074) ->1045^done
1: (1074) ->(gdb)
1: (1075) 1045: elapsed time 0
E output: {"category":"console","output":"Loaded 'shared libraries loaded at this time.'. Cannot find or open the symbol file.\n","data":null,"type":"output"}
Loaded 'shared libraries loaded at this time.'. Cannot find or open the symbol file.
1: (1084) <-1046-thread-info
1: (1162) ->1046^done,threads=[{id="36",target-id="Thread 5122",frame={level="-1",addr="0xb3474728",func="??",args=[]},state="stopped",core="0"},{id="35",target-id="Thread 5121",frame={level="-1",addr="0xb3474728",func="??",args=[]},state="stopped",core="0"},{id="34",target-id="Thread 5070",frame={level="0",addr="0xb3474728",func="??",args=[]},state="stopped",core="0"},{id="33",target-id="Thread 5069",frame={level="0",addr="0xb3474728",func="??",args=[]},state="stopped",core="0"},{id="32",target-id="Thread 5068",frame={level="0",addr="0xb3474ac4",func="??",args=[]},state="stopped",core="0"},{id="31",target-id="Thread 5067",frame={level="0",addr="0xb3474ac4",func="??",args=[]},state="stopped",core="0"},{id="30",target-id="Thread 5066",frame={level="-1",addr="0xb3474728",func="??",args=[]},state="stopped",core="0"},{id="29",target-id="Thread 5065",frame={level="0",addr="0xb3474728",func="??",args=[]},state="stopped",core="0"},{id="28",target-id="Thread 5064",frame={level="0",addr="0xb3474ac4",func="??",args=[]},state="stopped",core="0"},{id="27",target-id="Thread 5063",frame={level="-1",addr="0xb3474728",func="??",args=[]},state="stopped",core="0"},{id="26",target-id="Thread 5062",frame={level="0",addr="0xb3474728",func="??",args=[]},state="stopped",core="0"},{id="25",target-id="Thread 5061",frame={level="0",addr="0xb2e93104",func="??",args=[]},state="stopped",core="0"},{id="24",target-id="Thread 5060",frame={level="0",addr="0xb2e93104",func="??",args=[]},state="stopped",core="0"},{id="23",target-id="Thread 5059",frame={level="0",addr="0xb3474728",func="??",args=[]},state="stopped",core="0"},{id="22",target-id="Thread 5058",frame={level="0",addr="0xb3474728",func="??",args=[]},state="stopped",core="0"},{id="21",target-id="Thread 5057",frame={level="0",addr="0xb2e93104",func="??",args=[]},state="stopped",core="0"},{id="20",target-id="Thread 5056",frame={level="0",addr="0xb3474ac4",func="??",args=[]},state="stopped",core="0"},{id="19",target-id="Thread 5055",frame={level="0",addr="0xb3474728",func="??",args=[]},state="stopped",core="0"},{id="18",target-id="Thread 5046",frame={level="0",addr="0xb2e8bfc4",func="??",args=[]},state="stopped",core="0"},{id="17",target-id="Thread 5044",frame={level="0",addr="0xb2e93104",func="??",args=[]},state="stopped",core="0"},{id="16",target-id="Thread 5043",frame={level="-1",addr="0xb3478d60",func="??",args=[]},state="stopped",core="0"},{id="15",target-id="Thread 5038",frame={level="0",addr="0xb2e89e10",func="??",args=[]},state="stopped",core="0"},{id="14",target-id="Thread 5037",frame={level="0",addr="0xb2e93104",func="??",args=[]},state="stopped",core="0"},{id="13",target-id="Thread 5033",frame={level="-1",addr="0xb2e61c70",func="??",args=[]},state="stopped",core="0"},{id="12",target-id="Thread 5032",frame={level="0",addr="0xb2e8bfc4",func="??",args=[]},state="stopped",core="0"},{id="11",target-id="Thread 5030",frame={level="0",addr="0xb2e8bfc4",func="??",args=[]},state="stopped",core="0"},{id="10",target-id="Thread 5029",frame={level="0",addr="0xb2e8bfc4",func="??",args=[]},state="stopped",core="0"},{id="9",target-id="Thread 5028",frame={level="0",addr="0xb3474ac4",func="??",args=[]},state="stopped",core="0"},{id="8",target-id="Thread 5027",frame={level="0",addr="0xb3474ac4",func="??",args=[]},state="stopped",core="0"},{id="7",target-id="Thread 5026",frame={level="0",addr="0xb2e89e10",func="??",args=[]},state="stopped",core="0"},{id="6",target-id="Thread 5025",frame={level="0",addr="0xb3474ac4",func="??",args=[]},state="stopped",core="0"},{id="5",target-id="Thread 5024",frame={level="0",addr="0xb2e8b66c",func="??",args=[]},state="stopped",core="1"},{id="4",target-id="Thread 4999",frame={level="0",addr="0xb2e8bfc4",func="??",args=[]},state="stopped",core="0"},{id="3",target-id="Thread 4998",frame={level="0",addr="0xb2e8bfc4",func="??",args=[]},state="stopped",core="1"},{id="2",target-id="Thread 4997",frame={level="0",addr="0xb2e8bfc4",func="??",args=[]},state="stopped",core="1"},{id="1",target-id="Thread 4879",frame={level="-1",addr="0xb3471194",func="??",args=[]},state="stopped",core="1"}],current-thread-id="1"
1: (1162) ->(gdb)
1: (1163) 1046: elapsed time 79
E output: {"category":"stderr","output":"ERROR: Unable to start debugging. Unrecognized format of field \"level\" in result: {level=-1,addr=0xb3474728,func=??,args=[]}\n","data":null,"type":"output"}
ERROR: Unable to start debugging. Unrecognized format of field "level" in result: {level=-1,addr=0xb3474728,func=??,args=[]}
1: (1175) <--gdb-exit
1: (1175) ->^exit
1: (1181) ->=thread-group-exited,id="i1"
E output: {"category":"console","output":"The program '[REMOVED]' has exited with code 42 (0x0000002a).\r\n\n","data":null,"type":"output"}
The program '[REMOVED]' has exited with code 42 (0x0000002a).

[ihalip]

I can repro this 100% of the time with my scenario.

I recompiled Microsoft.MIDebugEngine.dll (with dnSpy just because the Visual Studio recompiled output wouldn't run in Linux) using this change and I was able to successfully start a remote debugging session.

[Hao-Yan]

I also can repro it 100% from my side with latest vscode on linux. When will this be fixed?

[agrieve]

Hitting this as well.

[pieandcakes]

@Hao-Yan, @agrieve can either of you provide me a consistent repro or can I give you private bits to try? I'm trying to understand what frame="-1" signifies.

[agrieve]

@pieandcakes - I'm hitting when trying to debug Chrome on a Nexus 5 KitKat device. Might be pretty painful for you to checkout & build chrome and find a physical KitKat device to run it on. Does not happen for the x86 emulator, nor does it happen on my Android O device. I'm happy to try out fixes if you want to send me things to try.

[agrieve]

Bump - is there a way to patch this locally to have -1 map to 0 to see if this causes reasonable behavior?

[pieandcakes]

I think we are trying to understand what a framelevel of -1 is supposed to mean and how variables would resolved when it is -1. We are looking to get a repro so we can do some testing to figure out the best way to fix the problem. We are worried that forcing it to be 0 can cause incorrect or strange behavior. Unfortunately, we haven't gotten a consistent repro to test this on.

From what I was told is that this can happen if the gdb <--> gdbserver architectures are mismatched and the correct way to fix it is to make sure that the local gdb is cross compiled to support the same version of gdbserver running on the remote.

[axu217]

Bump - Local + server gdb architectures are both the same yet still getting frame level -1.

[todi1856]

Bump, also hitting this

[sunyz]

also meet. Running in command line can run nromally, but in MI will "level = -1" and exit

[todi1856]

Here's a very good description what happens - https://github.com/brendanzagaeski/gists/wiki/0001 it actually helped me to find a workaround for this issue.

In my case I was running ARM7 application on ARM64 device (for ex., NVidia Shield Console), Brendan Zagaeski (from the link above) gave a pretty good hint, by matching application's CPU architecture you can workaround this.

So in my case, I switched to ARM64 in my application and ran it again on ARM64 device, and debugging started working fine

[todi1856]

P.S here's the repro showing the problem https://oc.unity3d.com/index.php/s/klWoLc9PFPR9cUs if anyone cares

[brendanzagaeski]

In case it might be useful additional info to add here, I revisited the behavior of armeabi-v7a Android apps on arm64-v8a devices yesterday and noticed that I was able to get the expected output from -thread-info (without any level="-1" items) if I ran the 32-bit gdbserver rather than the 64-bit gdbserver. (This time I was testing on a Google Pixel 3 device running Android 9 Pie (API level 28). In the previous testing that todi1856 mentioned, I hadn't been able to use the 32-bit gdbserver because it wouldn't run on the arm64-v8a Android 7.1 Nougat (API level 25) emulator I was using back then. I'm not sure if that limitation is specific to the emulators or also affects arm64-v8a hardware devices running the older Android 7.1 Nougat. But the good news is it looks like the 32-bit gdbserver works as desired on at least some hardware. So running MIEngine with TargetArchitecture.ARM for armeabi-v7a apps should work correctly on those devices.)

I also tried some quick tests with lldb-server and lldb on the command line. In those brief tests, both the 32-bit and 64-bit lldb-server produced good stack backtraces for armeabi-v7a apps running on a Google Pixel 3.