Stuck at Breakpoints: netcoredbg ARM Debugging Issue with Stepping

[ShkumbinMNokia]

It appears that I'm encountering an issue with netcoredbg while attempting to debug my ARM target. I've successfully managed to set breakpoints, and they are being triggered as expected. However, I'm facing a peculiar problem when I try to step through the code after hitting a breakpoint. Instead of progressing line by line as I would expect, the debugger seems to get stuck on a single line. It's worth mentioning that the breakpoints seem to be recognized only from the file on my virtual machine, and not from the corresponding file on the target.

This issue could potentially be related to how netcoredbg handles source mapping between my development environment (virtual machine) and the target environment.

[ShkumbinMNokia]

Here is the output:

stopped, reason: interrupted, thread id: 24350, stopped threads: all, frame={LoopNumbers.Program.LoopNN() at /home/smorina/Desktop/LoopNumbersNew/LoopNumbersNew/Program.cs:20} ncdb> b /home/smorina/Desktop/LoopNumbersNew/LoopNumbersNew/Program.cs:19 Breakpoint 1 at /home/smorina/Desktop/LoopNumbersNew/LoopNumbersNew/Program.cs:19 ncdb> info break

Enb Rslvd Hits Source/Function

---

1    y  y      0          /home/smorina/Desktop/LoopNumbersNew/LoopNumbersNew/Program.cs:19

ncdb> s ^running ncdb> stopped, reason: breakpoint 1 hit, thread id: 24350, stopped threads: all, times= 1, frame={LoopNumbers.Program.LoopNN() at /home/smorina/Desktop/LoopNumbersNew/LoopNumbersNew/Program.cs:19} ncdb> s ^running ncdb> stopped, reason: breakpoint 1 hit, thread id: 24350, stopped threads: all, times= 2, frame={LoopNumbers.Program.LoopNN() at /home/smorina/Desktop/LoopNumbersNew/LoopNumbersNew/Program.cs:19} ncdb> s ^running ncdb> stopped, reason: breakpoint 1 hit, thread id: 24350, stopped threads: all, times= 3, frame={LoopNumbers.Program.LoopNN() at /home/smorina/Desktop/LoopNumbersNew/LoopNumbersNew/Program.cs:19} ncdb> tar -tf /gitworkspace/distributions/dotnet/runtime/aspnetcore-runtime-6.0.5-linux-armel.tar.gz | grep libdbgshim.so

[gbalykov]

Can you share the contents of Program.cs too?

[ShkumbinMNokia]

namespace LoopNumbers { internal class Program { static void Main(string[] args) { Console.WriteLine("Hello, World!"); LoopNN(); }

    static void LoopNN()
    {
        while (true)
        {
            for (int i = 0; i < 10; i++)
            {
                for (int e = 0; e < 10; e++)
                {
                    Console.Write($"{i}{e} ");
                    Thread.Sleep(500);
                }
                Console.WriteLine();
            }
        }
    }
}

}

[viewizard]

Wait, you set breakpoint inside cycle, right (while (true))? Is not this mean it works as should and stop on each cycle?

[viewizard]

Could you please clarify, where is line 19 and 20 in your code?

[ShkumbinMNokia]

Im setting the breakpoint on Console.Write($"{i}{e} "). I've tried it on my VM and it works there perfectly, but on the ARM Target it doesnt go any steps further. Line 19 Console.Write($"{i}{e} "); Line 20 Thread.Sleep(500);

[viewizard]

Can't reproduce this issue on ARM 32bit board with CoreCLR 6.0 rutime, program build with .NET 6.0 SDK:

using System;
using System.Threading.Tasks;
using System.Threading;

namespace test
{
    internal class Program
    {
    static void Main(string[] args)
    {
        Console.WriteLine("Hello, World!");
        LoopNN();
    }

        static void LoopNN()
        {
        while (true)
        {
            for (int i = 0; i < 10; i++)
            {
                for (int e = 0; e < 10; e++)
                {
                    Console.Write($"{i}{e} ");       // <-- line number 23
                    Thread.Sleep(500);               // <-- line number 24
                }
                Console.WriteLine();
            }
        }
        }
    }
}

Test log:

[root@localhost sdk_tools]# ./netcoredbg/netcoredbg --interpreter=cli -- dotnet ./test.dll
ncdb> b Program.cs:23
 Breakpoint 1 at Program.cs:23 --pending, warning: No executable code of the debugger's target code type is associated with this line.
ncdb> b Program.cs:24
 Breakpoint 2 at Program.cs:24 --pending, warning: No executable code of the debugger's target code type is associated with this line.
ncdb> r

thread created, id: 26014
^running

library loaded: /usr/share/dotnet/shared/Microsoft.NETCore.App/6.0.9/System.Private.CoreLib.dll
no symbols loaded, base address: 0xae8e0000, size: 10616832(0xa20000)

library loaded: /opt/usr/home/owner/share/tmp/sdk_tools/test.dll
symbols loaded, base address: 0xb6f7d000, size: 5120(0x1400)
breakpoint modified,  Breakpoint 2 at /home/viewizard/Desktop/projects_test/test/Program.cs:24
breakpoint modified,  Breakpoint 1 at /home/viewizard/Desktop/projects_test/test/Program.cs:23

library loaded: /usr/share/dotnet/shared/Microsoft.NETCore.App/6.0.9/System.Runtime.ni.dll
no symbols loaded, base address: 0xb4c53000, size: 245760(0x3c000)

library loaded: /usr/share/dotnet/shared/Microsoft.NETCore.App/6.0.9/System.Console.ni.dll
no symbols loaded, base address: 0xb4268000, size: 618496(0x97000)

library loaded: /usr/share/dotnet/shared/Microsoft.NETCore.App/6.0.9/System.Threading.ni.dll
no symbols loaded, base address: 0xb4222000, size: 286720(0x46000)

library loaded: /usr/share/dotnet/shared/Microsoft.NETCore.App/6.0.9/Microsoft.Win32.Primitives.ni.dll
no symbols loaded, base address: 0xb38c1000, size: 253952(0x3e000)
Hello, World!

library loaded: /usr/share/dotnet/shared/Microsoft.NETCore.App/6.0.9/System.Threading.Thread.ni.dll
no symbols loaded, base address: 0xb388d000, size: 212992(0x34000)

stopped, reason: breakpoint 1 hit, thread id: 26014, stopped threads: all, times= 1, frame={test.Program.LoopNN() at /home/viewizard/Desktop/projects_test/test/Program.cs:23}
ncdb> s
^running

stopped, reason: end stepping range, thread id: 26014, stopped threads: all, frame={test.Program.LoopNN() at /home/viewizard/Desktop/projects_test/test/Program.cs:24}
ncdb> s
^running
00 
stopped, reason: end stepping range, thread id: 26014, stopped threads: all, frame={test.Program.LoopNN() at /home/viewizard/Desktop/projects_test/test/Program.cs:25}
ncdb> s
^running

stopped, reason: end stepping range, thread id: 26014, stopped threads: all, frame={test.Program.LoopNN() at /home/viewizard/Desktop/projects_test/test/Program.cs:26}
ncdb> s
^running

stopped, reason: end stepping range, thread id: 26014, stopped threads: all, frame={test.Program.LoopNN() at /home/viewizard/Desktop/projects_test/test/Program.cs:22}
ncdb> s
^running

stopped, reason: end stepping range, thread id: 26014, stopped threads: all, frame={test.Program.LoopNN() at /home/viewizard/Desktop/projects_test/test/Program.cs:22}
ncdb> s
^running

stopped, reason: end stepping range, thread id: 26014, stopped threads: all, frame={test.Program.LoopNN() at /home/viewizard/Desktop/projects_test/test/Program.cs:23}
ncdb> s
^running

stopped, reason: end stepping range, thread id: 26014, stopped threads: all, frame={test.Program.LoopNN() at /home/viewizard/Desktop/projects_test/test/Program.cs:24}
ncdb> s
^running
01 
stopped, reason: end stepping range, thread id: 26014, stopped threads: all, frame={test.Program.LoopNN() at /home/viewizard/Desktop/projects_test/test/Program.cs:25}
ncdb> s
^running

stopped, reason: end stepping range, thread id: 26014, stopped threads: all, frame={test.Program.LoopNN() at /home/viewizard/Desktop/projects_test/test/Program.cs:26}
ncdb> s
^running

stopped, reason: end stepping range, thread id: 26014, stopped threads: all, frame={test.Program.LoopNN() at /home/viewizard/Desktop/projects_test/test/Program.cs:22}
ncdb> s
^running

stopped, reason: end stepping range, thread id: 26014, stopped threads: all, frame={test.Program.LoopNN() at /home/viewizard/Desktop/projects_test/test/Program.cs:22}
ncdb> s
^running

stopped, reason: end stepping range, thread id: 26014, stopped threads: all, frame={test.Program.LoopNN() at /home/viewizard/Desktop/projects_test/test/Program.cs:23}
ncdb> s
^running

stopped, reason: end stepping range, thread id: 26014, stopped threads: all, frame={test.Program.LoopNN() at /home/viewizard/Desktop/projects_test/test/Program.cs:24}
ncdb> s
^running
02 
stopped, reason: end stepping range, thread id: 26014, stopped threads: all, frame={test.Program.LoopNN() at /home/viewizard/Desktop/projects_test/test/Program.cs:25}
ncdb> s
^running

stopped, reason: end stepping range, thread id: 26014, stopped threads: all, frame={test.Program.LoopNN() at /home/viewizard/Desktop/projects_test/test/Program.cs:26}
ncdb> q

stopped, reason: exited, exit-code: 1
^exit

Could you please provide more info about what CoreCLR version you have installed on your target, how you build program (what .NET version), is your target ARM 32bit or ARM 64 bit?

[viewizard]

Test for sequence "interrupted" -> bp set -> stepping:

[root@localhost sdk_tools]# ./netcoredbg/netcoredbg --interpreter=cli -- dotnet ./vscode_test.dll
ncdb> r
^running

library loaded: /usr/share/dotnet/shared/Microsoft.NETCore.App/6.0.9/System.Private.CoreLib.dll
no symbols loaded, base address: 0xae8e0000, size: 10616832(0xa20000)

thread created, id: 787

library loaded: /opt/usr/home/owner/share/tmp/sdk_tools/vscode_test.dll
symbols loaded, base address: 0xb6f7f000, size: 5120(0x1400)

library loaded: /usr/share/dotnet/shared/Microsoft.NETCore.App/6.0.9/System.Runtime.ni.dll
no symbols loaded, base address: 0xb4c53000, size: 245760(0x3c000)

library loaded: /usr/share/dotnet/shared/Microsoft.NETCore.App/6.0.9/System.Console.ni.dll
no symbols loaded, base address: 0xb4268000, size: 618496(0x97000)

library loaded: /usr/share/dotnet/shared/Microsoft.NETCore.App/6.0.9/System.Threading.ni.dll
no symbols loaded, base address: 0xb4222000, size: 286720(0x46000)

library loaded: /usr/share/dotnet/shared/Microsoft.NETCore.App/6.0.9/Microsoft.Win32.Primitives.ni.dll
no symbols loaded, base address: 0xb38c1000, size: 253952(0x3e000)
Hello, World!

library loaded: /usr/share/dotnet/shared/Microsoft.NETCore.App/6.0.9/System.Threading.Thread.ni.dll
no symbols loaded, base address: 0xb388d000, size: 212992(0x34000)
00 01 02 03 04 05 06 07 ^C
stopped, reason: interrupted, thread id: 787, stopped threads: all, frame={test.Program.LoopNN() at /home/viewizard/Desktop/projects_test/vscode_test/Program.cs:25}
ncdb> b /home/viewizard/Desktop/projects_test/vscode_test/Program.cs:23
 Breakpoint 1 at /home/viewizard/Desktop/projects_test/vscode_test/Program.cs:23
ncdb> s
^running

stopped, reason: end stepping range, thread id: 787, stopped threads: all, frame={test.Program.LoopNN() at /home/viewizard/Desktop/projects_test/vscode_test/Program.cs:26}
ncdb> s
^running

stopped, reason: end stepping range, thread id: 787, stopped threads: all, frame={test.Program.LoopNN() at /home/viewizard/Desktop/projects_test/vscode_test/Program.cs:22}
ncdb> s
^running

stopped, reason: end stepping range, thread id: 787, stopped threads: all, frame={test.Program.LoopNN() at /home/viewizard/Desktop/projects_test/vscode_test/Program.cs:22}
ncdb> s
^running

stopped, reason: end stepping range, thread id: 787, stopped threads: all, frame={test.Program.LoopNN() at /home/viewizard/Desktop/projects_test/vscode_test/Program.cs:23}
ncdb> s
^running

stopped, reason: end stepping range, thread id: 787, stopped threads: all, frame={test.Program.LoopNN() at /home/viewizard/Desktop/projects_test/vscode_test/Program.cs:24}
ncdb> s
^running
08 
stopped, reason: end stepping range, thread id: 787, stopped threads: all, frame={test.Program.LoopNN() at /home/viewizard/Desktop/projects_test/vscode_test/Program.cs:25}
ncdb> s
^running
s
stopped, reason: end stepping range, thread id: 787, stopped threads: all, frame={test.Program.LoopNN() at /home/viewizard/Desktop/projects_test/vscode_test/Program.cs:26}
ncdb> s
^running

stopped, reason: end stepping range, thread id: 787, stopped threads: all, frame={test.Program.LoopNN() at /home/viewizard/Desktop/projects_test/vscode_test/Program.cs:22}
ncdb> s
^running

stopped, reason: end stepping range, thread id: 787, stopped threads: all, frame={test.Program.LoopNN() at /home/viewizard/Desktop/projects_test/vscode_test/Program.cs:22}
ncdb> q

stopped, reason: exited, exit-code: 1
^exit

[ShkumbinMNokia]

We are currently using the Microsoft.AspNetCore.App 6.0.5 ,Microsoft.NETCore.App 6.0.5 and our ARM 32 armv7l.

[viewizard]

I tested on RPI4 with 32 bit armv7l image:

[root@localhost sdk_tools]# uname -a
Linux localhost 5.15.79-arm-rpi4-v7l #1 SMP Tue Aug 8 03:35:12 UTC 2023 armv7l GNU/Linux

Our current runtime for arm32/arm64 testing is 6.0.9.

[gbalykov]

Which version of netcoredbg do you use? Is your app built in Debug mode? Are all pdbs on ARM target too?

[ShkumbinMNokia]

Im currently using the 2.2.3-1 version. Ive tried building it in Debug mode, but I it didnt change anything. All pdbs are on ARM target as well. on our sim VM the runtime is 6.0.3 and everything works there fine. Do i have to do something else if i wanna retry downloading it?

[gbalykov]

Can you share your built dlls & pdbs? We'll check them on our side

[ShkumbinMNokia]

It is working on the x86 thats why im pretty sure it doesnt have to do [anything] with the dll and pdb. I'll give it to you. Loop.zip

[viewizard]

Just tested your dll/pdb, no issues on our ARM target ("interrupted" -> bp set -> stepping):

[root@localhost sdk_tools]# uname -a
Linux localhost 5.15.79-arm-rpi4-v7l #1 SMP Tue Aug 8 03:35:12 UTC 2023 armv7l GNU/Linux

[root@localhost sdk_tools]# ./netcoredbg/netcoredbg --interpreter=cli -- dotnet ./LoopNumbersNew.dll
ncdb> r
^running

library loaded: /usr/share/dotnet/shared/Microsoft.NETCore.App/6.0.9/System.Private.CoreLib.dll
no symbols loaded, base address: 0xae8e0000, size: 10616832(0xa20000)

thread created, id: 26534

library loaded: /opt/usr/home/owner/share/tmp/sdk_tools/LoopNumbersNew.dll
symbols loaded, base address: 0xb6efb000, size: 6656(0x1a00)

library loaded: /usr/share/dotnet/shared/Microsoft.NETCore.App/6.0.9/System.Runtime.ni.dll
no symbols loaded, base address: 0xb42c3000, size: 245760(0x3c000)

library loaded: /usr/share/dotnet/shared/Microsoft.NETCore.App/6.0.9/System.Console.ni.dll
no symbols loaded, base address: 0xb422c000, size: 618496(0x97000)

library loaded: /usr/share/dotnet/shared/Microsoft.NETCore.App/6.0.9/System.Threading.ni.dll
no symbols loaded, base address: 0xb389a000, size: 286720(0x46000)

library loaded: /usr/share/dotnet/shared/Microsoft.NETCore.App/6.0.9/Microsoft.Win32.Primitives.ni.dll
no symbols loaded, base address: 0xb384c000, size: 253952(0x3e000)
Hello, World!

library loaded: /usr/share/dotnet/shared/Microsoft.NETCore.App/6.0.9/System.Threading.Thread.ni.dll
no symbols loaded, base address: 0xb3818000, size: 212992(0x34000)
00 01 02 03 ^C
stopped, reason: interrupted, thread id: 26534, stopped threads: all, frame={LoopNumbers.Program.LoopNN() at Z:\LoopNumbersNew\LoopNumbersNew\Program.cs:20}
ncdb> b Program.cs:19
 Breakpoint 1 at Z:\LoopNumbersNew\LoopNumbersNew\Program.cs:19
ncdb> s
^running

stopped, reason: end stepping range, thread id: 26534, stopped threads: all, frame={LoopNumbers.Program.LoopNN() at Z:\LoopNumbersNew\LoopNumbersNew\Program.cs:21}
ncdb> s
^running

stopped, reason: end stepping range, thread id: 26534, stopped threads: all, frame={LoopNumbers.Program.LoopNN() at Z:\LoopNumbersNew\LoopNumbersNew\Program.cs:17}
ncdb> s
^running

stopped, reason: end stepping range, thread id: 26534, stopped threads: all, frame={LoopNumbers.Program.LoopNN() at Z:\LoopNumbersNew\LoopNumbersNew\Program.cs:17}
ncdb> s
^running

stopped, reason: end stepping range, thread id: 26534, stopped threads: all, frame={LoopNumbers.Program.LoopNN() at Z:\LoopNumbersNew\LoopNumbersNew\Program.cs:18}
ncdb> s
^running

stopped, reason: end stepping range, thread id: 26534, stopped threads: all, frame={LoopNumbers.Program.LoopNN() at Z:\LoopNumbersNew\LoopNumbersNew\Program.cs:19}
ncdb> s
^running
04 
stopped, reason: end stepping range, thread id: 26534, stopped threads: all, frame={LoopNumbers.Program.LoopNN() at Z:\LoopNumbersNew\LoopNumbersNew\Program.cs:20}
ncdb> s
^running

stopped, reason: end stepping range, thread id: 26534, stopped threads: all, frame={LoopNumbers.Program.LoopNN() at Z:\LoopNumbersNew\LoopNumbersNew\Program.cs:21}
ncdb> q

stopped, reason: exited, exit-code: 1
^exit

[gbalykov]

@ShkumbinMNokia can you try to run your program with newer .net, for example .net 7?

[ShkumbinMNokia]

We had to build our own dotnet as we run on a softfloat arm.

[gbalykov]

We had to build our own dotnet as we run on a softfloat arm.

This might be the problem, try to run coreclr tests (https://github.com/dotnet/runtime/blob/main/docs/workflow/testing/coreclr/testing.md), as well as netcoredbg tests (https://github.com/Samsung/netcoredbg#running-the-tests) on your arm32 target

[gbalykov]

Closing since this seems related to runtime. Feel free to reopen if you have any more questions.