Debugging/Stepping into managed code on macOS (AOT managed assemblies)

[lemonmojo]

Is there some way, using E4k to debug/step into managed code? It certainly doesn't work directly from Xcode which would be awesome but probably hard to do or impossible. Although I've seen "competing" products, like RemObjects C# that can do exactly that I would also be fine with running Visual Studio for Mac side by side with Xcode and attaching the debugger to a running instance of my app.

I'm used to being able to start mono programs using something like this: --debug --debugger-agent=transport=dt_socket,server=y,address=127.0.0.1:55555, and being able to attach the debugger later on. Is something like this possible in E4k? If not, are there plans to add such functionality?

This is one of the main reasons we haven't yet fully adopted the project, along with missing support for events/delegates/actions/etc.

Thx!

[chamons]

We would need to Ahead of Time compile the managed assemblies.

In theory, cooking up a mono invocation to AOT the assemblies after they were packaged in a framework could work.

We generate those for XM here - https://github.com/xamarin/xamarin-macios/blob/master/tools/mmp/aot.cs

Looking at our launcher code, unless you do hybrid AOT (deleting the managed libs) there shouldn't be any launcher changes needed.

This is an interesting feature request. Maybe poke around with it yourself and see what you can get working?

[lemonmojo]

Hmm, interesting! Haven't thought about AOT yet. While I will definitely try this in combination with E4k, now that I think about it I'd probably even prefer debugging managed stuff from Visual Studio.

One reason is that it wouldn't require AOT (I think for some of my projects this is a no-go because of their plugin-based nature) and another is that all the tools that make sense for .NET debugging are already there in VS.

So basically, I'd like to see some way of passing the debugging flags that I normally append to my mono command (--debug --debugger-agent...) to E4k's invocation of the mono runtime. Is that possible?

[chamons]

So anything is possible, given a big enough hammer. :)

The reason I suggest AOT is that if you want to "step into" managed code from a native debugger, JIT frames are not easily debuggable at all.

I guess you could try to the other way around and convince the spun up mono to enable debugging. I have no idea if/how that'd work. I'd start looking in https://github.com/mono/Embeddinator-4000/blob/master/support/mono_embeddinator.c#L97

[lemonmojo]

Thx for the pointers!

So I just tried the AOT approach without any luck. I AOT'd my single managed assembly inside the MonoBundle folder of the .framework. The compilation worked without errors and I verified that the .dylib sits next to the .dll. There's even a .dll.dylib.dSYM bundle.

Unfortunately stepping into managed code this way doesn't appear to work. Also, when running with Instruments I don't see the AOT'd frames, just the regular old jit addresses so I assume the AOT'd bits don't get loaded at all.

Any suggestions?

[lemonmojo]

Also, I found mono_debugger_agent_init and injected it here, recompiled E4k and my framework then launched my app using the previously mentioned debugger arguments but that doesn't appear to do anything at all.

Not really sure if that's even the correct way to tell mono to launch the debugger agent but it sure sounds like it.

[chamons]

Hmm, You could try setting the MONO_LOG_LEVEL=debug environmental variable and scan the output. There should be some line about loading the assembly, possibly noting why mono did not use the AOT version.

You have to use the exact version of mono that we bundle and the correct arguments, or it will fail a sanity check and be ignored. You might want to AOT a use of your library in a XM test project and compare your invocations.

I know little about our startup process / mono debugger init, so I can't comment much on that bit.

[lemonmojo]

So launching with MONO_LOG_LEVEL=debug tells me info: AOT: module /PATH/LIBNAME.dll.dylib is unusable: compiled with --aot=full..

My managed library is compiled with the "Xamarin.Mac Modern" target framework. I tried AOTing with /Library/Frameworks/Mono.framework/Versions/5.8.1/bin/mono and /Library/Frameworks/Xamarin.Mac.framework/Versions/4.2.1.29/bin/bmac-mobile-mono. Both result in the same log message mentioned above.

Any ideas on which mono I should use to AOT my libs?

[lemonmojo]

So I was somehow able to AOT the assemblies and get them to load by copying the csc.exe invocation from the build output in VS, then appening --aot=all to it. The resulting files were placed in another location but by simply overwriting the build output in my actual build location I was able to get them to load and even step into the managed code from within Xcode.

I'd still be interested how to "properly" call AOT after building a Xamarin.Mac library targeting the modern framework.

Unfortunately, the debugging information available in Xcode is not really helpful (at least for people like me who aren't that comfortable with machine code ;-)). So I'd still be interested in the other option of setting up the debugging agent for mono before running the app and then attaching the mono debugger to it.

Could you please point someone who's familiar with how the debugging infrastructure in mono works to this thread to see if they have some suggestions on how this could be implemented? I'm unfortunately out of ideas at the moment and not really sure where to continue looking...

thx!

[chamons]

A few things:

• I forgot you need -v set in your mmp arguments to see the AOT invocation.
• Each one looks like this roughly /Library/Frameworks/Xamarin.Mac.framework/Versions/Current/bin/bmac-mobile-mono --aot --runtime=mobile /Users/donblas/Projects/ModernTest/bin/Debug/ModernTest.app/Contents/MonoBundle/mscorlib.dll
• There should be symbol information next to each dylib (Foo.dll.dylib.dSYM) that Xcode loads. Did those get copied over?
• I think @rolfbjarne knows the debugging infrastructure, but it may be significant amount of work to get something work, likely much more than getting AOT w\ symbols.

[lemonmojo]

I tried your invocation (/Library/Frameworks/Xamarin.Mac.framework/Versions/Current/bin/bmac-mobile-mono --aot --runtime=mobile /PATH/LIBNAME.dll) but that resulted in the following error:

The assembly mscorlib.dll was not found or could not be loaded. It should have been installed in the '/Library/Frameworks/Xamarin.Mac.framework/Versions/4.2.1.29/lib/mono/2.1/mscorlib.dll' directory.'

Regarding There should be symbol information next to each dylib (Foo.dll.dylib.dSYM) that Xcode loads.: Yes, as mentioned I was able to get the AOT bits to load and actually was able to step into the code from within Xcode, it's just not very useful for me. ;-)

I'm not sure why this topic hasn't been brought up before as at least to me this is what keeps me from using E4k for serious projects and I don't think I'm alone with the requirement of being able to debug managed code.

[chamons]

I forgot, there is one more step for AOT, you have to set MONO_PATH environmental variable (https://github.com/xamarin/xamarin-macios/blob/master/tools/mmp/aot.cs#L207).

Try adding MONO_PATH=/Library/Frameworks/Xamarin.Mac.framework/Versions/Current/lib/mono/Xamarin.Mac/ at the beginning of your invocation.

If you have the dsyms for all of the managed code, then you should be getting at least symbols in your call stack, are you seeing that? I'm having trouble tracking what things aren't working that you expect.

And Embeddinator is rather new and still rough around the edges, so it doesn't surprise me that it hasn't been brought up before. :)

[lemonmojo]

Tried your suggestion and AOT did appear to work: MONO_PATH=/Library/Frameworks/Xamarin.Mac.framework/Versions/Current/lib/mono/Xamarin.Mac/ /Library/Frameworks/Xamarin.Mac.framework/Versions/Current/bin/bmac-mobile-mono --aot --runtime=mobile /PATH/LIB.dll

Mono Ahead of Time compiler - compiling assembly /PATH/LIB.dll AOTID 691CD1DF-0C02-067A-1699-708EE366DB4D Code: 681(42%) Info: 30(1%) Ex Info: 162(10%) Unwind Info: 115(7%) Class Info: 30(1%) PLT: 60(3%) GOT Info: 388(24%) Offsets: 148(9%) GOT: 296 Compiled: 11/11 (100%), No GOT slots: 6 (54%), Direct calls: 2 (200%) Executing the native assembler: "clang" -c -x assembler -o /var/folders/6y/25zmzyq1265g3h4w2nk19kvr0000gn/T/mono_aot_rkrNEJ.o /var/folders/6y/25zmzyq1265g3h4w2nk19kvr0000gn/T/mono_aot_rkrNEJ Executing the native linker: clang --shared -o /PATH/LIB.dll.dylib.tmp /var/folders/6y/25zmzyq1265g3h4w2nk19kvr0000gn/T/mono_aot_rkrNEJ.o Executing dsymutil: dsymutil "/PATH/LIB.dll.dylib" JIT time: 8 ms, Generation time: 3 ms, Assembly+Link time: 1641 ms.

Both, the .dylib and the .dll.dylib.dSYM are there. But when launching the App I get: info: AOT: module /PATH/LIB.dll.dylib is unusable (GUID of dependent assembly mscorlib doesn't match (expected 'AF513545-56F9-4BA9-960A-6B4DEAF7CB7A', got 'FE1ECCF0-BCBC-48BB-807D-D2FB8EB5AC2E').

[chamons]

That sounds like a mismatch of BCLs. Are you sure you are using Modern https://developer.xamarin.com/guides/mac/advanced_topics/target-framework/?

Posting a small sample or a build log of your project would help here.

In any case, you must use the same mscorlib / etc for AOT as your application has, or they will refuse to load. If you are using Full you need Library/Frameworks/Xamarin.Mac.framework/Versions/Current/lib/mono/4.5 and such.

[lemonmojo]

@chamons Yes, I'm sure I'm using Modern. I double-checked before posting. Also, I'm aware that system libs have to match when AOTing.

Anyway, here's a sample project: https://github.com/lemonmojo/EmbeddinatorTest18 Both, E4k and AOT invocations are implemented in the ELib/Embeddinator-4000.targets file so you just have to build from within VS for Mac.

The native project is inside the Embeddinator18Tester folder. The Xcode build scheme has the MONO_LOG_LEVEL variable set to debug so you can instantly see the AOT issue when launching the app (just search the output for ELib.dll.dylib).

[chamons]

So I'm currently not working on Embeddinator but your sample was so self contained and perfect I couldn't resist.

As you noted, we don't use the AOT image in your sample:

2018-05-08 12:32:59.311 Embeddinator18Tester[6048:238564] info: AOT: module /Users/donblas/Projects/EmbeddinatorTest18/Embeddinator18Tester/build/Release/Embeddinator18Tester.app/Contents/Frameworks/ELib.framework/Versions/A/MonoBundle/ELib.dll.dylib is unusable (GUID of dependent assembly mscorlib doesn't match (expected '656FD3D6-3372-46DF-AB4E-11BBB39C6FAD', got 'B3B8BC50-5415-4309-B554-0EAC04B334F9').

The command looks perfect and I can't figure out (within half an hour) what's wrong. Somehow the mscorlib we're packaging in our monobundle isn't the one we AOT against. Possibly a packaging bug in Embeddinator.

In any case, there is a trivial "workaround", just AOT compile after generating your native project:

(from build/Release/Embeddinator18Tester.app/Contents/Frameworks/ELib.framework/Versions/A/MonoBundle/) MONO_PATH=. /Library/Frameworks/Xamarin.Mac.framework/Versions/Current/bin/bmac-mobile-mono --aot --runtime=mobile ELib.dll

That will force AOT to get the right library.

2018-05-08 12:45:07.987 Embeddinator18Tester[6295:246567] info: AOT: image '/Users/donblas/Projects/EmbeddinatorTest18/Embeddinator18Tester/build/Release/Embeddinator18Tester.app/Contents/Frameworks/ELib.framework/Versions/A/MonoBundle/ELib.dll.dylib' found.

I'll need to look at this further, but hopefully that'll unblock you.

[lemonmojo]

Thx for looking into it, Chris! Unfortunately I don't quite understand your suggestion since I'm already AOTing after invoking E4k.

Take a look at my AOT invocation: MONO_PATH=/Library/Frameworks/Xamarin.Mac.framework/Versions/Current/lib/mono/Xamarin.Mac/ /Library/Frameworks/Xamarin.Mac.framework/Versions/Current/bin/bmac-mobile-mono --aot --runtime=mobile $(OutputPath)/$(AssemblyName).framework/Versions/A/MonoBundle/$(AssemblyName).dll

The target library is actually inside the .framework that got created by E4k in the previous step so I'm not sure what you mean by just AOT compile after generating your native project as that's exactly what I'm doing.

[equinox2k]

When I get a chance looking forward to playing and hopefully getting it to work, would be great to add to documentation once we do. Maybe we could add a objcgen parameter to do some of the work?

[lemonmojo]

@equinox2k Even if we can get mono to AOT the libs correctly (which I was able to using some hacking explained in https://github.com/mono/Embeddinator-4000/issues/668#issuecomment-385950605), the AOT debugging experience from within Xcode is unfortunately not very helpful. For instance, there's no way to inspect local (managed) variables, at least from what I saw.

Providing a way to launch the mono debugging agent from E4k and attaching to it using VS for Mac would still be the best way to enable debugging IMHO.

[chamons]

I'm suggesting there is some step between "build your binary with E4K" and "build your app with xcode" that is causing the issue.

I wasn't able to figure it out in the limited time I've spent so far, what I was suggesting was to run AOT after building your project in Xcode. I ran AOT inside the generated bundle and it appeared to work.

Obviously, I think what you did should work. At some point I'll have to spend more time looking into this use case.

[chamons]

This may also be helpful when debugging managed from xcode:

http://www.mono-project.com/docs/debug+profile/debug/#debugging-with-lldb

[lemonmojo]

So I tried your suggestion by doing this:

• Cleaned managed and native build outputs
• Removed AOT invocation from managed .targets file
• Added a new build phase in Xcode project to invoke AOT after compiling native project
• Built managed project
• Built native project
• Checked output and again got 2018-05-10 10:55:11.024194+0200 Embeddinator18Tester[3805:185916] info: AOT: module /PATH/ELib.dll.dylib is unusable (GUID of dependent assembly mscorlib doesn't match (expected '89087A58-E09E-4B89-A98E-91D7AA6B103B', got 'E238B018-F5F7-40DD-99FF-8616A4EE0046').
• I then even tried invoking AOT manually from the terminal and targeting the .dll inside the native build output but that didn't change anything either

I pushed my changes to the sample project on Github so you can see for yourself: https://github.com/lemonmojo/EmbeddinatorTest18

[chamons]

I finally got around to looking at your sample, and by changing your AOT build step to do:

MONO_PATH=${TARGET_BUILD_DIR}/${TARGET_NAME}.app/Contents/Frameworks/ELib.framework/Versions/Current/MonoBundle/ /Library/Frameworks/Xamarin.Mac.framework/Versions/Current/bin/bmac-mobile-mono --aot --runtime=mobile ${TARGET_BUILD_DIR}/${TARGET_NAME}.app/Contents/Frameworks/ELib.framework/Versions/Current/MonoBundle/ELib.dll
MONO_PATH=${TARGET_BUILD_DIR}/${TARGET_NAME}.app/Contents/Frameworks/ELib.framework/Versions/Current/MonoBundle/ /Library/Frameworks/Xamarin.Mac.framework/Versions/Current/bin/bmac-mobile-mono --aot --runtime=mobile ${TARGET_BUILD_DIR}/${TARGET_NAME}.app/Contents/Frameworks/ELib.framework/Versions/Current/MonoBundle/mscorlib.dll
MONO_PATH=${TARGET_BUILD_DIR}/${TARGET_NAME}.app/Contents/Frameworks/ELib.framework/Versions/Current/MonoBundle/ /Library/Frameworks/Xamarin.Mac.framework/Versions/Current/bin/bmac-mobile-mono --aot --runtime=mobile ${TARGET_BUILD_DIR}/${TARGET_NAME}.app/Contents/Frameworks/ELib.framework/Versions/Current/MonoBundle/System.dll

it appears to work.

Now we really shouldn't need this hack, but that should hopefully get you going.

[lemonmojo]

@chamons I can confirm that it works using your latest suggestion. I've also updated the repo. Unfortunately the debugging experience still isn't what I'm looking for or at least I'm not sure how exactly I'm supposed to extract information out of it.

For instance, take a look at the following screenshot:

The stack frames are fine and at least allow me to see which method I'm currently in so that's good. However, local variables show up named V_0 and V_1 which leaves me wondering how I should identify which one's which in more complex examples than this one. Also, I can't see the values of some properties of the class. I'm guessing that's because they're mono strings and not easily convertible to C strings. Furthermore, the FullName property, which has a custom getter (and no setter) is missing from the Xcode debugger and I'm wondering how I can inspect it's value.

[chamons]

So native debugging of AOT code will never be that "great" of an experience, as there is a mismatch between C# and Obj-C. Something things that might help:

• I'm not sure on the string thing, but mono strings are UTF16 and that can't be trivial. You can try executing mono_string_to_utf8_ignore inline in lldb and see if you get somethign
• The script https://github.com/mono/mono/blob/master/data/lldb/monobt.py and the notes http://www.mono-project.com/docs/debug+profile/debug/#debugging-with-lldb might be useful on JIT frames
• You might have to resort to some CWL (Console.WriteLine) debugging to get some information.

[lemonmojo]

First of all, thx for all the effort, I really appreciate it! Like you mentioned, the AOT debugging leaves a lot to be desired and is IMHO only usable for very simple cases.

Regarding strings: I haven't tested your suggestion but will give it a shot later. Regarding the monobt.py script: I tried this and while it does work, I don't see what value it provides since I can already see stack frames in Xcode just fine. Console.WriteLine: Well, yes, as a last resort that's obviously possible but not really what I'm looking for.

So basically, the original feature request to be able to attach Visual Studio's debugger to a running native binary built on top of mono still stands.

[chamons]

Absolutely, the AOT option is just a work around. The original feature request stands.

The monobt.py script is only useful if you see a bunch of JIT frames, even with AOT there are some cases where we JIT things for performance reasons.