[XM][Legacy] mmp does not appear to run dsymutil on output to fully generate native symbols

[tipa]

Steps to Reproduce

I started using the recently released AppCenter SDK for macOS to track crashes in my app. One crash report I would want to symbolicate and the AppCenter website offers a button to upload debug symbols with these hints:

How to find the symbols

- Choose Window > Organizer in Xcode.
- Select the tab Archives.
- Select your app in the left sidebar.
- Right-click on the archive of the needed version and select Show in Finder.
- Right-click on the .xcarchive in Finder and select Show Package Contents.
- You should see a folder named dSYMs which contains your dSYM bundle.

Unfortunately, all my packages do not contain a dSYM folder. I researched and tried various stuff for hours (like using --package-debug-symbols=true) - without success. How can I generate debug symbols for my MacOS app?

Environment

Visual Studio Community 2022 for Mac Preview
Version 17.0 Preview (17.0 build 8754)
Installation UUID: 74feca92-64cf-497e-93e5-0e5202aaa7de

Runtime
.NET 6.0.3 (64-bit)
Architecture: Arm64

Roslyn (Language Service)
4.1.0-3.22075.3+592501cbb9c9394072a245c15b3458ff88155d85

NuGet
Version: 6.0.0.262

.NET SDK (Arm64)
SDK: /usr/local/share/dotnet/sdk/6.0.202/Sdks
SDK Versions:
    6.0.202
    6.0.201
    6.0.100-rc.1.21463.6
    5.0.402 (x64 – Unsupported)
    5.0.401 (x64 – Unsupported)
    5.0.400 (x64 – Unsupported)
    5.0.302 (x64 – Unsupported)
    5.0.301 (x64 – Unsupported)
    5.0.203 (x64 – Unsupported)
    5.0.202 (x64 – Unsupported)
    5.0.201 (x64 – Unsupported)
    3.1.414 (x64 – Unsupported)
    3.1.413 (x64 – Unsupported)
    3.1.412 (x64 – Unsupported)
    3.1.411 (x64 – Unsupported)
    3.1.410 (x64 – Unsupported)
    3.1.409 (x64 – Unsupported)
    3.1.408 (x64 – Unsupported)
    3.1.407 (x64 – Unsupported)
MSBuild SDKs: /usr/local/share/dotnet/sdk/6.0.202/Sdks

.NET SDK (x64)
SDK Versions:
    6.0.202
    6.0.201
    6.0.200
    5.0.407
    5.0.406
    5.0.405
    3.1.418
    3.1.417
    3.1.416

.NET Runtime (Arm64)
Runtime: /usr/local/share/dotnet/dotnet
Runtime Versions:
    6.0.4
    6.0.3
    6.0.0-rc.1.21451.13
    5.0.11 (x64 – Unsupported)
    5.0.10 (x64 – Unsupported)
    5.0.9 (x64 – Unsupported)
    5.0.8 (x64 – Unsupported)
    5.0.7 (x64 – Unsupported)
    5.0.6 (x64 – Unsupported)
    5.0.5 (x64 – Unsupported)
    5.0.4 (x64 – Unsupported)
    3.1.20 (x64 – Unsupported)
    3.1.19 (x64 – Unsupported)
    3.1.18 (x64 – Unsupported)
    3.1.17 (x64 – Unsupported)
    3.1.16 (x64 – Unsupported)
    3.1.15 (x64 – Unsupported)
    3.1.14 (x64 – Unsupported)
    3.1.13 (x64 – Unsupported)

.NET Runtime (x64)
Runtime: /usr/local/share/dotnet/x64/dotnet
Runtime Versions:
    6.0.4
    6.0.3
    6.0.2
    5.0.16
    5.0.15
    5.0.14
    3.1.24
    3.1.23
    3.1.22

Xamarin.Profiler
'/Applications/Xamarin Profiler.app' not found

Updater
Version: 11

Apple Developer Tools
Xcode 13.3.1 (20103)
Build 13E500a

Xamarin.Mac
Version: 8.8.0.3 (Visual Studio Community)
Hash: eb7f8ed35
Branch: xcode13.3
Build date: 2022-03-30 03:16:09-0400

Xamarin.iOS
Version: 15.8.0.3 (Visual Studio Community)
Hash: eb7f8ed35
Branch: xcode13.3
Build date: 2022-03-30 03:16:09-0400

Xamarin Designer
Version: 17.1.5.88
Hash: 25a718e34
Branch: remotes/origin/d17-0-vsmac
Build date: 2022-04-22 14:55:54 UTC

Xamarin.Android
Not Installed

Microsoft Build of OpenJDK
Java SDK: Not Found

Eclipse Temurin JDK
Java SDK: Not Found

Android SDK Manager
Version: 17.2.0.32
Hash: 358904c
Branch: remotes/origin/d17-2
Build date: 2022-04-22 14:55:59 UTC

Android Device Manager
Version: 0.0.0.1110
Hash: 6cb643c
Branch: remotes/origin/d17-2
Build date: 2022-04-22 14:55:59 UTC

Build Information
Release ID: 1700008754
Git revision: b1c3df15b587d32846f3254e47d654dfb6d3627b
Build date: 2022-04-22 14:54:05+00
Build branch: release-17.0
Build lane: release-17.0

Operating System
Mac OS X 12.3.1
Darwin 21.4.0 Darwin Kernel Version 21.4.0
    Fri Mar 18 00:47:26 PDT 2022
    root:xnu-8020.101.4~15/RELEASE_ARM64_T8101 arm64

[chamons]

So package-debug-symbols deals with managed symbols, as noted in the help:

% /Library/Frameworks/Xamarin.Mac.framework/Versions/Current/bin/mmp --help | grep package-debug-symbols -C 2
      --package-debug-symbols[=VALUE]
                             Specify whether debug info files (*.mdb / *.pdb)
                               should be packaged in the app. Default is 'true'

For a Xamarin.Mac application, by default there actually is very little that has "reasonable" dSYM's for us to give you, just the launcher. This is because we JIT (Just in Time) compile your assemblies at application launch, which is different that you may see say on iOS. These JIT'ed assemblies are only in memory, and thus don't have a dSYM.

However, as documented here you can enable AOT (Ahead of Time) compilation where we compile your application down to mostly native code. This will increase your app bundle size, along with build time, but also speed up startup time and produce a dSYM that may improve stack trace reports.

Given your description, I would try enabling AOT and see if that is the behavior you desire.

[ghost]

Hi @tipa. We have added the "need-info" label to this issue, which indicates that we have an open question for you before we can take further action. This issue will be closed automatically in 7 days if we do not hear back from you by then - please feel free to re-open it if you come back to this issue after that time.

[tipa]

I have enabled AOT already, that's why I am confused that there's no dSYM available:

This is how the crashes show up in AppCenter:

[tipa]

@chamons do you need any additional info? The dsym folder is created for my iOS project, but never for my macOS project. Should I post my .csproj here as well?

[tipa]

Any additional information I could provide?

[chamons]

Apologies for the delay in response.

Could you please attach a build log for your project, ideally a binary build log, so I can look into this further.

[tipa]

@chamons I have sent a build log via mail as it contains confidential information that I don't want to share publicly. I am not familiar how to create a binary build, but if it's required, I will try to figure it out and provide that as well.

[chamons]

@tipa - Could you please attach (or mail to me) @/Users/timopartl/GitHub/workinghours/WorkingHours.MacOS/obj/Release/response-file.rsp as the arguments to mmp are not listed in your log.

[tipa]

I've sent the file via mail. It doesn't contain any mention of "--xml=LinkDescription.xml" however, which I specified for the Release builds as additional mmp arguments.

[tipa]

Did the mail arrive? Do you need any additional details?

[tipa]

@chamons is there any additional information I can provide? Or will this be fixed with .NET6 perhaps?

[chamons]

Apologies for the delay in response, your e-mail got filtered strangely.

I was able to get both e-mails, and the response file seems reasonable. The XML argument you mentioned is passed directly to mmp, not through the response file, as you can see on this line:

/Library/Frameworks/Xamarin.Mac.framework/Versions/Current/bin/mmp @/Users/timopartl/GitHub/workinghours/WorkingHours.MacOS/obj/Release/response-file.rsp --xml=LinkDescription.xml 

However, you pass multiple quiets to mmp:

-q -q -q -q

which drastically reduces the information mmp reports, I am unable to see the clang invocation at all. Can you please fix that and send another copy.

[tipa]

Thanks for your response! I don't know where I pass those "-q -q -q -q" to mmp. There's no mention of it in the .csproj. Could you please guide me to where I find it and what exactly I have to change? I am building the archive via the Visual Studio IDE. I can send you my .csproj file if it helps.

[chamons]

Hmm, do you have a verbosity setting here:

in your Visual Studio -> Preferences -> Build and Debug -> Build?

According to the code it appears those are added when you are set to 'Quiet' logs, which will be very difficult to get anything from.

Whenever you are looking into a build issues, Detailed or Diagnostic is a good default.

[tipa]

This verbosity setting is set to "Diagnostic". I did another build and inspected the response-file.rps, but still it looks like these "-q -q -q -q" parameters are being used:

Could this be a bug in the IDE?

[chamons]

That is strange, I can reproduce the fact that setting Diagnostic in the IDE is still adding 4 q's. Filing an issue on that...

[chamons]

Filed that issue in https://github.com/xamarin/xamarin-macios/issues/15506

@tipa - Try setting 8 -v's in your additional mtouch arguments:

[ghost]

Hi @tipa. We have added the "need-info" label to this issue, which indicates that we have an open question for you before we can take further action. This issue will be closed automatically in 7 days if we do not hear back from you by then - please feel free to re-open it if you come back to this issue after that time.

[chamons]

Oh, sorry this is macOS, same idea but mmp args:

[tipa]

Thanks for the info! I did another build and the log seems to contain more information. I've send you another email incl. the build log and the .rsp file

[chamons]

Let me do some digging, something seems wrong...

[chamons]

After doing some research on the code in mmp, it does not appear we are correctly calling dsymutil to process native symbols.

If I'm reading the code correctly, we are correctly running it on AOT'ed code.

[chamons]

I started looking at fixing this, but it turned out to take too long for my side aside triage slot.

Patch: https://gist.github.com/123458ee81c3b2f64d765b5f90e0746a

What I figured out:

• We aren't calling CreateDsym anywhere in mmp, unlike in iOS (Application.mtouch.cs), so we need to add a call
• However, the argument order seems to be different, if I used the iOS version I get errors that appear fixed just be reordering
• However, even with that we fail with the lipo from RunLipoAndCreateDsym (App, AppPath, compiledExecutables);.
• I could be calling RunLipoAndCreateDsym incorrectly, or it could have never worked with macOS dsyms, since we never make them in non-AOT use cases fwict.

[chamons]

I have confirmed this is legacy only, as a dotnet6 project with <ArchiveOnBuild>true</ArchiveOnBuild> set will call dsymutil as part of a publish step.

[ankasani]

Hi Team, Is there any solution for this? even i am facing the issue!

we use xamarin.ios 15.4 then we can see native symbols are generating, with xcode 13.2.1 we changed xamarin.ios 16.0 then symbols are not generating, with xcode 14.1

[rolfbjarne]

@ankasani this particular problem is limited to Xamarin.Mac, which means that if you have a problem when using Xamarin.iOS, then it's a different problem. Please file a new issue and we'll have a look: https://github.com/xamarin/xamarin-macios/issues/new

[tipa]

As I now migrated my apps to .NET7, this is no longer an issue for me. I hope for AOT compilation to be possible again soon, as dealing with crash reports got much more difficult with JIT-compiled apps (even with debug symbols)