BadImageFormatException when loading any found SDK on Mac with M1/M2 (osx-arm64) from osx-x64 application

[rotanov]

On macOS when running an osx-x64 app on M1/M2 (osx-arm64) machine which tries to load Microsoft.Build.dll using MSBuildLocator after executing Microsoft.Build.Locator.MSBuildLocator.RegisterDefaults(); it crashes with exception BadImageFormatException.

Probably because it locates osx-arm64 SDK instead of osx-x64.

Repro

1. Build the following project with dotnet build -c Debug example.csproj on Mac machine with M1/M2 CPU
2. Run it with ./bin/Debug/net7.0-macos/osx-x64/example.app/Contents/MacOS/example ../../../../../example.csproj
3. Observe the result:

2023-10-18 21:15:33.258 example[76111:7444793] Could not find `Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a` referenced by assembly `example, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null`.
Discovery type: DotNetSdk
MS Build path: "/usr/local/share/dotnet/sdk/7.0.309"
Discovery type: DotNetSdk
MS Build path: "/usr/local/share/dotnet/sdk/6.0.415"
Unhandled exception. System.BadImageFormatException: Could not load file or assembly 'Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'. An attempt was made to load a program with an incorrect format.

File name: 'Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' ---> System.BadImageFormatException: An attempt was made to load a program with an incorrect format.
 (0x8007000B)
   at System.Runtime.Loader.AssemblyLoadContext.<LoadFromPath>g____PInvoke|5_0(IntPtr ptrNativeAssemblyBinder, UInt16* ilPath, UInt16* niPath, ObjectHandleOnStack retAssembly)
   at System.Runtime.Loader.AssemblyLoadContext.LoadFromAssemblyPath(String assemblyPath)
   at System.Reflection.Assembly.LoadFrom(String assemblyFile)
   at Microsoft.Build.Locator.MSBuildLocator.<>c__DisplayClass15_0.<RegisterMSBuildPath>g__TryLoadAssembly|3(AssemblyName assemblyName)
   at Microsoft.Build.Locator.MSBuildLocator.<>c__DisplayClass15_0.<RegisterMSBuildPath>b__2(AssemblyLoadContext _, AssemblyName assemblyName)
   at System.Runtime.Loader.AssemblyLoadContext.GetFirstResolvedAssemblyFromResolvingEvent(AssemblyName assemblyName)
   at System.Runtime.Loader.AssemblyLoadContext.ResolveUsingEvent(AssemblyName assemblyName)
   at System.Runtime.Loader.AssemblyLoadContext.ResolveUsingResolvingEvent(IntPtr gchManagedAssemblyLoadContext, AssemblyName assemblyName)
   at Builder.GetOutputPath(String projectPath)
   at Program.Main() in ./example/Main.cs:line 29
[1]    76111 abort      ./bin/Debug/net7.0-macos/osx-x64/example.app/Contents/MacOS/example

example.csproj:

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net7.0-macos</TargetFramework>
    <ApplicationId>com.company.example</ApplicationId>
    <RuntimeIdentifier>osx-x64</RuntimeIdentifier>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.Build.Locator" Version="1.6.10" />
    <PackageReference Include="Microsoft.Build" Version="17.7.2" ExcludeAssets="runtime" />
    <PackageReference Include="Microsoft.Build.Framework" Version="17.7.2" ExcludeAssets="runtime" />
  </ItemGroup>

</Project>

main.cs:

using System;
using System.Collections.Generic;
using Microsoft.Build.Evaluation;

public class Builder
{
  public string GetOutputPath(string projectPath)
  {
    var buildEngine = new ProjectCollection();
    var project = buildEngine.LoadProject(
      projectPath,
      new Dictionary<string, string> { ["Configuration"] = "Release" },
      "Current"
    );
    return project.GetPropertyValue("OutputPath");
  }
}

public class Program
{
    public static void Main()
    {
        var vss = Microsoft.Build.Locator.MSBuildLocator.QueryVisualStudioInstances();
        foreach (var vs in vss) {
            Console.WriteLine($"Discovery type: {vs.DiscoveryType}");
            Console.WriteLine($"MS Build path: \"{vs.MSBuildPath}\"");
        }
        Microsoft.Build.Locator.MSBuildLocator.RegisterDefaults();
        var path = (new Builder()).GetOutputPath(System.Environment.GetCommandLineArgs()[1]);
        Console.WriteLine($"Project output path: \"{path}\"");
    }
}

Environment

macOS Sonoma, XCode 15, freshly installed Visual Studio 2022

dotnet --info:

.NET SDK:
 Version:   7.0.309
 Commit:    552bd6c89b

Runtime Environment:
 OS Name:     Mac OS X
 OS Version:  14.0
 OS Platform: Darwin
 RID:         osx-arm64
 Base Path:   /usr/local/share/dotnet/sdk/7.0.309/

Host:
  Version:      7.0.12
  Architecture: arm64
  Commit:       4a824ef37c

.NET SDKs installed:
  6.0.415 [/usr/local/share/dotnet/sdk]
  7.0.309 [/usr/local/share/dotnet/sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.App 6.0.23 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 7.0.12 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 6.0.23 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 7.0.12 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]

Other architectures found:
  x64   [/usr/local/share/dotnet/x64]
    registered at [/etc/dotnet/install_location_x64]

Environment variables:
  Not set

global.json file:
  Not found

[rainersigwald]

It is surprising to me that we managed to get that far into the execution; on a client-app/SDK architecture mismatch I would have expected something more like a crash when we call into hostfxr. I guess maybe macOS transparently virtualizes the shared-library load?

[rotanov]

@rainersigwald I'm not sure I understand, by "that far into execution" do you imply that all other dependencies loaded by the app (before the Microsoft.Build.dll) are also have wrong architecture? Or are you talking only about that far into loading Microsoft.Build.dll?

[rainersigwald]

Yeah sorry, that was me talking to other folks with knowledge of the details of how MSBuildLocator is implemented, but I didn't say that. . . .

MSBuildLocator has two main phases:

1. figure out where the SDK is
2. configure .NET to load MSBuild stuff from there.

Your crash is showing a failure in step 2.

Step 1 is implemented by calling a native library, hostfxr.dll/hostfxr.dylib. My surprise is that that apparently works in this situation, even though I would have expected the dylib to be ARM64 and then fail to load in an x64 process.

To set expectations here, I think we should have a better error experience--but I don't expect to get this combination of x64-app + ARM64-SDK to Just Work.

[rotanov]

Oh, I see.

I also don't expect x64 app + arm64 sdk to just work, but there's must be a x64 sdk on the machine installed, I'm not sure, by vs2022 installer, or by dotnet workload install macos, but otherwise x64 application shouldn't have successfully build and then run.

The only SDKs that were listed by MSBuildLocator are:

MS Build path: "/usr/local/share/dotnet/sdk/7.0.309"
MS Build path: "/usr/local/share/dotnet/sdk/6.0.415"

both are apparently same as machine architecture.

As per dotnet --info:

...
Other architectures found:
  x64   [/usr/local/share/dotnet/x64]
    registered at [/etc/dotnet/install_location_x64]

They are actually not present by the path specified.

But, as mentioned above x64 SDK must be somewhere on the machine.

Here's what I found, may be it'll be of some use:

-> sudo find /usr -name libhostfxr.dylib | xargs file

/usr/local/share/dotnet/host/fxr/7.0.12/libhostfxr.dylib: Mach-O 64-bit dynamically linked shared library arm64
/usr/local/share/dotnet/host/fxr/6.0.23/libhostfxr.dylib: Mach-O 64-bit dynamically linked shared library arm64

-> sudo find /Applications/Visual\ Studio.app/ -name libhostfxr.dylib | tr \\n \\0 | xargs -0 file

/Applications/Visual Studio.app//Contents/MacOS/Visual Studio Update.app/Contents/MonoBundle/libhostfxr.dylib:                              Mach-O 64-bit dynamically linked shared library arm64
/Applications/Visual Studio.app//Contents/dotnet/host/fxr/7.0.3/libhostfxr.dylib:                                                           Mach-O 64-bit dynamically linked shared library arm64
/Applications/Visual Studio.app//Contents/SharedSupport/VSMonitor.app/Contents/MonoBundle/libhostfxr.dylib:                                 Mach-O universal binary with 2 architectures: [x86_64:Mach-O 64-bit dynamically linked shared library x86_64] [arm64:Mach-O 64-bit dynamically linked shared library arm64]
/Applications/Visual Studio.app//Contents/SharedSupport/VSMonitor.app/Contents/MonoBundle/libhostfxr.dylib (for architecture x86_64):  Mach-O 64-bit dynamically linked shared library x86_64
/Applications/Visual Studio.app//Contents/SharedSupport/VSMonitor.app/Contents/MonoBundle/libhostfxr.dylib (for architecture arm64):  Mach-O 64-bit dynamically linked shared library arm64
/Applications/Visual Studio.app//Contents/MonoBundle/MSBuild/Current/bin/SdkResolvers/Microsoft.DotNet.MSBuildSdkResolver/libhostfxr.dylib: Mach-O 64-bit dynamically linked shared library x86_64
/Applications/Visual Studio.app//Contents/MonoBundle/libhostfxr.dylib:                                                                      Mach-O 64-bit dynamically linked shared library arm64
/Applications/Visual Studio.app//Contents/MonoBundle/MSBuildBuilder/net8.0/osx-x64/libhostfxr.dylib:                                        Mach-O 64-bit dynamically linked shared library x86_64
/Applications/Visual Studio.app//Contents/MonoBundle/MSBuildBuilder/net8.0/osx-arm64/libhostfxr.dylib:                                      Mach-O 64-bit dynamically linked shared library arm64
/Applications/Visual Studio.app//Contents/MonoBundle/MSBuildBuilder/net7.0/osx-x64/libhostfxr.dylib:                                        Mach-O 64-bit dynamically linked shared library x86_64
/Applications/Visual Studio.app//Contents/MonoBundle/MSBuildBuilder/net7.0/osx-arm64/libhostfxr.dylib:                                      Mach-O 64-bit dynamically linked shared library arm64
/Applications/Visual Studio.app//Contents/MonoBundle/AddIns/WebEditors/LanguageServerHost/libhostfxr.dylib:                                 Mach-O 64-bit dynamically linked shared library arm64
/Applications/Visual Studio.app//Contents/MonoBundle/AddIns/MonoDevelop.AzureFunctions/azure-functions-cli/v4/libhostfxr.dylib:             Mach-O 64-bit dynamically linked shared library arm64
/Applications/Visual Studio.app//Contents/MonoBundle/AddIns/MonoDevelop.AzureFunctions/azure-functions-cli/v3/libhostfxr.dylib:             Mach-O 64-bit dynamically linked shared library x86_64

[rotanov]

Also maybe somehow related to this? https://github.com/dotnet/sdk/issues/26052

I mean, if x64 SDK is actually present at the path defined in /etc/dotnet/install_location_x64, will MSBuildLocator list it?

[YuliiaKovalova]

Hi @rotanov,

Could you tell me the values of the next env variables on the machine where the issue exists? DOTNET_ROOT DOTNET_HOST_PATH DOTNET_MSBUILD_SDK_RESOLVER_CLI_DIR

Thank you!

[rotanov]

@YuliiaKovalova they're all empty.

[YuliiaKovalova]

@rotanov, as I workaround, we can advice you to set DOTNET_HOST_PATH environment variable that points to the proper dotnet.exe.

[YuliiaKovalova]

Hi @rotanov,

Please let us know if you still need for help for resolving this issue.

[rotanov]

@YuliiaKovalova No, thank you. I decided to use -getProperty:<propertyName> cli feature of msbuild instead.

[YuliiaKovalova]

@rotanov thank you for the quick update. Feel free to reopen the issue if it becomes relevant to you!