[NativeAOT-LLVM] Publishing Hello World for WebAssembly target on Windows arm64 fails

[ChristianWeyer]

When trying to publish the Hello World project (https://github.com/dotnet/runtimelab/tree/feature/NativeAOT-LLVM/samples/HelloWorld) for WebAssembly target on a Windows ARM machine, with this command:

dotnet publish -r browser-wasm -c Debug /p:TargetArchitecture=wasm /p:PlatformTarget=AnyCPU /p:MSBuildEnableWorkloadResolver=false --self-contained

I get this error:

MSBuild version 17.4.0+18d5aef85 for .NET
  Determining projects to restore...
  Restored C:\Users\christianweyer\source\repos\ConsoleApp1\ConsoleApp1\ConsoleApp1.csproj (in 41.75 sec).
  ConsoleApp1 -> C:\Users\christianweyer\source\repos\ConsoleApp1\ConsoleApp1\bin\Debug\net7.0\browser-wasm\ConsoleApp1.dll
  Optimizing assemblies for size may change the behavior of the app. Be sure to test after publishing. See: https://aka.ms/dotnet-illink
  Generating compatible native code. To optimize for size or speed, visit https://aka.ms/OptimizeCoreRT
**EXEC : error : Unable to load DLL 'clrjit_browser_wasm32_arm64' or one of its dependencies: The specified module could not be found. (0x8007007E) [C:\Users\christianweyer\source\repos\ConsoleApp1\C
onsoleApp1\ConsoleApp1.csproj]**
  System.DllNotFoundException: Unable to load DLL 'clrjit_browser_wasm32_arm64' or one of its dependencies: The specified module could not be found. (0x8007007E)
     at System.Runtime.InteropServices.NativeLibrary.LoadByName(String libraryName, QCallAssembly callingAssembly, Boolean hasDllImportSearchPathFlag, UInt32 dllImportSearchPathFlag, Boolean throwO
  nError)
     at System.Runtime.InteropServices.NativeLibrary.LoadLibraryByName(String libraryName, Assembly assembly, Nullable`1 searchPath, Boolean throwOnError)
     at System.Runtime.InteropServices.NativeLibrary.Load(String libraryName, Assembly assembly, Nullable`1 searchPath)
     at Internal.JitInterface.JitConfigProvider.<>c__DisplayClass5_0.<Initialize>b__0(String libName, Assembly assembly, Nullable`1 searchPath)
     at System.Runtime.InteropServices.NativeLibrary.LoadLibraryCallbackStub(String libraryName, Assembly assembly, Boolean hasDllImportSearchPathFlags, UInt32 dllImportSearchPathFlags)
     at Internal.JitInterface.CorInfoImpl.jitStartup(IntPtr host)
     at Internal.JitInterface.CorInfoImpl.Startup()
     at ILCompiler.LLVMCodegenCompilationBuilder.ToCompilation()
     at ILCompiler.Program.Run(String[] args)
     at ILCompiler.Program.Main(String[] args)
C:\Users\christianweyer\.nuget\packages\microsoft.dotnet.ilcompiler.llvm\7.0.0-alpha.1.22567.1\build\Microsoft.NETCore.Native.targets(313,5): **error MSB3073: The command ""C:\Users\christianweyer\.n
uget\packages\runtime.win-arm64.microsoft.dotnet.ilcompiler.llvm\7.0.0-alpha.1.22567.1\tools\\ilc" @"obj\Debug\net7.0\browser-wasm\native\ConsoleApp1.ilc.rsp"" exited with code 1.** [C:\Users\christanweyer\source\repos\ConsoleApp1\ConsoleApp1\ConsoleApp1.csproj]

Did anyone ever see this?

Thanks!

---

dotnet --info
.NET SDK:
 Version:   7.0.100
 Commit:    e12b7af219

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.22621
 OS Platform: Windows
 RID:         win10-arm64
 Base Path:   C:\Program Files\dotnet\sdk\7.0.100\

Host:
  Version:      7.0.0
  Architecture: arm64
  Commit:       d099f075e4

[jkotas]

This is same problem as https://github.com/dotnet/runtimelab/issues/2119 . The project is not publishing Windows Arm64 and Apple M1 packages yet.

[ChristianWeyer]

... OK, so I am running out of machines now ... 😅

[yowl]

This might be easier than MacOS. If you want to get the prerequisites and try

build clr.jit+clr.tools+clr.nativeaotlibs+clr.wasmjit -c Debug

could start from there?

[jkotas]

... OK, so I am running out of machines now ...

You may be able to work around this by installing x64 SDK on Arm64 machine, and use the existing x64 package. https://github.com/dotnet/sdk/issues/22380 has more details about this config.

[ChristianWeyer]

OK. I have installed the x64 SDK and adjusted the PATH.

dotnet --info
.NET SDK:
 Version:   7.0.101
 Commit:    bb24aafa11

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.22621
 OS Platform: Windows
 RID:         win10-x64
 Base Path:   C:\Program Files\dotnet\x64\sdk\7.0.101\

Host:
  Version:      7.0.1
  Architecture: x64
  Commit:       97203d38ba

And changed the package back to: <PackageReference Include="Microsoft.DotNet.ILCompiler.LLVM; runtime.win-x64.Microsoft.DotNet.ILCompiler.LLVM" Version="7.0.0-*" />

Now, when I run the publish command, I get this (again):

C:\Users\christianweyer\.nuget\packages\microsoft.dotnet.ilcompiler.llvm\7.0.0-alpha.1.22567.1\build\Microsoft.NETCore.Native.Publish.targets(71,5): error : Add a PackageReference for 'runtime.win-arm64.Microsoft.DotNet.ILCompiler.LLVM' to allow cross-compilation for wasm [C:\Users\christianweyer\source\repos\ConsoleApp1\ConsoleApp1\ConsoleApp1.csproj]

Maybe I am still missing a piece...? Ideas on your side @jkotas?

Thanks!

[SingleAccretion]

What if you force it with /p:OSHostArch=/* typo: arm64 */x64? Apparently we try to uncover the "true" architecture in the .targets:

    <OSHostArch>$([System.Runtime.InteropServices.RuntimeInformation]::OSArchitecture.ToString().ToLowerInvariant)</OSHostArch>
    <!-- OSArchitecture does not report the true OS architecture for x86 and x64 processes running on Windows ARM64. -->
    <!-- The following condition checks those cases. -->
    <OSHostArch Condition="$([MSBuild]::IsOSPlatform('Windows')) and
        $([System.Environment]::GetEnvironmentVariable('PROCESSOR_ARCHITECTURE', EnvironmentVariableTarget.Machine)) == 'ARM64'">arm64</OSHostArch>

[jkotas]

/p:IlcHostArch=x64 should work around this problem.

Apparently we try to uncover the "true" architecture in the .targets

We agreed that this is not the expected behavior. https://github.com/dotnet/runtime/issues/79253 tracks fixing it.

[ChristianWeyer]

/p:IlcHostArch=x64 should work around this problem.

Apparently we try to uncover the "true" architecture in the .targets

We agreed that this is not the expected behavior. dotnet/runtime#79253 tracks fixing it.

This brings this error:

MSBuild version 17.4.0+18d5aef85 for .NET
  Determining projects to restore...
  All projects are up-to-date for restore.
  ConsoleApp1 -> C:\Users\christianweyer\source\repos\ConsoleApp1\ConsoleApp1\bin\Debug\net7.0\browser-wasm\ConsoleApp1.dll
  Optimizing assemblies for size may change the behavior of the app. Be sure to test after publishing. See: https://aka.ms/dotnet-illink
emcc : error : /sdk/libPortableRuntime.a: No such file or directory ("/sdk/libPortableRuntime.a" was expected to be an input file, based on the commandline arguments provided) [C:\Users\christianweyer\source\repos\ConsoleApp1\ConsoleApp1\ConsoleApp1.csproj]
C:\Users\christianweyer\.nuget\packages\microsoft.dotnet.ilcompiler.llvm\7.0.0-alpha.1.22567.1\build\Microsoft.NETCore.Native.targets(437,5): error MSB3073: The command ""C:/sources/emsdk/upstream/emscripten/emcc.bat" "obj\Debug\net7.0\browser-
wasm\native\ConsoleApp1.o" "obj\Debug\net7.0\browser-wasm\native\ConsoleApp1clrjit.o" -o "bin\Debug\net7.0\browser-wasm\native\ConsoleApp1.html" -s ALLOW_MEMORY_GROWTH=1 -s ERROR_ON_UNDEFINED_SYMBOLS=0 -s DISABLE_EXCEPTION_CATCHING=0 "/sdk/libPortableRuntime.a" "/sdk/libbootstrapper.a" "/framework/libSystem.Native.a" "/framework/libSystem.Globalization.Native.a"  -g3" exited with code 1. [C:\Users\christianweyer\source\repos\ConsoleApp1\ConsoleApp1\ConsoleApp1.csproj]

[yowl]

This is the package publish failing problem. The last PR that was merged should fix it, but we'll have to wait a few hours to be sure.

[ChristianWeyer]

⏱️...

[ChristianWeyer]

Can you see whether the build and publishing succeeded now @yowl ?

[yowl]

@ChristianWeyer You can check at https://dev.azure.com/dnceng/public/_artifacts/feed/dotnet-experimental - but there is no new package yet. @jkotas is helping out a lot with getting it working, hang in there!

[yowl]

@ChristianWeyer tada:

[ChristianWeyer]

:-)

Now we have:

dotnet publish -r browser-wasm -c Debug /p:TargetArchitecture=wasm /p:PlatformTarget=AnyCPU /p:MSBuildEnableWorkloadResolver=false /p:IlcHostArch=x64 --self-contained
MSBuild version 17.4.0+18d5aef85 for .NET
  Determining projects to restore...
  Restored C:\Users\christianweyer\source\repos\ConsoleApp1\ConsoleApp1\ConsoleApp1.csproj (in 10.08 sec).
  ConsoleApp1 -> C:\Users\christianweyer\source\repos\ConsoleApp1\ConsoleApp1\bin\Debug\net7.0\browser-wasm\ConsoleApp1.dll
  Generating compatible native code. To optimize for size or speed, visit https://aka.ms/OptimizeCoreRT
EXEC : error : Unable to load DLL 'clrjit_browser_wasm32_x64' or one of its dependencies: The specified module could not be found. (0x8007007E) [C:\Users\christianweyer\source\repos\ConsoleApp1\ConsoleApp1\ConsoleApp1.csproj]
  System.DllNotFoundException: Unable to load DLL 'clrjit_browser_wasm32_x64' or one of its dependencies: The specified module could not be found. (0x8007007E)
     at System.Runtime.InteropServices.NativeLibrary.LoadByName(String libraryName, QCallAssembly callingAssembly, Boolean hasDllImportSearchPathFlag, UInt32 dllImportSearchPathFlag, Boolean throwOnError)
     at System.Runtime.InteropServices.NativeLibrary.LoadLibraryByName(String libraryName, Assembly assembly, Nullable`1 searchPath, Boolean throwOnError)
     at System.Runtime.InteropServices.NativeLibrary.Load(String libraryName, Assembly assembly, Nullable`1 searchPath)
     at Internal.JitInterface.JitConfigProvider.<>c__DisplayClass5_0.<Initialize>b__0(String libName, Assembly assembly, Nullable`1 searchPath)
     at System.Runtime.InteropServices.NativeLibrary.LoadLibraryCallbackStub(String libraryName, Assembly assembly, Boolean hasDllImportSearchPathFlags, UInt32 dllImportSearchPathFlags)
     at Internal.JitInterface.CorInfoImpl.jitStartup(IntPtr host)
     at Internal.JitInterface.CorInfoImpl.Startup(CORINFO_OS os)
     at ILCompiler.LLVMCodegenCompilationBuilder.ToCompilation()
     at ILCompiler.Program.Run(String[] args)
     at ILCompiler.Program.Main(String[] args)
C:\Users\christianweyer\.nuget\packages\microsoft.dotnet.ilcompiler.llvm\7.0.0-alpha.1.22617.1\build\Microsoft.NETCore.Native.targets(329,5): error MSB3073: The command ""C:\Users\christianweyer\.nuget\packages\runtime.win-x64.microsoft.dotnet.
ilcompiler.llvm\7.0.0-alpha.1.22617.1\tools\\ilc" @"obj\Debug\net7.0\browser-wasm\native\ConsoleApp1.ilc.rsp"" exited with code 1. [C:\Users\christianweyer\source\repos\ConsoleApp1\ConsoleApp1\ConsoleApp1.csproj]

[SingleAccretion]

The host runtime package doesn't have the necessary Jit. This is quite odd; building the package locally puts it there...

[ChristianWeyer]

... how could this be solved?

[SingleAccretion]

With more .yaml fixes, as all things should be.

I think I see what is going on. The runtime package on our hands is the one published by the not for Wasm configuration, so that is why it doesn't have the required cross-targeting Jit.

This is a bit tricky. The way this works, we need to publish two runtime packages: the host one and the WASM one. The WASM one is currently published by the Browser_wasm_win configuration. The host one - by the corresponding host configuration.

One idea could be to build the host package as part of Browser_wasm_win as well. It would be consistent with how the rest of the build is set up (it first builds the WASM part, then the host part). But then we would need to disable the building of host packages from the host configuration (otherwise they'd collide).

@yowl thoughts?

(Edit: I will be offline for the next 10+ hours)

[yowl]

My first thought is how I manage to so completely break things from one merge to the next .

Sent from Outlook for Androidhttps://aka.ms/AAb9ysg

---

From: SingleAccretion @.> Sent: Saturday, December 17, 2022 5:06:13 PM To: dotnet/runtimelab @.> Cc: Scott Waye @.>; Mention @.> Subject: Re: [dotnet/runtimelab] [NativeAOT-LLVM] Publishing Hello World for WebAssembly target on Windows arm64 fails (Issue #2121)

With more .yaml fixes, as all things should be.

I think I see what is going on. The runtime package on our hands is the one published by the not for Wasm configuration, so that is why it doesn't have the required cross-targeting Jit.

This is a bit tricky. The way this works, we need to publish two runtime packages: the host one and the WASM one. The WASM one is currently published by the Browser_wasm_win configuration. The host one - by the corresponding host configuration.

One idea could be to build the host package as part of Browser_wasm_win as well. It would be consistent with how the rest of the build is set up (it first builds the WASM part, then the host part). But then we would need to disable the building of host packages from the host configuration (otherwise they'd collide).

@yowlhttps://github.com/yowl thoughts?

— Reply to this email directly, view it on GitHubhttps://github.com/dotnet/runtimelab/issues/2121#issuecomment-1356485115, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AAUYCKPTJUTDYNME5J2ESDTWNY2NLANCNFSM6AAAAAATA74734. You are receiving this because you were mentioned.Message ID: @.***>

[SingleAccretion]

I went ahead with the proposed fix in #2133.

[hez2010]

I hit the same issue even on x64:

EXEC : error : Unable to load DLL 'clrjit_browser_wasm32_x64' or one of its dependencies: 指定されたモジュールが見つかりません。 (0x8007007E) [D:\source\repos\Test\WasmAotTest\WasmAotTest.csproj]
  System.DllNotFoundException: Unable to load DLL 'clrjit_browser_wasm32_x64' or one of its dependencies: 指定されたモジュールが見つかりません。 (0x8007007E)
     at System.Runtime.InteropServices.NativeLibrary.LoadByName(String libraryName, QCallAssembly callingAssembly, Boolean hasDllImportSearchPathFlag, UInt32 dllImportSearchPathFlag, Boolean throwOnError)
     at System.Runtime.InteropServices.NativeLibrary.LoadLibraryByName(String libraryName, Assembly assembly, Nullable`1 searchPath, Boolean throwOnError)
     at System.Runtime.InteropServices.NativeLibrary.Load(String libraryName, Assembly assembly, Nullable`1 searchPath)
     at Internal.JitInterface.JitConfigProvider.<>c__DisplayClass5_0.<Initialize>b__0(String libName, Assembly assembly, Nullable`1 searchPath)
     at System.Runtime.InteropServices.NativeLibrary.LoadLibraryCallbackStub(String libraryName, Assembly assembly, Boolean hasDllImportSearchPathFlags, UInt32 dllImportSearchPathFlags)
     at Internal.JitInterface.CorInfoImpl.jitStartup(IntPtr host)
     at Internal.JitInterface.CorInfoImpl.Startup(CORINFO_OS os)
     at ILCompiler.LLVMCodegenCompilationBuilder.ToCompilation()
     at ILCompiler.Program.Run(String[] args)
     at ILCompiler.Program.Main(String[] args)

[yowl]

@hez2010 we are working through the publishing problem. We will update here and at https://github.com/dotnet/runtimelab/issues/2120 when we get it working

@jkotas if you get a moment (and are not yet on holiday for xmas), could you see if there was a publish error from the last PR? Many thanks

[yowl]

Please try with the latest packages.

[hez2010]

With the latest packages (7.0.0-dev) it can be published, but the .js and .wasm files didn't get copied from /native to /publish automatically. I also noted that there was 7.0.0-preview.2.22621.1 of Microsoft.DotNet.ILCompiler.LLVM in the package source, but no corresponding version of runtime.win-x64.Microsoft.DotNet.ILCompiler.LLVM.

[ChristianWeyer]

With the latest packages (7.0.0-dev) it can be published, but the .js and .wasm files didn't get copied from /native to /publish automatically. I also noted that there was 7.0.0-preview.2.22621.1 of Microsoft.DotNet.ILCompiler.LLVM in the package source, but no corresponding version of runtime.win-x64.Microsoft.DotNet.ILCompiler.LLVM.

I can confirm all of this.

[yowl]

publish is now copying the wasm and js with the latest packages

[ruby0b]

I'm also running into the issue on linux-x64 using 7.0.0-preview.5.23113.1:

  System.DllNotFoundException: Unable to load shared library 'clrjit_browser_wasm32_x64' or one of its dependencies. In order to help diagnose loading problems, consider using a tool like strace. If you're using glibc, consider setting the LD_DEBUG environment variable: libclrjit_browser_wasm32_x64: cannot open shared object file: No such file or directory
     at System.Runtime.InteropServices.NativeLibrary.<LoadByName>g____PInvoke__|2_0(UInt16* libraryName, QCallAssembly callingAssembly, Int32 hasDllImportSearchPathFlag, UInt32 dllImportSearchPathFlag, Int32 throwOnError)
     at System.Runtime.InteropServices.NativeLibrary.LoadByName(String libraryName, QCallAssembly callingAssembly, Boolean hasDllImportSearchPathFlag, UInt32 dllImportSearchPathFlag, Boolean throwOnError)
     at System.Runtime.InteropServices.NativeLibrary.LoadLibraryByName(String libraryName, Assembly assembly, Nullable`1 searchPath, Boolean throwOnError)
     at Internal.JitInterface.JitConfigProvider.<>c__DisplayClass5_0.<Initialize>b__0(String libName, Assembly assembly, Nullable`1 searchPath)
     at System.Runtime.InteropServices.NativeLibrary.LoadLibraryCallbackStub(String libraryName, Assembly assembly, Boolean hasDllImportSearchPathFlags, UInt32 dllImportSearchPathFlags)
     at Internal.JitInterface.CorInfoImpl.jitStartup(IntPtr host)
     at Internal.JitInterface.CorInfoImpl.Startup(CORINFO_OS os)
     at ILCompiler.RyuJitCompilationBuilder.ToCompilation()
     at ILCompiler.Program.Run(String[] args)
     at ILCompiler.Program.Main(String[] args)

[SingleAccretion]

linux-x64

Unfortunately, we do not currently support compiling on not-Windows. It would require some (believed to be significant) work to make CoreCLR+LLVM build work there (see #1797).