dotnet / runtime

.NET is a cross-platform runtime for cloud, mobile, desktop, and IoT apps.
https://docs.microsoft.com/dotnet/core/
MIT License
15.16k stars 4.72k forks source link

dotnet publish with NativeAOT doesn't copy/bundle native libs from NuGet packages #78191

Closed Sergio0694 closed 1 year ago

Sergio0694 commented 1 year ago

Description

I'm adding NAOT tests to my CI for ComputeSharp as part of updating to the .NET 7 SDK (figured it would be nice to also start validating NAOT builds end to end as part of my pipeline), and I've noticed that my two end to end test projects using native libraries are not working at all with NAOT. That is, using dotnet publish with NAOT results in the .exe being generated that doesn't include the native libs it needs either statically linker nor copied next to it in the output folder. As a result, that .exe is completely unusable (it just crashes at runtime).

I'd expect NAOT to see that a given NuGet package includes native libs, and automatically link them statically into the generated .exe, or at least copy them next to the output .exe in the "native" folder that results from dotnet publish. For context, dotnet build and dotnet run on any other framework (from .NET Framework all the way to .NET 6) work just fine in this case.

Reproduction Steps

  1. Checkout to https://github.com/Sergio0694/ComputeSharp/pull/425
  2. Build the necessary NuGet packages:
    dotnet build src\ComputeSharp.Core.Package\ComputeSharp.Core.Package.msbuildproj -c Release;
    dotnet build src\ComputeSharp.Package\ComputeSharp.Package.msbuildproj -c Release;
    dotnet build src\ComputeSharp.Dynamic.Package\ComputeSharp.Dynamic.Package.msbuildproj -c Release;
  3. Try running the NAOT end to end test for the ComputeSharp.Dynamic.NuGet project:
    dotnet publish tests\ComputeSharp.Dynamic.NuGet\ComputeSharp.Dynamic.NuGet.csproj -c Release -f net7.0 -r win-x64 -v n;
    tests\ComputeSharp.Dynamic.NuGet\bin\Release\net7.0\win-x64\native\ComputeSharp.Dynamic.NuGet.exe

Expected behavior

This should just run correctly.

Actual behavior

Crashes due to a missing native library:

Unhandled Exception: System.DllNotFoundException: Unable to load DLL 'dxcompiler' or one of its dependencies: The specified module could not be found.
   at System.Runtime.InteropServices.NativeLibrary.LoadLibErrorTracker.Throw(String) + 0x79
   at System.Runtime.InteropServices.NativeLibrary.LoadLibraryByName(String, Assembly, Nullable`1, Boolean) + 0x93
   at TerraFX.Interop.Windows.Windows.OnDllImport(String libraryName, Assembly assembly, Nullable`1 searchPath) + 0x4b
   at System.Runtime.InteropServices.NativeLibrary.LoadLibraryCallbackStub(String, Assembly, Boolean, UInt32) + 0x8b
   at Internal.Runtime.CompilerHelpers.InteropHelpers.FixupModuleCell(InteropHelpers.ModuleFixupCell*) + 0x5e
   at Internal.Runtime.CompilerHelpers.InteropHelpers.ResolvePInvokeSlow(InteropHelpers.MethodFixupCell*) + 0x2f
   at TerraFX.Interop.DirectX.DirectX.DxcCreateInstance(Guid*, Guid*, Void**) + 0x2d
   at ComputeSharp.Shaders.Translation.ShaderCompiler..ctor() + 0x53
   at ComputeSharp.Shaders.Translation.ShaderCompiler.get_Instance() + 0x33
   at ComputeSharp.__Internals.ShaderCompiler.LoadDynamicBytecode[TLoader, T](TLoader&, Int32, Int32, Int32, T&) + 0x41
   at ComputeSharp.Shaders.Loading.PipelineDataLoader`1.LoadShader(Int32, Int32, Int32, T&, ICachedShader&) + 0x2a
   at ComputeSharp.Shaders.Loading.PipelineDataLoader`1.GetPipelineData(GraphicsDevice, Int32, Int32, Int32, T&) + 0x99
   at ComputeSharp.ComputeContext.Run[T](Int32, Int32, Int32, Int32, Int32, Int32, T&) + 0x243
   at ComputeSharp.GraphicsDeviceExtensions.For[T](GraphicsDevice, Int32, T&) + 0x70
   at Program.<Main>$(String[]) + 0xaf
   at ComputeSharp.Dynamic.NuGet!<BaseAddress>+0x21374b

The output folder doesn't include the native libs (which are package into the NuGet package), and the .exe is clearly too small to possibly have them statically linked into it (as they are about 20MB in side on their own, but the .exe is 4MB):

ghost commented 1 year ago

Tagging subscribers to this area: @agocke, @MichalStrehovsky, @jkotas See info in area-owners.md if you want to be subscribed.

Issue Details
### Description I'm adding NAOT tests to my CI for ComputeSharp as part of updating to the .NET 7 SDK (figured it would be nice to also start validating NAOT builds end to end as part of my pipeline), and I've noticed that my two end to end test projects using native libraries are not working at all with NAOT. That is, using `dotnet publish` with NAOT results in the .exe being generated that doesn't include the native libs it needs either statically linker nor copied next to it in the output folder. As a result, that .exe is completely unusable (it just crashes at runtime). I'd expect NAOT to see that a given NuGet package includes native libs, and automatically link them statically into the generated .exe, or at least copy them next to the output .exe in the "native" folder that results from `dotnet publish`. For context, `dotnet build` and `dotnet run` on any other framework (from .NET Framework all the way to .NET 6) work just fine in this case. ### Reproduction Steps 1. Checkout to https://github.com/Sergio0694/ComputeSharp/pull/425 2. Build the necessary NuGet packages: ``` dotnet build src\ComputeSharp.Core.Package\ComputeSharp.Core.Package.msbuildproj -c Release; dotnet build src\ComputeSharp.Package\ComputeSharp.Package.msbuildproj -c Release; dotnet build src\ComputeSharp.Dynamic.Package\ComputeSharp.Dynamic.Package.msbuildproj -c Release; ``` 3. Try running the NAOT end to end test for the ComputeSharp.Dynamic.NuGet project: ``` dotnet publish tests\ComputeSharp.Dynamic.NuGet\ComputeSharp.Dynamic.NuGet.csproj -c Release -f net7.0 -r win-x64 -v n; tests\ComputeSharp.Dynamic.NuGet\bin\Release\net7.0\win-x64\native\ComputeSharp.Dynamic.NuGet.exe ``` ### Expected behavior This should just run correctly. ### Actual behavior Crashes due to a missing native library: ``` Unhandled Exception: System.DllNotFoundException: Unable to load DLL 'dxcompiler' or one of its dependencies: The specified module could not be found. at System.Runtime.InteropServices.NativeLibrary.LoadLibErrorTracker.Throw(String) + 0x79 at System.Runtime.InteropServices.NativeLibrary.LoadLibraryByName(String, Assembly, Nullable`1, Boolean) + 0x93 at TerraFX.Interop.Windows.Windows.OnDllImport(String libraryName, Assembly assembly, Nullable`1 searchPath) + 0x4b at System.Runtime.InteropServices.NativeLibrary.LoadLibraryCallbackStub(String, Assembly, Boolean, UInt32) + 0x8b at Internal.Runtime.CompilerHelpers.InteropHelpers.FixupModuleCell(InteropHelpers.ModuleFixupCell*) + 0x5e at Internal.Runtime.CompilerHelpers.InteropHelpers.ResolvePInvokeSlow(InteropHelpers.MethodFixupCell*) + 0x2f at TerraFX.Interop.DirectX.DirectX.DxcCreateInstance(Guid*, Guid*, Void**) + 0x2d at ComputeSharp.Shaders.Translation.ShaderCompiler..ctor() + 0x53 at ComputeSharp.Shaders.Translation.ShaderCompiler.get_Instance() + 0x33 at ComputeSharp.__Internals.ShaderCompiler.LoadDynamicBytecode[TLoader, T](TLoader&, Int32, Int32, Int32, T&) + 0x41 at ComputeSharp.Shaders.Loading.PipelineDataLoader`1.LoadShader(Int32, Int32, Int32, T&, ICachedShader&) + 0x2a at ComputeSharp.Shaders.Loading.PipelineDataLoader`1.GetPipelineData(GraphicsDevice, Int32, Int32, Int32, T&) + 0x99 at ComputeSharp.ComputeContext.Run[T](Int32, Int32, Int32, Int32, Int32, Int32, T&) + 0x243 at ComputeSharp.GraphicsDeviceExtensions.For[T](GraphicsDevice, Int32, T&) + 0x70 at Program.
$(String[]) + 0xaf at ComputeSharp.Dynamic.NuGet!+0x21374b ``` The output folder doesn't include the native libs (which are package into the NuGet package), and the .exe is clearly too small to possibly have them statically linked into it (as they are about 20MB in side on their own, but the .exe is 4MB): ![](https://user-images.githubusercontent.com/10199417/201195422-2fcfaadd-a8c4-4d7d-ab91-faf014c75a86.png)
Author: Sergio0694
Assignees: -
Labels: `area-Interop-coreclr`, `untriaged`, `area-NativeAOT-coreclr`
Milestone: -
jkotas commented 1 year ago

tests\ComputeSharp.Dynamic.NuGet\bin\Release\net7.0\win-x64\native

This needs to be publish directory. native directory is just for intermediate files, it does not contain everything.

We actually have opposite problem: There are things in the publish directory that are not supposed to be there.

Sergio0694 commented 1 year ago

Ooh my bad, I incorrectly thought "native" was the right folder for the output 😅 Yeah then "publish" actually seems correct, and everything works just fine:

Sorry! 😄