"dotnet publish" (Linux, MSBuild GenerateBundle task) does not include assembly metadata in PE binary for self-contained win-x64 console application

sps-nathan-alden-sr commented 2 years ago

Describe the bug

I have an Azure Pipelines release pipeline for a .NET 6 console application. My build agent is a self-hosted Ubuntu 20.04 LTS server. The pipeline is installing .NET SDK 6.0.101 before executing any other steps. I am pushing published releases to GitHub. I noticed my published application was crashing because it was attempting to read assembly metadata that appears to have been stripped from the assembly during the dotnet publish step, which is configured to perform a self-contained publish. Because I control the build agent, I was able to perform an extensive investigation to narrow down the problem.

`dotnet build`

Earlier in the pipeline, I perform the following dotnet build command:

/home/nathanaldensr/azure-pipelines-agent/_work/_tool/dotnet/dotnet build /home/nathanaldensr/azure-pipelines-agent/_work/2/s/Redacted.DatabaseUtilities.sln -dl:CentralLogger,"/home/nathanaldensr/azure-pipelines-agent/_work/_tasks/DotNetCoreCLI_5541a522-603c-47ad-91fc-a4b1d163081b/2.198.0/dotnet-build-helpers/Microsoft.TeamFoundation.DistributedTask.MSBuild.Logger.dll"*ForwardingLogger,"/home/nathanaldensr/azure-pipelines-agent/_work/_tasks/DotNetCoreCLI_5541a522-603c-47ad-91fc-a4b1d163081b/2.198.0/dotnet-build-helpers/Microsoft.TeamFoundation.DistributedTask.MSBuild.Logger.dll" --configuration Release --no-restore --verbosity detailed

This results in an assembly being created for the console application at ~/azure-pipelines-agent/_work/2/s/artifacts/bin/Redacted.DatabaseUtilities.PostgreSql/Release/net6.0/Redacted.DatabaseUtilities.PostgreSql.dll.

I downloaded this file to my PC and opened it in Visual Studio to inspect the PE information and also JetBrains Assembly Explorer to inspect the assembly metadata:

[assembly: CompilationRelaxations(8)]
[assembly: RuntimeCompatibility(WrapNonExceptionThrows = true)]
[assembly: Debuggable(DebuggableAttribute.DebuggingModes.IgnoreSymbolStoreSequencePoints)]
[assembly: TargetFramework(".NETCoreApp,Version=v6.0", FrameworkDisplayName = "")]
[assembly: AssemblyCompany("Redacted")]
[assembly: AssemblyConfiguration("Release")]
[assembly: AssemblyCopyright("Copyright (c) Redacted.")]
[assembly: AssemblyFileVersion("1.0.222.8005")]
[assembly: AssemblyInformationalVersion("1.0.2+f9988f5c152416e949d1de30d643279217c1e237")]
[assembly: AssemblyProduct("Database Utilities")]
[assembly: AssemblyTitle("Redacted.DatabaseUtilities.PostgreSql")]
[assembly: AssemblyMetadata("RepositoryUrl", "https://github.com/redacted/databaseutilities.git")]
[assembly: NeutralResourcesLanguage("en-US")]
[assembly: AssemblyVersion("1.0.0.0")]

As you can see, for the .NET assembly itself, everything is perfect.

`dotnet publish`

Here's where the problem lies, I believe. Here is the dotnet publish command as shown by the Azure Pipelines log:

/home/nathanaldensr/azure-pipelines-agent/_work/_tool/dotnet/dotnet publish /home/nathanaldensr/azure-pipelines-agent/_work/2/s/source/PostgreSql/Redacted.DatabaseUtilities.PostgreSql.csproj --configuration Release --runtime win-x64 --self-contained true --verbosity detailed -p:PublishSingleFile=true

This results in a self-contained PE binary being created at ~/azure-pipelines-agent/_work/2/s/artifacts/bin/Redacted.DatabaseUtilities.PostgreSql/Release/net6.0/win-x64/publish/Redacted.DatabaseUtilities.PostgreSql.exe.

I downloaded this file to my PC and opened it in Visual Studio to inspect the PE information:

As you can see, there is no version information whatsoever. Windows Explorer confirms this:

Here's the strange thing: if I run dotnet publish on my local Windows 10 PC, the executable contains the correct PE information!

PS > dotnet publish .\source\PostgreSql\Redacted.DatabaseUtilities.PostgreSql.csproj --configuration Release --runtime win-x64 --self-contained true -p:PublishSingleFile=true
Microsoft (R) Build Engine version 17.0.0+c9eb9dd64 for .NET
Copyright (C) Microsoft Corporation. All rights reserved.

  Determining projects to restore...
  Restored D:\github\redacted\databaseutilities\source\PostgreSql\Redacted.DatabaseUtilities.PostgreSql.csproj (in 331 ms).
  Redacted.DatabaseUtilities.PostgreSql -> D:\github\redacted\databaseutilities\artifacts\bin\Redacted.DatabaseUtilities.PostgreSql\Release\net6.0\win-x64\Redacted.DatabaseUtilities.PostgreSql.dll
  Redacted.DatabaseUtilities.PostgreSql -> D:\github\redacted\databaseutilities\artifacts\bin\Redacted.DatabaseUtilities.PostgreSql\Release\net6.0\win-x64\publish\

I ran the same CLI command through SSH, but this time I specified --verbosity diagnostic -bl:msbuild.binlog. Then I downloaded the binlog and viewed it with MSBuild Structured Log Viewer. I located the GenerateSingleFileBundle task and, inside it, the GenerateBundle task, and found this:

It appears that the file being included in the single file bundle is not the same file as the one output to bin by dotnet build. I downloaded the .dll selected in the screenshot and, sure enough, Windows Explorer shows the expected PE information is present. In other words, given the actual files being used by GenerateBundle, I can't see how this isn't a bug.

Here are some additional input parameters to the GenerateBundle task:

Unfortunately, further behavior of dotnet publish is opaque to me, so I'm not sure what other information I can gather. This appears to be a bug with dotnet publish in the context of Linux (hard for me to tell for sure).

I can upload the binlog somewhere as long as it's private and secure.

KalleOlaviNiemitalo commented 2 years ago

See https://github.com/dotnet/runtime/issues/3828.

KalleOlaviNiemitalo commented 2 years ago

I noticed my published application was crashing because it was attempting to read assembly metadata

Which API do you use for this:

Attribute.GetCustomAttributes - examining attributes of an assembly that has been loaded by the CLR. I would expect this to find the AssemblyInformationalVersionAttribute etc. no matter how your app is built and bundled.
MetadataReader.GetCustomAttributes - examining attributes of an assembly that your app opened from a file. I'm not sure this can work if you open a bundle of multiple assemblies.
FileVersionInfo.GetVersionInfo - examining the Win32 VERSIONINFO resource of a file. This would depend on the build system copying the Win32 resources from the DLL to the EXE, and AFAIK that uses the Win32 API and doesn't work when you build on Linux; see the issue I linked above.

sps-nathan-alden-sr commented 2 years ago

I don't want to get distracted with that conversation since it's orthogonal to the problem here. It honestly doesn't matter how my app determines this information; what matters is the information is simply not present in the bundle executable.

That said, I am using this code, which works perfectly when the binary contains the PE information:

var version = FileVersionInfo.GetVersionInfo(Environment.ProcessPath!).ProductVersion!;

I will take a look at the issue you referenced.

sps-nathan-alden-sr commented 2 years ago

It looks like there are many issues related to this exact problem. It would've been great if Microsoft had added a build error when building win-x64 on Linux as there are clearly unsupported scenarios that are never even warned about in the build logs.

sps-nathan-alden-sr commented 2 years ago

@KalleOlaviNiemitalo Thanks for pointing me in the right direction. It seems I'm stuck for now--forced to use a Windows build agent for my builds.

KalleOlaviNiemitalo commented 2 years ago

Can you change your application to use Attribute.GetCustomAttributes, perhaps with Assembly.GetEntryAssembly? That way, the version lookup would not depend on Win32 resources.

If you instead need to publish a VERSIONINFO resource for other software to read (e.g. to let Windows Installer decide whether it should replace a file), that's harder. Some possibilities come to mind:

GNU binutils might include a program capable of copying resources between PE files. I didn't find one in the documentation, though.
Implement a custom utility that uses the Win32 API to copy the resources. Your CI system could still build the projects on lower-cost Linux agents but then run the resource-copying tool on Windows. Or perhaps it could even be run on Wine.
Implement the resource copying tool based on the PE specification without involving the Win32 API. This is what .NET should eventually do anyway but if you do it as a separate tool then you can have it sooner.

sps-nathan-alden-sr commented 2 years ago

I had changed my code to the FileVersionInfo method because of some information I read online about single-file executables. However, I changed my code to Assembly.GetEntryAssembly() and tested with a published executable and it worked. So it seems I can at least mitigate the issue with this new code.

marcpopMSFT commented 2 years ago

Per comment above, the SDK can't resolve this until https://github.com/dotnet/runtime/issues/3828 is fixed and we may not require any work once that's solved.

KalleOlaviNiemitalo commented 10 months ago

https://github.com/dotnet/runtime/issues/3828 has been fixed.

dotnet / sdk