search

microsoft / vcrt-forwarders

App-local DLLs that forward to Visual C++ runtime components
MIT License
32 stars 17 forks source link

PublishSingleFile error on .NET Core 3.0 Preview 8 "Invalid input specification: Found multiple entries with the same BundleRelativePath" #8

Open sudoudaisuke opened 5 years ago

sudoudaisuke commented 5 years ago

Hello. I got error dotnet publish with vcrt-forwarders. Is this dotnet/core-setup bug?

[environment] .NET Core 3.0 Preview 8 Microsoft.VCRTForwarders.140 Version=1.0.1-rc csproj has "PublishSingleFile" element Console app

[command] dotnet publish ConsoleApp1.csproj /p:PublishProfile=\PublishProfiles\FolderProfile.pubxml

[error message]

C:\Program Files (x86)\dotnet\sdk\3.0.100-preview8-013656\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.Publish.targets(807,5): error MSB4018: "GenerateBundle" task failed unexpectedly. [  ConsoleApp1\ConsoleApp1.csproj]
C:\Program Files (x86)\dotnet\sdk\3.0.100-preview8-013656\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.Publish.targets(807,5): error MSB4018: System.ArgumentException: Invalid input specification: Found multiple entries with the same BundleRelativePath [  ConsoleApp1\ConsoleApp1.csproj]
C:\Program Files (x86)\dotnet\sdk\3.0.100-preview8-013656\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.Publish.targets(807,5): error MSB4018:    at Microsoft.NET.HostModel.Bundle.Bundler.GenerateBundle(IReadOnlyList`1 fileSpecs) [  ConsoleApp1\ConsoleApp1.csproj]
C:\Program Files (x86)\dotnet\sdk\3.0.100-preview8-013656\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.Publish.targets(807,5): error MSB4018:    at Microsoft.NET.Build.Tasks.GenerateBundle.ExecuteCore() in /_/src/Tasks/Microsoft.NET.Build.Tasks/GenerateBundle.cs:line 35 [    ConsoleApp1\ConsoleApp1.csproj]
C:\Program Files (x86)\dotnet\sdk\3.0.100-preview8-013656\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.Publish.targets(807,5): error MSB4018:    at Microsoft.NET.Build.Tasks.TaskBase.Execute() in /_/src/Tasks/Common/TaskBase.cs:line 38 [   ConsoleApp1\ConsoleApp1.csproj]
C:\Program Files (x86)\dotnet\sdk\3.0.100-preview8-013656\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.Publish.targets(807,5): error MSB4018:    at Microsoft.Build.BackEnd.TaskExecutionHost.Microsoft.Build.BackEnd.ITaskExecutionHost.Execute() [    ConsoleApp1\ConsoleApp1.csproj]
C:\Program Files (x86)\dotnet\sdk\3.0.100-preview8-013656\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.Publish.targets(807,5): error MSB4018:    at Microsoft.Build.BackEnd.TaskBuilder.ExecuteInstantiatedTask(ITaskExecutionHost taskExecutionHost, TaskLoggingContext taskLoggingContext, TaskHost taskHost, ItemBucket bucket, TaskExecutionMode howToExecuteTask) [    ConsoleApp1\ConsoleApp1.csproj]

[Sample csproj] ConsoleApp1.zip

sudoudaisuke commented 5 years ago

Sorry, move this issue to dotnet/core-setup#7746.

vitek-karas commented 5 years ago

Please see dotnet/core-setup#7746 for more details: The targets file this package includes adds the duplicate items to the ReferenceCopyLocalPaths, and only does so if the Platform property is defined. So simple publish from command line actually doesn't invoke those targets - which in itself is probably a bug.

Additionally the native assets (the facades in this case) will be added by NuGet runtime resolution anyway, so there should be no need to add a custom build task to copy them again.

The targets file also has a minor issue with the error message (it supports ARM64, but the error message states x86 and x64 only).

manodasanW commented 5 years ago

I spent some time looking at this and I agree that if we remove our custom ReferenceCopyLocalPaths, dotnet core does copy our specific nuget references during publishing. But, this seems to break the scenario where you just build and hit F5 as dotnet core seems to copy the entire runtimes folder from our nuget with binaries for each runtime rather than copying the one for the specific runtime being built for.

@vitek-karas Do you know of any reason why dotnet core does this when building and how we can get it to copy the specific references similar to what it does during publishing and basically what we were trying to achieve with our custom copy commands.

vitek-karas commented 5 years ago

By default .NET Core builds (F5 runs build for example) a "portable app" - that is app which can be copied as is and runs on any platform. In order to do this, it needs the native (RID specific) assets for all supported platforms.

Note that these will be in RID specific subfolders. The .deps.json file contains records for all these assets, specifying in which case to use which assets. So it contains something like "When running on linux-x64, use the asset from linux-x64 folder" and "When running on win-x64 use the one from win-x64". At runtime the host will pick the right ones for the platform it's running on.

When you publish for a specific platform (RID, typically done via dotnet publish -r linux-x64), this asset resolution is done by NuGet and the output will only get assets for the specified RID. There are no subdirectories in this case - all the files are in the output directly. The .deps.json file is still there, but it's basically just listing the files in the directory - no special magic.

So the systems assumes that the NuGet package has assets:

The resolution then uses all those which are not runtime specific and then for all other assets it picks the one copy which is the best match for the given platform.

What is so special about these native libraries, that they have to live in the main output folder and not in subfolders for "portable apps"?

manodasanW commented 5 years ago

@vitek-karas Thanks for that explanation. The difference between build and publish makes sense now.

The issue is that in the build / F5 scenario, I am not seeing our native nuget libraries get resolved from the runtimes folder when a DLL that depends on it is loaded by LoadLibrary. Based on loader snaps, it seems the native dependency directories from .deps.json are not on the search path for LoadLibrary to find. I was able to confirm by setting COREHOST_TRACE that the folder with the dependency is detected by the host and is part of NATIVE_DLL_SEARCH_DIRECTORIES, but I guess those directories are not included as a search directory for LoadLibrary.

Are the native dependency directories in .deps.json for the current RID expected to be included as part of search path for LoadLibrary by the host and if not, how are those dependencies expected to be resolved by LoadLibrary?

vitek-karas commented 5 years ago

@swaroop-sridhar can you please take a look at this one?

swaroop-sridhar commented 5 years ago

@manodasanW: Do I understand the following correctly:

If so, can you please share a repro/instructions for the second issue? Thanks.

manodasanW commented 5 years ago

@swaroop-sridhar Right, we are able to resolve the issue with PublishSingleFile by removing ReferenceCopyLocalPaths, but we had put that there due to this 2nd problem where in a portable app our native nuget libraries are not picked up from the runtimes folder by LoadLibrary.

Attached is a sample which should let you observe the issue. You should set ConsoleApp to be the startup project and you should delete all the ReferenceCopyLocalPath entires from our vcrt-forwarders target file for .net core (.nuget\packages\microsoft.vcrtforwarders.140\1.0.2-rc\build\netcoreapp2.0\Microsoft.VCRTForwarders.140.targets). Once you build it and run the console app you should observe a file not found error and this is due to our native nuget libraries under the runtimes folder are not getting picked up. With the ReferenceCopyLocalPath we had previously, they would have been copied alongside the app and be picked up.

I also built the app and attached the binaries which you should also be able to run and observe the issue with.

Sample.zip x64.zip

swaroop-sridhar commented 5 years ago

Thanks for sharing the repro @manodasanW. I also talked to @vitek-karas and @AaronRobinsonMSFT about the issue.

The resolution of the native library loads from within the WinRT component is not handled by the .net core runtime. The libraries need to be in the same directory, or a directory understood by LoadLibrary() or have a custom mechanism built-in to locate the DLL.

In order for ReferenceCopyLocalPaths route to work, we need to fix dotnet/sdk#3510.

CC: @nguerrera

claudioscatoli commented 3 years ago

Hi, i know this is an old issue, but I stumbled upon it and was desperate to find a solution. In the end, i tried deleting 1 by 1 the nuget packages in my .csproj, and I found out that these 2 were the culprits!

`

    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="16.8.3" />
    <PackageReference Include="BenchmarkDotNet" Version="0.12.1" />

`

I don't know if it's a matter of compatibility, but having even just one of those packages (either one) resulted in this issue's error while publishing.

Here's the full list of packages of the project

`

<ItemGroup>  
    <PackageReference Include="App.Metrics.AspNetCore.Health.Endpoints" Version="3.2.0" />
    <PackageReference Include="AWSSDK.Core" Version="3.3.107.40" />  
    <PackageReference Include="BenchmarkDotNet" Version="0.12.1" />  
    <PackageReference Include="FluentFTP" Version="33.0.3" />  
    <PackageReference Include="JWT" Version="7.3.1" />  
    <PackageReference Include="Marten" Version="4.0.0-alpha.9" />
    <PackageReference Include="MassTransit" Version="7.1.3" />
    <PackageReference Include="MassTransit.AmazonSQS" Version="7.1.3" />
    <PackageReference Include="MassTransit.AspNetCore" Version="7.1.3" />
    <PackageReference Include="MediatR" Version="9.0.0" />
    <PackageReference Include="MediatR.Extensions.Microsoft.DependencyInjection" Version="9.0.0" />
    <PackageReference Include="Microsoft.AspNetCore.Authentication.JwtBearer" Version="5.0.2" />
    <PackageReference Include="Microsoft.AspNetCore.Mvc.NewtonsoftJson" Version="5.0.2" />
    <PackageReference Include="Microsoft.AspNetCore.Mvc.Testing" Version="5.0.2" />
    <PackageReference Include="Microsoft.EntityFrameworkCore.Sqlite" Version="5.0.2" />
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="16.8.3" />
    <PackageReference Include="BenchmarkDotNet" Version="0.12.1" />
    <PackageReference Include="Minio" Version="3.1.13" />
    <PackageReference Include="Newtonsoft.Json" Version="12.0.3" />
    <PackageReference Include="Npgsql.EntityFrameworkCore.PostgreSQL" Version="5.0.2" />
    <PackageReference Include="Npgsql.Json.NET" Version="5.0.3" />
    <PackageReference Include="PuppeteerSharp" Version="2.0.4" />
    <PackageReference Include="Quartz" Version="3.2.4" />
    <PackageReference Include="Quartz.Extensions.DependencyInjection" Version="3.2.4" />
    <PackageReference Include="Quartz.Extensions.Hosting" Version="3.2.4" />
    <PackageReference Include="Serilog" Version="2.10.0" />
    <PackageReference Include="Serilog.AspNetCore" Version="3.4.0" />
    <PackageReference Include="Serilog.Extensions.Logging.File" Version="2.0.0" />
    <PackageReference Include="Serilog.Sinks.File" Version="5.0.0-dev-00909" />
    <PackageReference Include="Serilog.Sinks.PostgreSQL" Version="2.1.0" />
    <PackageReference Include="System.Net.Http.Json" Version="3.2.1" />
    <PackageReference Include="System.Threading.AccessControl" Version="5.0.0" />
    <PackageReference Include="xunit" Version="2.4.1" />
</ItemGroup>

`

I tried removing all the source code (leaving only Program.cs and Startup.cs) along with all these packages (leaving only either "Microsoft.NET.Test.Sdk" or "BenchmarkDotNet") and it still gives the same error when publishing as a single file.

Here's some info on my system immagine

I hope this helps somebody!

vitek-karas commented 3 years ago

This should be improved or maybe even fixed in .NET 6. https://github.com/dotnet/runtime/pull/48336 adds a better error message exactly specifying which two file paths caused the problem (should help figuring out where it came from) https://github.com/dotnet/runtime/pull/50476 avoids the error if the input contains exact duplicates (which is by far the most common case for this error as far as we know).

Note that these fixes are all in the SDK (not in the runtime, ignore the repo). So you can try to build your app with .NET 6 Preview 3 SDK and it should fix it.