search

ikvmnet / ikvm-maven

Support for adding dependencies on Maven artifacts to .NET projects, using IKVM.
MIT License
52 stars 5 forks source link

MavenReference changes publish behavior for a project with custom satellite assemblies #29

Closed NightOwl888 closed 1 year ago

NightOwl888 commented 1 year ago

Continued from #28 (although unrelated).

Our build makes custom satellite assemblies. We had to go entirely custom because we have cultures that may not be included in .NET. We had to make these satellite assemblies from scratch by using al.exe to do the linking. Another reason we do this is so we don't have to include multiple copies of the resource data in the NuGet package, we simply depend on a single netstandard1.0 targeted NuGet package that has a single copy of the resources that can be used by all target frameworks.

Now, because we do this all outside of MSBuild, it doesn't recognize our satellite assemblies. IIRC, MSBuild will automatically keep a manifest of satellite assemblies in the .deps.json file and that is what it uses to decide which assemblies to include in the publish. If you use a "normal" satellite assembly process it should update that manifest. But since we build them outside of this process, we simply added a copy operation to our project to ensure the satellite assemblies are included whenever a build or publish happens. We only need to worry about this in our own project - after that NuGet manages the satellite assembly copy.

  <ItemGroup Label="Specifies generated sattelite assemblies should be copied to the output folder (and dependent projects as well)">
    <None Include="$(ICU4NSatelliteAssemblyOutputDir)/**/*.resources.dll" CopyToOutputDirectory="PreserveNewest" Visible="false" />
  </ItemGroup>

$(ICU4NSatelliteAssemblyOutputDir) is a single directory that services all target frameworks in the project.

This worked for both build and publish until a reference was added to IKVM.Maven.Sdk with a MavenReference dependency. With that added, upon publish it attempts to copy the satellite assemblies again from the original copy the build does. I suspect that is probably because it ignores the .deps.json manifest and just makes decisions on what is in the obj and/or bin directory when it does the publish.

8>C:\Program Files\dotnet\sdk\7.0.101\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.ConflictResolution.targets(112,5): error NETSDK1152: Found multiple publish output files with the same relative path: F:\Projects\ICU4N\_artifacts\SatelliteAssemblies\Release\af-NA\ICU4N.resources.dll, F:\Projects\ICU4N\src\ICU4N\bin\Release\netstandard2.0\af-NA\ICU4N.resources.dll, F:\Projects\ICU4N\_artifacts\SatelliteAssemblies\Release\af-ZA\ICU4N.resources.dll, F:\Projects\ICU4N\src\ICU4N\bin\Release\netstandard2.0\af-ZA\ICU4N.resources.dll, F:\Projects\ICU4N\_artifacts\SatelliteAssemblies\Release\af\ICU4N.resources.dll, F:\Projects\ICU4N\src\ICU4N\bin\Release\netstandard2.0\af\ICU4N.resources.dll, ...

So, the workaround is simple - just add an optional parameter and pass it to the build during publish.

  <ItemGroup Label="Specifies generated sattelite assemblies should be copied to the output folder (and dependent projects as well)"
             Condition=" '$(IgnoreSatelliteAssemblyCopy)' != 'true' ">
    <None Include="$(ICU4NSatelliteAssemblyOutputDir)/**/*.resources.dll" CopyToOutputDirectory="PreserveNewest" Visible="false" />
  </ItemGroup>
dotnet publish --output "F:\publishTemp" --framework "net6.0" --configuration "Release" --no-build --no-restore --verbosity normal /p:TestAllTargetFrameworks=true /p:IgnoreSatelliteAssemblyCopy=true

With that in place, the publish succeeds. Maybe I am being petty, but it feels like this should work the same way whether I use MavenReference or not. Or does IKVM also not update the .deps.json file from what it does and that is why it just grabs everything?

Apparently, it is possible to disable .deps.jsonfiles usingPreserveCompilationContext` which I haven't tried.

According to the docs:

So, without generating the file, the publish process will grab everything. Does MavenReference respect PreserveCompilationContext or if that is set to false will it try to copy all of the files in the directory twice?

NightOwl888 commented 1 year ago

I take that back. The solution isn't so simple because not all of my published projects depend on MavenReference.

NightOwl888 commented 1 year ago

This is worse than I thought. The publish fails because of the duplicate file error. However, MavenReference doesn't actually copy the satellite assemblies. It just puts their names in a list somewhere that MSBuild checks to ensure there are no duplicates. So, the satellite assemblies still need to be copied manually it just cannot be done with the basic <None Include=""/> element inline with the publish.

This is complicated by the fact that the copy only exists in 1 project and then the copy happens in every project that depends on it. I haven't worked out a way to figure out which project is actually requesting the copy.

wasabii commented 1 year ago

I'll be honest, I am lost about what you're trying to describe. Maven Reference doesn't do anything with your custom copy stuff.

It just makes IkvmReferences.

It doesn't add items to publish. It doesn't add items to build. It doesn't mess with any item groups other than IkvmReference.

It does one other thing: add a POM file to Package files. But that's it.

It doesn't copy any assemblies at all. Satellite or not.

NightOwl888 commented 1 year ago

I suspect something else is going on. It is probably unintentional. Like enabling or disabling a MSBuild feature that causes it to read the files from the bin directory into a list somewhere, and then it uses that list for validation to ensure there are no duplicate destination paths even though no copy operation takes place.

Whatever the case, simply adding IKVM breaks the build and removing it makes it work again.

I don't have time to deal with it now, but after this project is done I will setup a repro project and link it here so you can investigate.

I finally got past the duplicate file issue using:


  <ItemGroup Label="Specifies generated sattelite assemblies should be copied to the output folder (and dependent projects as well)"
             Condition=" '$(_IsPublishing)' != 'true' ">
    <None Include="$(ICU4NSatelliteAssemblyOutputDir)/**/*.resources.dll" CopyToOutputDirectory="PreserveNewest" Visible="false" />
  </ItemGroup>

  <Target Name="PublishSatelliteAssemblies" AfterTargets="Publish" Condition=" '$(_IsPublishing)' == 'true' ">
    <ItemGroup>
      <SatelliteAssemblies Include="$(ICU4NSatelliteAssemblyOutputDir)/**/*.resources.dll"/>
    </ItemGroup>
    <Message Importance="high" Text="Publishing Satellite Assemblies for $(TargetFramework)..."/>

    <Copy SourceFiles="@(SatelliteAssemblies)"
          DestinationFolder="$(PublishDir)"
          SkipUnchangedFiles="true" />
  </Target>

and now I am seeing the original issue happen again.

2023-01-17T18:21:11.7997159Z ##[error]C:\Users\VssAdministrator\.nuget\packages\ikvm\8.4.0\buildTransitive\netstandard2.0\IKVM.Tasks.targets(31,9): Error MSB4018: The "IkvmCompiler" task failed unexpectedly.
System.IO.DirectoryNotFoundException: Could not find a part of the path 'D:\a\1\s\tests\ICU4N.Tests.Collation\obj\Release\net461\ikvm\stage\1\7b2eef3348259756ae2190ad641d7985\icu4j.dll.rsp'.
   at Microsoft.Win32.SafeHandles.SafeFileHandle.CreateFile(String fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options)
   at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize)
   at System.IO.Strategies.OSFileStreamStrategy..ctor(String path, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize)
   at System.IO.File.WriteAllText(String path, String contents)
   at IKVM.Tools.Runner.Compiler.IkvmCompilerLauncher.ExecuteAsync(IkvmCompilerOptions options, Cancellation*** cancellation***) in D:\a\ikvm\ikvm\src\IKVM.Tools.Runner\Compiler\IkvmCompilerLauncher.cs:line 259
   at IKVM.MSBuild.Tasks.IkvmCompiler.ExecuteAsync(IkvmToolTaskDiagnosticWriter writer, Cancellation*** cancellation***) in D:\a\ikvm\ikvm\src\IKVM.MSBuild.Tasks\IkvmCompiler.cs:line 273
   at IKVM.MSBuild.Tasks.IkvmToolExecTask.Execute() in D:\a\ikvm\ikvm\src\IKVM.MSBuild.Tasks\IkvmToolExecTask.cs:line 79
   at Microsoft.Build.BackEnd.TaskExecutionHost.Microsoft.Build.BackEnd.ITaskExecutionHost.Execute()
   at Microsoft.Build.BackEnd.TaskBuilder.ExecuteInstantiatedTask(ITaskExecutionHost taskExecutionHost, TaskLoggingContext taskLoggingContext, TaskHost taskHost, ItemBucket bucket, TaskExecutionMode howToExecuteTask)
2023-01-17T18:21:11.8033493Z      3>C:\Users\VssAdministrator\.nuget\packages\ikvm\8.4.0\buildTransitive\netstandard2.0\IKVM.Tasks.targets(31,9): error MSB4018: The "IkvmCompiler" task failed unexpectedly. [D:\a\1\s\tests\ICU4N.Tests\ICU4N.Tests.csproj]
2023-01-17T18:21:11.8038302Z C:\Users\VssAdministrator\.nuget\packages\ikvm\8.4.0\buildTransitive\netstandard2.0\IKVM.Tasks.targets(31,9): error MSB4018: System.IO.DirectoryNotFoundException: Could not find a part of the path 'D:\a\1\s\tests\ICU4N.Tests.Collation\obj\Release\net461\ikvm\stage\1\7b2eef3348259756ae2190ad641d7985\icu4j.dll.rsp'. [D:\a\1\s\tests\ICU4N.Tests\ICU4N.Tests.csproj]
2023-01-17T18:21:11.8057197Z C:\Users\VssAdministrator\.nuget\packages\ikvm\8.4.0\buildTransitive\netstandard2.0\IKVM.Tasks.targets(31,9): error MSB4018:    at Microsoft.Win32.SafeHandles.SafeFileHandle.CreateFile(String fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options) [D:\a\1\s\tests\ICU4N.Tests\ICU4N.Tests.csproj]
2023-01-17T18:21:11.8066323Z C:\Users\VssAdministrator\.nuget\packages\ikvm\8.4.0\buildTransitive\netstandard2.0\IKVM.Tasks.targets(31,9): error MSB4018:    at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize) [D:\a\1\s\tests\ICU4N.Tests\ICU4N.Tests.csproj]
2023-01-17T18:21:11.8070185Z C:\Users\VssAdministrator\.nuget\packages\ikvm\8.4.0\buildTransitive\netstandard2.0\IKVM.Tasks.targets(31,9): error MSB4018:    at System.IO.Strategies.OSFileStreamStrategy..ctor(String path, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize) [D:\a\1\s\tests\ICU4N.Tests\ICU4N.Tests.csproj]
2023-01-17T18:21:11.8078614Z C:\Users\VssAdministrator\.nuget\packages\ikvm\8.4.0\buildTransitive\netstandard2.0\IKVM.Tasks.targets(31,9): error MSB4018:    at System.IO.File.WriteAllText(String path, String contents) [D:\a\1\s\tests\ICU4N.Tests\ICU4N.Tests.csproj]
2023-01-17T18:21:11.8082119Z C:\Users\VssAdministrator\.nuget\packages\ikvm\8.4.0\buildTransitive\netstandard2.0\IKVM.Tasks.targets(31,9): error MSB4018:    at IKVM.Tools.Runner.Compiler.IkvmCompilerLauncher.ExecuteAsync(IkvmCompilerOptions options, Cancellation*** cancellation***) in D:\a\ikvm\ikvm\src\IKVM.Tools.Runner\Compiler\IkvmCompilerLauncher.cs:line 259 [D:\a\1\s\tests\ICU4N.Tests\ICU4N.Tests.csproj]
2023-01-17T18:21:11.8265423Z C:\Users\VssAdministrator\.nuget\packages\ikvm\8.4.0\buildTransitive\netstandard2.0\IKVM.Tasks.targets(31,9): error MSB4018:    at IKVM.MSBuild.Tasks.IkvmCompiler.ExecuteAsync(IkvmToolTaskDiagnosticWriter writer, Cancellation*** cancellation***) in D:\a\ikvm\ikvm\src\IKVM.MSBuild.Tasks\IkvmCompiler.cs:line 273 [D:\a\1\s\tests\ICU4N.Tests\ICU4N.Tests.csproj]
2023-01-17T18:21:11.8269311Z C:\Users\VssAdministrator\.nuget\packages\ikvm\8.4.0\buildTransitive\netstandard2.0\IKVM.Tasks.targets(31,9): error MSB4018:    at IKVM.MSBuild.Tasks.IkvmToolExecTask.Execute() in D:\a\ikvm\ikvm\src\IKVM.MSBuild.Tasks\IkvmToolExecTask.cs:line 79 [D:\a\1\s\tests\ICU4N.Tests\ICU4N.Tests.csproj]
2023-01-17T18:21:11.8271167Z C:\Users\VssAdministrator\.nuget\packages\ikvm\8.4.0\buildTransitive\netstandard2.0\IKVM.Tasks.targets(31,9): error MSB4018:    at Microsoft.Build.BackEnd.TaskExecutionHost.Microsoft.Build.BackEnd.ITaskExecutionHost.Execute() [D:\a\1\s\tests\ICU4N.Tests\ICU4N.Tests.csproj]
2023-01-17T18:21:11.8273169Z C:\Users\VssAdministrator\.nuget\packages\ikvm\8.4.0\buildTransitive\netstandard2.0\IKVM.Tasks.targets(31,9): error MSB4018:    at Microsoft.Build.BackEnd.TaskBuilder.ExecuteInstantiatedTask(ITaskExecutionHost taskExecutionHost, TaskLoggingContext taskLoggingContext, TaskHost taskHost, ItemBucket bucket, TaskExecutionMode howToExecuteTask) [D:\a\1\s\tests\ICU4N.Tests\ICU4N.Tests.csproj]
2023-01-17T18:21:11.8376620Z      3>Done Building Project "D:\a\1\s\tests\ICU4N.Tests\ICU4N.Tests.csproj" (Build target(s)) -- FAILED.
2023-01-17T18:21:11.8378740Z      1>Done Building Project "D:\a\1\s\ICU4N.sln" (default targets) -- FAILED.
2023-01-17T18:21:11.8379667Z      3>Done Building Project "D:\a\1\s\tests\ICU4N.Tests\ICU4N.Tests.csproj" (default targets) -- FAILED.
2023-01-17T18:21:11.8381228Z      4>Done Building Project "D:\a\1\s\tests\ICU4N.Tests.Collation\ICU4N.Tests.Collation.csproj" (default targets) -- FAILED.
2023-01-17T18:21:11.8382808Z      3>Done Building Project "D:\a\1\s\tests\ICU4N.Tests\ICU4N.Tests.csproj" (Build target(s)) -- FAILED.
2023-01-17T18:21:11.8384376Z      3>Done Building Project "D:\a\1\s\tests\ICU4N.Tests\ICU4N.Tests.csproj" (Build target(s)) -- FAILED.
2023-01-17T18:21:11.8648499Z      4>Done Building Project "D:\a\1\s\tests\ICU4N.Tests.Collation\ICU4N.Tests.Collation.csproj" (Build target(s)) -- FAILED.
2023-01-17T18:21:11.9595325Z      7>Done Building Project "D:\a\1\s\src\ICU4N.TestFramework\ICU4N.TestFramework.csproj" (default targets) -- FAILED.

Not sure why a project that doesn't reference IKVM at all is looking for a .rsp file, but that is another can of worms.

We do all building, packing and publishing at the solution level using the IsPackable and IsPublishable properties to configure the projects that opt in or out of the step. Perhaps that has something to do with this.

wasabii commented 1 year ago

So the error you finally resolved to appears to be from IKVM. Some sort of race condition when it's trying to create the cache directory or something.

NightOwl888 commented 1 year ago

There appears to be at least 2 different things going on here:

  1. When we do dotnet publish with the --no-build option, it appears to be attempting to build the Maven assemblies rather than simply publishing them without building.
  2. IKVM is ignoring the IsPublishable property that we have set when running dotnet publish.

There may also be a race condition, not sure. But the main issue is that we are separating our build into separate steps and IKVM doesn't seem to recognize that publish can be done as an independent step that doesn't do an implicit build.

We have been using a "build once, run everywhere" strategy for our tests. We do a build and a pack prior to doing publish. We don't allow the publish to build again to ensure that we test the same binaries that are in the NuGet package. And due to the fact that we multi-target on multiple projects, we cannot do all of this in one step (publish requires a single TFM and pack requires all TFMs to be built).

Let me put together an example project with the structure that we use to organize our projects and run the builds in Azure DevOps so you can troubleshoot.

wasabii commented 1 year ago

yeah, I don't know about any of that stuff.

IkvmReference isn't exactly complicated. Before RAR, it just makes sure the assemblies exist. It doesn't pay attention to build, compile, etc. It emits <Reference> values, just like anything else. It pays no attention to any of those MSBuild things you're talking about.

But the last error you mentioned was explicit: directory does not exist. We know what that is.

NightOwl888 commented 1 year ago

Okay, I have isolated the issues. They don't seem to always happen, although they happen most of the time. To reproduce, use the ikvm-publish branch. The same setup is on the main branch without IKVM dependencies and it works fine as can be seen here.

The satellite assembly copy issue can be reproduced by running the following commands locally. The output directory must exist prior to running the publish command.


dotnet build --configuration Release --verbosity normal /p:Platform="Any CPU" /p:InformationalVersion=60.1.0 /p:FileVersion=60.1.0.0 /p:PackageVersion=60.1.0 /p:AssemblyVersion=60.1.0.0 /p: /p:TestAllTargetFrameworks=true /p:PortableDebugTypeOnly=true

dotnet publish --output "F:\publishTemp" --framework "net6.0" --configuration "Release" --no-build --no-restore --verbosity normal /p:TestAllTargetFrameworks=true /p:IgnoreSatelliteAssemblyCopy=true

This produces the error(s):

Click me ``` "F:\Projects\_Demos\IkvmPublishDemo\IkvmPublishDemo.sln" (Publish target) (1) -> "F:\Projects\_Demos\IkvmPublishDemo\tests\ICU4N.Tests.Transliterator\ICU4N.Tests.Transliterator.csproj" (P ublish target) (4) -> (CopyTraceDataCollectorArtifacts target) -> F:\Users\shad\.nuget\packages\microsoft.codecoverage\16.1.1\build\netstandard1.0\Microsoft.CodeCoverage. targets(23,5): warning MSB3026: Could not copy "F:\Users\shad\.nuget\packages\microsoft.codecoverage\16.1. 1\build\netstandard1.0\CodeCoverage\amd64\msdia140.dll" to "F:\publishTemp\CodeCoverage\amd64\msdia140.dll ". Beginning retry 1 in 1000ms. The process cannot access the file 'F:\publishTemp\CodeCoverage\amd64\msdi a140.dll' because it is being used by another process. [F:\Projects\_Demos\IkvmPublishDemo\tests\ICU4N.Te sts.Transliterator\ICU4N.Tests.Transliterator.csproj::TargetFramework=net6.0] "F:\Projects\_Demos\IkvmPublishDemo\IkvmPublishDemo.sln" (Publish target) (1) -> "F:\Projects\_Demos\IkvmPublishDemo\tests\ICU4N.Tests.Collation\ICU4N.Tests.Collation.csproj" (Publish tar get) (3) -> F:\Users\shad\.nuget\packages\microsoft.codecoverage\16.1.1\build\netstandard1.0\Microsoft.CodeCoverage. targets(23,5): warning MSB3026: Could not copy "F:\Users\shad\.nuget\packages\microsoft.codecoverage\16.1. 1\build\netstandard1.0\CodeCoverage\amd64\covrun64.dll" to "F:\publishTemp\CodeCoverage\amd64\covrun64.dll ". Beginning retry 1 in 1000ms. The process cannot access the file 'F:\publishTemp\CodeCoverage\amd64\covr un64.dll' because it is being used by another process. [F:\Projects\_Demos\IkvmPublishDemo\tests\ICU4N.Te sts.Collation\ICU4N.Tests.Collation.csproj::TargetFramework=net6.0] "F:\Projects\_Demos\IkvmPublishDemo\IkvmPublishDemo.sln" (Publish target) (1) -> "F:\Projects\_Demos\IkvmPublishDemo\tests\ICU4N.Tests.Transliterator\ICU4N.Tests.Transliterator.csproj" (P ublish target) (4) -> F:\Users\shad\.nuget\packages\microsoft.codecoverage\16.1.1\build\netstandard1.0\Microsoft.CodeCoverage. targets(23,5): warning MSB3026: Could not copy "F:\Users\shad\.nuget\packages\microsoft.codecoverage\16.1. 1\build\netstandard1.0\CodeCoverage\msdia140.dll" to "F:\publishTemp\CodeCoverage\msdia140.dll". Beginning retry 1 in 1000ms. The process cannot access the file 'F:\publishTemp\CodeCoverage\msdia140.dll' because it is being used by another process. [F:\Projects\_Demos\IkvmPublishDemo\tests\ICU4N.Tests.Transliterator \ICU4N.Tests.Transliterator.csproj::TargetFramework=net6.0] "F:\Projects\_Demos\IkvmPublishDemo\IkvmPublishDemo.sln" (Publish target) (1) -> "F:\Projects\_Demos\IkvmPublishDemo\tests\ICU4N.Tests.Collation\ICU4N.Tests.Collation.csproj" (Publish tar get) (3) -> (_HandleFileConflictsForPublish target) -> C:\Program Files\dotnet\sdk\7.0.101\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.ConflictResolution.targ ets(112,5): error NETSDK1152: Found multiple publish output files with the same relative path: F:\Projects \_Demos\IkvmPublishDemo\_artifacts\SatelliteAssemblies\Release\af-NA\ICU4N.resources.dll, F:\Projects\_Dem os\IkvmPublishDemo\src\ICU4N\bin\Release\net6.0\af-NA\ICU4N.resources.dll, F:\Projects\_Demos\IkvmPublishD emo\_artifacts\SatelliteAssemblies\Release\af-ZA\ICU4N.resources.dll, F:\Projects\_Demos\IkvmPublishDemo\s rc\ICU4N\bin\Release\net6.0\af-ZA\ICU4N.resources.dll, F:\Projects\_Demos\IkvmPublishDemo\_artifacts\Satel liteAssemblies\Release\af\ICU4N.resources.dll, F:\Projects\_Demos\IkvmPublishDemo\src\ICU4N\bin\Release\ne t6.0\af\ICU4N.resources.dll. [F:\Projects\_Demos\IkvmPublishDemo\tests\ICU4N.Tests.Collation\ICU4N.Tests.C ollation.csproj::TargetFramework=net6.0] "F:\Projects\_Demos\IkvmPublishDemo\IkvmPublishDemo.sln" (Publish target) (1) -> "F:\Projects\_Demos\IkvmPublishDemo\tests\ICU4N.Tests\ICU4N.Tests.csproj" (Publish target) (5) -> C:\Program Files\dotnet\sdk\7.0.101\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.ConflictResolution.targ ets(112,5): error NETSDK1152: Found multiple publish output files with the same relative path: F:\Projects \_Demos\IkvmPublishDemo\_artifacts\SatelliteAssemblies\Release\af-NA\ICU4N.resources.dll, F:\Projects\_Dem os\IkvmPublishDemo\src\ICU4N\bin\Release\net6.0\af-NA\ICU4N.resources.dll, F:\Projects\_Demos\IkvmPublishD emo\_artifacts\SatelliteAssemblies\Release\af-ZA\ICU4N.resources.dll, F:\Projects\_Demos\IkvmPublishDemo\s rc\ICU4N\bin\Release\net6.0\af-ZA\ICU4N.resources.dll, F:\Projects\_Demos\IkvmPublishDemo\_artifacts\Satel liteAssemblies\Release\af\ICU4N.resources.dll, F:\Projects\_Demos\IkvmPublishDemo\src\ICU4N\bin\Release\ne t6.0\af\ICU4N.resources.dll. [F:\Projects\_Demos\IkvmPublishDemo\tests\ICU4N.Tests\ICU4N.Tests.csproj::Tar getFramework=net6.0] "F:\Projects\_Demos\IkvmPublishDemo\IkvmPublishDemo.sln" (Publish target) (1) -> "F:\Projects\_Demos\IkvmPublishDemo\tests\ICU4N.Tests.Transliterator\ICU4N.Tests.Transliterator.csproj" (P ublish target) (4) -> C:\Program Files\dotnet\sdk\7.0.101\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.ConflictResolution.targ ets(112,5): error NETSDK1152: Found multiple publish output files with the same relative path: F:\Projects \_Demos\IkvmPublishDemo\_artifacts\SatelliteAssemblies\Release\af-NA\ICU4N.resources.dll, F:\Projects\_Dem os\IkvmPublishDemo\src\ICU4N\bin\Release\net6.0\af-NA\ICU4N.resources.dll, F:\Projects\_Demos\IkvmPublishD emo\_artifacts\SatelliteAssemblies\Release\af-ZA\ICU4N.resources.dll, F:\Projects\_Demos\IkvmPublishDemo\s rc\ICU4N\bin\Release\net6.0\af-ZA\ICU4N.resources.dll, F:\Projects\_Demos\IkvmPublishDemo\_artifacts\Satel liteAssemblies\Release\af\ICU4N.resources.dll, F:\Projects\_Demos\IkvmPublishDemo\src\ICU4N\bin\Release\ne t6.0\af\ICU4N.resources.dll. [F:\Projects\_Demos\IkvmPublishDemo\tests\ICU4N.Tests.Transliterator\ICU4N.Te sts.Transliterator.csproj::TargetFramework=net6.0] ```

Indeed, this looks like a race condition when copying files.

Environment

The .rsp issue can be seen on Azure DevOps here.

Or, you can set up your own Azure DevOps pipeline by loading the azure-pipelines.yml file and then running the pipeline based on the ikvm-publish branch. All of the variables have sensible defaults so there is nothing else to configure.

It should be reproducible locally by doing the following replacement in ICU4N.csproj. Note that you don't need to do this change if the above issue is fixed.

Find
  <!-- This is the bit that fails in https://github.com/ikvm-revived/ikvm-maven/issues/29 -->
  <ItemGroup Label="Specifies generated sattelite assemblies should be copied to the output folder (and dependent projects as well)">
    <None Include="$(ICU4NSatelliteAssemblyOutputDir)/**/*.resources.dll" CopyToOutputDirectory="PreserveNewest" Visible="false" />
  </ItemGroup>
Replace with
  <ItemGroup Label="Specifies generated sattelite assemblies should be copied to the output folder (and dependent projects as well)"
             Condition=" '$(_IsPublishing)' != 'true' ">
    <None Include="$(ICU4NSatelliteAssemblyOutputDir)/**/*.resources.dll" CopyToOutputDirectory="PreserveNewest" Visible="false" />
  </ItemGroup>

  <Target Name="PublishSatelliteAssemblies" AfterTargets="Publish" Condition=" '$(_IsPublishing)' == 'true' ">
    <ItemGroup>
      <SatelliteAssemblies Include="$(ICU4NSatelliteAssemblyOutputDir)/**/*.resources.dll"/>
    </ItemGroup>
    <Message Importance="high" Text="Publishing Satellite Assemblies for $(TargetFramework)..."/>

    <Copy SourceFiles="@(SatelliteAssemblies)"
          DestinationFolder="$(PublishDir)"
          SkipUnchangedFiles="true" />
  </Target>

The _IsPublishing property is described here. Do note that it doesn't work when specifying the /t:Publish target directly.

Use the same build and publish commands above to reproduce.

wasabii commented 1 year ago

I can't quite dig apart what's supposed to be broken about what. Are there three issues?

I checked out your branch, and ran your two commands, a dozen times, and they ran fine.

wasabii commented 1 year ago

I can tell from the errors regarding CodeCoverage that you have some multi threading stuff happening that has nothing to do with IKVM......

wasabii commented 1 year ago

And the assembly conflict stuff, though it won't happen for me..... he problem is listed in the error output. You have multiple items in the itemgroup being published with the same relative path.

wasabii commented 1 year ago

In fact, I can see why this would happen. You're publishing a SLN file with multiple projects to the same output directory. That's not supported by the .NET SDK.

NightOwl888 commented 1 year ago

In fact, I can see why this would happen. You're publishing a SLN file with multiple projects to the same output directory. That's not supported by the .NET SDK.

Actually it is. This is the only way to test for real-world dependency conflicts.

The thing is, there is only 1 reference to each of those files. But when doing the publish with IKVM in the mix, it runs the command more than once.

Without IKVM, only the <IsPublishable> projects are supposed to be published. One of the projects that is erroring out ICU4N.TestFramework should not be published at all. But IKVM is trying to look for an .rsp file in the project folder anyway.

And all of the projects (including those with no reference at all to IKVM) are also being scanned for an .rsp file.

wasabii commented 1 year ago

https://github.com/dotnet/sdk/issues/7238

NightOwl888 commented 1 year ago

https://github.com/MicrosoftDocs/visualstudio-docs/issues/6643 - IsPublishable is now a thing (and has been for several years). That link is from 2016.

wasabii commented 1 year ago

And you've set it on three different projects.....

The link from 2016 hasn't been closed yet, since then. It describes exactly what and why it fail and under what situations.

wasabii commented 1 year ago

The gist is that the SDK scripts attempt to run the publish target for each of the projects. And then the relative paths of the output are combined. And then in .NET 6 they introduced an error, which is the very error your receiving, to protect against the situation of duplicate files from diamond dependencies. You are hitting the error they added in .NET 5 to prevent you from doing what you're trying to do.

NightOwl888 commented 1 year ago

https://learn.microsoft.com/en-us/dotnet/core/project-sdk/msbuild-props#ispublishable

This property is useful if you run dotnet publish on a solution file, as it allows automatic selection of projects that should be published.

wasabii commented 1 year ago

Uh huh. But you can only set it on one project at a time. If there are duplicate files between projects. Which you seem to have.

wasabii commented 1 year ago

You have it set in a props file specific to tests/*, which applies to each of the test projects. So it is trying to publish each test project. And since they contain (when you're running it at least) files with duplicate relative output paths, it's erroring.

If you want to clobber files you have to opt in with ErrorOnDuplicatePublishOutputFiles. Though this can have indeterministic results if you don't know exactly what you're doing.

NightOwl888 commented 1 year ago

That makes no sense. The whole idea is that we are publishing all of the projects for a TFM into the same directory. We publish multiple projects together. This even works with dotnet tool publishing just fine. Except it doesn't work with IKVM.

We have been using this same build for half a dozen projects for several years and every new feature thrown at us has worked. Except IKVM.

wasabii commented 1 year ago

Well, I don't know what to tell you. I can read the code of the SDK and tell you how it works. Also, it works perfectly fine for me, with IKVM. But without your satellite assemblies I guess? I have no idea where those are.

NightOwl888 commented 1 year ago

We used to have it set up with a really complicated double loop that had to conditionally run. However, we removed that when I found out about IsPublishable, which works fine on multiple projects at the same time. We just let MSBuild handle that part and it doesn't have any duplication or locking issues. Something that IKVM is doing is causing that.

wasabii commented 1 year ago

Imagine the following structure:

Project A
|                    \
Project B        Project C

If you try to publish ProjectB and ProjectC into the same output directory, they will both include the full closure of files. That means they'll each include both their own files, and ProjectA's files. Since they each have a copy of ProjectA's files, with the same relative output paths, the SDK will error, telling you what it's telling you.

The error exists to prevent a situation where a dependency of B or C is inadvertidly upversioned by ProjectB, replacing the other's version of that file with a version it wasn't expecting.

You can ignore the error by setting the property I mentioned, but then you have the potential of clobbering a file that you didn't mean to. If you know exactly what you're doing, and have no such concerns, it's fine. In fact, we do it in the IKVM source base in a few places because the versions are unified. For instance, the tools and bin drops.

But yeah... I mean... that's what the SDK does.

One proposal in the bug I posted was to make -o publishDir to automatically add a sub directory per project to even prevent you from targeting the same directory. Never got implemented. Nothing really did. The bug remains with no overall solution.

wasabii commented 1 year ago

which works fine on multiple projects at the same time.

It works fine where fine is defined as "if there are duplicate files between the projects and you've specified a specific output path, it will error, but if there aren't, it won't."

NightOwl888 commented 1 year ago

Well, for now, I would say focus on the .rsp file issue. You can simply remove the satellite assembly copy line.

  <!-- This is the bit that fails in https://github.com/ikvm-revived/ikvm-maven/issues/29 -->
  <ItemGroup Label="Specifies generated sattelite assemblies should be copied to the output folder (and dependent projects as well)">
    <None Include="$(ICU4NSatelliteAssemblyOutputDir)/**/*.resources.dll" CopyToOutputDirectory="PreserveNewest" Visible="false" />
  </ItemGroup>

That is new and not tested very well, although it seems to work fine without IKVM. Although, the docs for Generate error for duplicate files in publish output, say that it is only a problem if CopyToOutputDirectory is set to Always and we aren't doing that.

The .rsp file is something that shouldn't even be looked for when doing a dotnet publish --no-build. All we should be doing is copying the already built assemblies to the publish output.

which works fine on multiple projects at the same time.

It works fine where fine is defined as "if there are duplicate files between the projects, it will error, but if there aren't, it won't."

Not true. MSBuild knows how to weed out the duplicate dependencies and doesn't cause any file copy errors in the build output. This is something I checked before switching to IsPublishable instead of looping through each project and publishing each one individually.

wasabii commented 1 year ago

Not true. MSBuild knows how to weed out the duplicate dependencies and doesn't cause any file copy errors in the build output. This is something I checked before switching to IsPublishable instead of looping through each project and publishing each one individually.

Build output yes, not publish output. Build output is unified based on the PackageReference graph. Project B/C upversion based on the most compatible version with their ProjectReferences.

NightOwl888 commented 1 year ago

I meant publish output. Setting IsPublishable on a single project in a solution makes absolutely no sense. That would basically mean there is no advantage at all over publishing at the project level. The whole point of publishing at the solution level is to weed out all of the duplicate publish operations that would end up copying the same files to the publish output multiple times. That is what it is for. But it fails to work as soon as IKVM is referenced. Works fine when IKVM is not referenced.

But again, focus on the .rsp file issue. There is no reason it should even be looking for that file in projects that don't reference IKVM, but it does.

I suspect both of these will end up being similar. IKVM is probably overriding a target that gets executed during both build and publish, but not distinguishing between the two operations.

wasabii commented 1 year ago

I meant publish output. Setting IsPublishable on a single project in a solution makes absolutely no sense. That would basically mean there is no advantage at all over publishing at the project level.

The real problem is the specifics are poorly documented. You can use IsPublishable, on multiple projects, but only if you don't specify PublishDir, or have a different one per project, or don't have overlapping files.

wasabii commented 1 year ago

I suspect both of these will end up being similar. IKVM is probably overriding a target that gets executed during both build and publish, but not distinguishing between the two operations.

We override no targets, and do not distinguish between publish and build. As we should not.

IkvmReference's only goal is to ensure the Reference ItemGroup exists before ResolveAssemblyReferences, and that the assemblies it points to exists. That can happen at Publish just as it can happen at Build. If they already exist in either, nothing should be happening. The .rsp file is simply the list of command line arguments to pass to ikvmc. It's involvement only happens if ikvmc is being executed. The only way ikvmc can be executed here is if the proper assemblies don't already exist.

So if you are truly seeing an attempt to generate IkvmReference assemblies during publish, it probably means they don't exist in the cache path. Are you clearing the cache path between runs or something? Or changing any of the dependencies? Or doing stuff that would be unexpected.

The failure to create the .rsp file is a real problem. There's some actual bug where the directory it wants to put the file in doesn't exist at the time it tries to put the file in it. That's being tracked down.

NightOwl888 commented 1 year ago

I suspect both of these will end up being similar. IKVM is probably overriding a target that gets executed during both build and publish, but not distinguishing between the two operations.

We override no targets, and do not distinguish between publish and build. As we should not.

IkvmReference's only goal is to ensure the Reference ItemGroup exists before ResolveAssemblyReferences, and that the assemblies it points to exists. That can happen at Publish just as it can happen at Build. If they already exist in either, nothing should be happening. The .rsp file is simply the list of command line arguments to pass to ikvmc. It's involvement only happens if ikvmc is being executed. The only way ikvmc can be executed here is if the proper assemblies don't already exist.

So if you are truly seeing an attempt to generate IkvmReference assemblies during publish, it probably means they don't exist in the cache path. Are you clearing the cache path between runs or something? Or changing any of the dependencies? Or doing stuff that would be unexpected.

The failure to create the .rsp file is a real problem. There's some actual bug where the directory it wants to put the file in doesn't exist at the time it tries to put the file in it. That's being tracked down.

No, it is the opposite of this. The build doesn't create the .rsp files because the projects don't reference IKVM. This is correct.

Then when the dotnet publish --no-build command is run, it looks for .rsp files that don't exist. All of the test projects as well as ICU4N.TestFramework are looking for an .rsp file even though they never generated one. It shouldn't even be trying to generate the files because this is not a build it is a publish.

wasabii commented 1 year ago

Then when the dotnet publish --no-build command is run, it looks for .rsp files that don't exist.

Nothing in your errors you've presented looks for an rsp file and fails if it doesn't exist. The messages involving the .rsp file is the code that CREATES the RSP file. What's missing in your error is the DIRECTORY in which the file is supposed to be created.

NightOwl888 commented 1 year ago

And the directory doesn't exist because the project in question doesn't reference IKVM. And it shouldn't exist.

wasabii commented 1 year ago

Does it depend on a project that does? If so it does.

NightOwl888 commented 1 year ago

Yes, the only project that references IKVM is ICU4N.Tests.csproj.

wasabii commented 1 year ago

So it does.... I'm confused. PackageReferences are transitive.

NightOwl888 commented 1 year ago

Oops. I read your question wrong.

No project depends on ICU4N.Tests.csproj. But when doing the publish at the solution level, IKVM is attempting to find an .rsp file in all of the projects in the solution.

NightOwl888 commented 1 year ago

Right. The build fails at that point. Previously I attempted to fix this by making all of the test projects depend on IKVM and com.ibm.icu:icu4j even though they don't need to. That got past the error for the test projects, then it tried to find .rsp files in ICU4N.TestFramework.

I haven't seen it look for ICU4N, but I suspect if I added IKVM to ICU4N.TestFramework (which I can't because it references netstandard2.0) it would look for an .rsp file in ICU4N, also.

wasabii commented 1 year ago

Okay. I have a suspicion here. I think because Response is considered a relative path, and because you're publishing multiple projects at the same time, MSBuild is not properly protecting Environment.CurrentDirectory. So it's a race condition, with differnet projects overwriting each other's values.

wasabii commented 1 year ago

Ahh, no. It's because IkvmCompiler allows Yield to the engine while ikvmc is executing, allowing another simultanious instance of IkvmCompiler to run, and the Environment.CurrentDirectory path might get swapped to a different projects.

NightOwl888 commented 1 year ago

The original issue reported had 2 contributors:

  1. IKVM was somehow attempting to copy the references twice. This was fixed by https://github.com/ikvm-revived/ikvm/issues/282.
  2. There is inconsistent behavior when publishing the satellite assemblies between my local machine and Azure DevOps. I attempted to sync up the SDK version, but it didn't change the results. Still many things to try, but this is purely an MSBuild publish issue, since it happens whether IKVM is referenced or not.