[Bug]: Build with references fails if absolute path contains a symlink

next-mad-hatter commented 8 months ago

Issue Description

A project containing references will fail to build/find reference assemblies if built using relative paths first.

Steps to Reproduce

Attached minimal sample dotnet-project.zip contains two projects -- main referencing lib. The error is reproducible whenever path to filesystem root contains a symlink:

#!/bin/sh
cd ~ && mkdir my-projects && cd my-projects

# either 
# unzip ~/dotnet-project.zip 
# or
mkdir dotnet-project && cd dotnet-project
dotnet new console -o main
dotnet new classlib -o lib
dotnet add main reference lib

cd ~ && ln -s my-projects/dotnet-project . && cd dotnet-project

dotnet build main/main.csproj
dotnet build `pwd`/main/main.csproj

Expected Behavior

Second invocation of dotnet build should succeed.

Actual Behavior

Second build above fails with following message:

CSC : error CS0006: Metadata file '/home/user/dotnet-project/lib/obj/Debug/net6.0/ref/lib.dll' could not be found [/home/user/dotnet-project/main/main.csproj]

Second invocation will succeed though.

While inconvenient in general this also breaks C# tooling in VS Code (I cannot compile/run/debug tests whenever a path to a project contains a symlink).

Analysis

For what it's worth, with the failed build above, the lib/obj/Debug/net6.0/ref/lib.dll is the only one not being created -- both lib/obj/Debug/net6.0/refint/lib.dll lib/bin/Debug/net6.0/lib.dll are still being built successfully.

Versions & Configurations

$ DOTNET_CLI_UI_LANGUAGE=en dotnet msbuild -version
MSBuild version 17.8.3+195e7f5a3 for .NET
17.8.3.51904%

$ echo $SHELL
/bin/zsh

$ /bin/zsh --version
zsh 5.9 (x86_64-unknown-linux-gnu)

f-alizada commented 7 months ago

Hello @next-mad-hatter, thank you for raising the issue! Initial look at the repro logs: The file lib.csproj.FileListAbsolute.txt does not contain the ref/lib.dll line when invoked with pwd (even if it was produced), however presents in file when invoked using relative path. This results the IncrementalClean to delete the 'outdated and not needed file'. when invoked second time since the file is not stored in the FileListAbsolute therefore it is not removed from IncrementalClean step and build succeeds So the next steps to understand why it is not reported.

f-alizada commented 7 months ago

`

` In case the FileWrites list contains the item with absolute path which has symblink it is not being considered under IntermediateOutputPath. however both of paths logically correct and are the same.

f-alizada commented 7 months ago

In summary:

TargetRefPath is generated and contains the link with symlink in it
One of the steps of IncrementalClean is to remove the not needed files from previous build
While searching /something1/something2/symlink-to-something3/file.dll under the directory /something1/something2/something-3/file.dll fails to find it based on the logic of comparing . (both links are to the same file one contains symlink another not)
Therefore reports file.dll as to be deleted

When invoked second time this does not happen, since the lib.dll is copied to output directory but not reported in files to be deleted results green build.

Having that there is a workarounds like running the command with the full path to project , or rerunning the command @next-mad-hatter please correct me if it is not the case. A solution to the bug itself that I was thinking is to update the logic of FindUnderPath task to respect the symlinks while comparing the file's directories. Unless there is a much simplier way to fix that. @rainersigwald could you please take a look on the investigation results and let me know if there anything else could be done in this scenario? :)

next-mad-hatter commented 7 months ago

@f-alizada Thank you for looking into this.

Yes, running the failed command succeeds on second attempt. Unfortunately, this does not work with the c#-dev-kit toolchain (I have not looked into that further though).

And yes, using the hard path will work -- if one knows that this bug exists.

f-alizada commented 7 months ago

Took a second look at the problem. The Solution provided in previous investigation will result performance issues in future (thank you @rainersigwald ), So capturing the path where the flow breaks:

RefLibFullDLL = /home/user/my-projects/dotnet-project/lib/obj/Debug/net6.0/ref/lib.dll(without symlink) RefLibDLL = /home/user/dotnet-project/lib/obj/Debug/net6.0/ref/lib.dll (with symlink) FileWrites =[];

The TargetRefPath when project file provided as a full path will be left untouched. TargetRefPath = RefLibDLL

Target CopyFilesToOutputDirectory Task CopyRefAssembly populates the Items FileWrites after copying FileWrites + RefDLL = [...,RefDLL,...]

Flow of refFullLibDll and RefLibDLL in IncrementalClean flow: _CleanPriorFileWrites = [..., RefLibFullDLL, ...] _CleanCurrentFileWritesInIntermediate = [...,RefLibDLL,... ] _CleanCurrentFileWritesWithNoReferences = _CleanCurrentFileWritesInIntermediate + [...] = [...,RefLibDLL,... ] _CleanCurrentFileWrites = RemoveDuplicates(_CleanCurrentFileWritesWithNoReferences) = [...,RefLibDLL,... ] _CleanOrphanFileWrites = _CleanPriorFileWrites - _CleanCurrentFileWrites = [..., RefLibFullDLL,...] _CleanOrphanFileWritesInOutput = FindUnderPath("obj/Debug/net6.0/", _CleanOrphanFileWrites) = [..., RefLibFullDLL,...] Delete(Files= _CleanOrphanFileWritesInOutput) -> deletes RefLibFullDLL However the RefLibFullDll == RefLibDLL.

@baronfel FYI regarding the c#-dev-kit just in case.

f-alizada commented 7 months ago

Did some research on possible solutions. 1 - Common fix: We can change the way how do we determine the TargetRefPath -> which may result changes into the flow of fetching msbuildproject directory => to follow the symlink and save only target folder. 2 - Local fix would be to apply common fix (1) to the CopyRefAssembly output, and get full path of the outputed file.

At this point the feature needs design before moving forward: To consider all possible outcomes and investigations on this topic.

f-alizada commented 7 months ago

Update based on the discussion and investigations: Currently the issue's priority is low since there is a workaround for the particular problem (for example rerun the build, or calling msbuild with the same path with or without symlink) . As an action that was done: the symlink label was created and assigned to the issues concerning symlink topic. The outcome of the addressing the issue by applying local fix for a specific problem brings risk introducing breaking changes. Unless there is a much simpler fix or the priority is higher , we'll need to prepare the design for the fix which will be taking into the account existing functionality.

dotnet / msbuild