MSBuild deletes portable debugging symbols after build

[Nihlus]

When using MSBuild under Mono 5.0.1.1, it deletes generated debugging symbols after copying them to the output directory. This prevents reliable debugging and can cause symbol-reliant postprocessing steps to fail

As a workaround, a postbuild copy target can be added which copies the debugging symbols again.

xbuild does not have this problem, and leaves the .pdb file in place.

An example of output by MSBuild where the symbols are deleted is available here: https://gist.github.com/Nihlus/193d91878530338d3a9e7238e594976c

The full diagnostic log for a failing project is available here: https://gist.github.com/Nihlus/278185a4ca1fb9cc43fe2ba73706f956

The source code of the failing project is available here: https://github.com/opentk/opentk

The culprit appears to be the imported target "IncrementalClean" which has a "Delete" task associated with it - see line 5428 in the diagnostic log.

-- System Info

Linux Mint 18.1 Cinnamon Edition Mono 5.0.1.1 (Xamarin release via repository) MSBuild 15.2.0.0 xbuild 14.0

[rainersigwald]

Looks like something's not correctly appending the output pdb path to @(FileWrites), but the target looks right. Unfortunately, that log doesn't include item manipulations that would help figure out why-- @Nihlus did is that really with /v:diag? If so, maybe Mono MSBuild isn't logging as much as it should.

@radical seen this before? I can't think of a reason why it would be different, but I know the .mdb/.pdb stuff has made symbols for Mono a bit more confusing.

[Nihlus]

@rainersigwald I reran the build and got a larger diagnostic log - I must've missed something the last time around. Here's the updated log: https://gist.github.com/Nihlus/773d9d7f0fb44048adcf0b08a5d2590a

An important thing to note is that I've done some more testing with another project which I've ported to the new csproj format (this project still uses the old one), and it does not appear to be an issue there.

Unfortunately, the projects which are having these problems cannot be ported yet for various reasons.

[rainersigwald]

Ok, so the way IncrementalClean works is:

1. As the build goes along, collect a list of written files
2. Read the list of files written in the last build from FileListAbsolute.txt
3. Remove the current files from the written-last-time files
4. Delete the files that were written last time but not this time
5. Write the new list of files to FileListAbsolute.txt for use in the next build

Your new log shows where this is going wrong:

Task "WriteLinesToFile" (TaskId:37)
  Task Parameter:File=obj/Debug/Generator.Bind.csproj.FileListAbsolute.txt (TaskId:37)
  Task Parameter:
      Lines=
          /home/jarl/Programming/opentk/src/Generator.Bind/bin/Debug//Bind.pdb
          /home/jarl/Programming/opentk/src/Generator.Bind/bin/Debug/Bind.exe
                  OriginalItemSpec=obj/Debug/Bind.exe
          /home/jarl/Programming/opentk/src/Generator.Bind/obj/Debug/Generator.Bind.csprojResolveAssemblyReference.cache
          /home/jarl/Programming/opentk/src/Generator.Bind/obj/Debug/Generator.Bind.csproj.CoreCompileInputs.cache
          /home/jarl/Programming/opentk/src/Generator.Bind/obj/Debug/Bind.exe
          /home/jarl/Programming/opentk/src/Generator.Bind/obj/Debug/Bind.pdb (TaskId:37)
  Task Parameter:Overwrite=True (TaskId:37)
Done executing task "WriteLinesToFile". (TaskId:37)

Note that Bind.pdb is listed twice--once with a double slash. The copy with a single slash is removed as a current write, but the double-slash one isn't (it's not an identical string).

Unfortunately, I can't see why the double-slash version is getting into the file in the first place--after the first time, it's there because it was in the file last time.

@Nihlus can you share a diag log after cleaning all build outputs (including and especially **/*.FileListAbsolute.txt)?

[Nihlus]

@rainersigwald Here's the new log: https://gist.github.com/Nihlus/ee0dc9d11cd55902729d92cf3184eccf Suddenly, it's decided to start working again. I've no idea why.

[rainersigwald]

@Nihlus My guess is that you did {something} that caused the bad entry to be written to the state file, and then it persisted across builds. After cleaning it out once, things returned to normal.

That's a frustrating answer, of course. I'm going to close this bug, but if you see this again and especially if you figure out what {something} is that's causing the bad entry to get into FileListAbsolute in the first place, please let me know and we can hopefully get a more robust fix in.

[Nihlus]

@rainersigwald I've had it happen across multiple machines and multiple branches and checkouts of different repositories, so I don't think it's a fluke. More than anything, it's a fluke that it's working now.

[Nihlus]

@rainersigwald One of my users is experiencing the same issue on Ubuntu 16.04 (ARM) and Mono 5.0.1. I had him clean the solution completely and run a diagnostic build. Here's his log. msbuild.txt

Note that he had to fall back to the mcs compiler due to csc crashing on his machine.

[radical]

@Nihlus which file got unexpectedly deleted here? The error seems to be because of a missing bin/Debug/../../../Generator.Rewrite/bin/Debug/Rewrite.exe but I don't see any other mention of this file in the log. Did you clean the workspace after that csc crash and before switching to mcs? Did you use xbuild on this too? And how did you fallback to mcs? If the issue shows up only on a second build, then could you try to capture both the logs?

[Nihlus]

The workspace was completely cleaned before building with mcs. The mcs fallback was done through adding

<PropertyGroup>
    <CscToolExe>mcs.exe</CscToolExe>
</PropertyGroup>

to the project file. Xamarin had some suggestions about this (https://releases.xamarin.com/common-issues-from-the-xamarin-15-2-release-being-tracked-by-the-xamarin-team/) related to csc crashes. The Rewrite.exe file is not missing, but crashing due to the missing debugging symbol file. It's an external tool which uses the debugging symbols to rewrite a number of methods with optimized IL instructions, but that's not super relevant.

This issue shows up consistently on all builds for this user (both initial and subsequent). In this case, it is the mdb debugging symbols generated by mcs which are not present. We've worked around the issue by adding

<Copy SourceFiles="obj\$(Configuration)\$(AssemblyName).dll.mdb" DestinationFolder="$(OutputPath)" />

to the AfterBuild target.

We have not tried using xbuild - this is all through msbuild.

[dasMulli]

(since it is Sunday and I've got nothing better to do..) In the build log a .pdb file is expected:

Initial Items:
[…] 
_DebugSymbolsIntermediatePath
    obj/Debug/OpenTK.pdb
_DebugSymbolsOutputPath
    bin/Debug/OpenTK.pdb

but the compiler doesn't seem to generate one so _CheckForCompileOutputs sets _DebugSymbolsProduced to false:

Target "_CheckForCompileOutputs: (TargetId:50)" in file "/usr/lib/mono/msbuild/15.0/bin/Microsoft.Common.CurrentVersion.targets" from project "/home/odroid/opentk/src/OpenTK/OpenTK.csproj" (target "CopyFilesToOutputDirectory" depends on it):
Added Item(s): FileWrites=obj/Debug/OpenTK.dll
Added Item(s): FileWrites=bin/Debug/OpenTK.xml
Set Property: _DebugSymbolsProduced=false

so no debug symbols are copied:

Task "Copy" skipped, due to false condition; ('$(_DebugSymbolsProduced)'=='true' and '$(SkipCopyingSymbolsToOutputDirectory)' != 'true' and '$(CopyOutputSymbolsToOutputDirectory)'=='true') was evaluated as ('false'=='true' and '' != 'true' and 'true'=='true').

[Nihlus]

@dasMulli Well, that explains that. I suppose we'll have to try and track down the csc crash and open a separate report for that instead. However, I would argue that mdb symbols should be added as possible output in order to maintain backwards compat in cases such as this. Both csc and msbuild have been an utter pain on Linux since day one where mcs and xbuild were solid as rock :(

[radical]

@Nihlus set $(CscDebugFileExt)=.mdb also. This is a msbuild/mono only feature right now, fyi.