WPF: CleanupTemporaryTargetAssembly step after MarkupCompilePass2 intermittently fails to delete target assembly

[vsfeedback]

This issue has been moved from a ticket on Developer Community.

---

[severity:Multiple people on my team are impacted (internal MSFT only)] Some WPF projects require a second pass compile, in which a temporary project is generated and then cleaned up. Sometimes, during this compilation, the "CleanupTemporaryTargetAssembly" step fails to perform cleanup successfully due to the target assembly being locked.

Sample error:

MarkupCompilePass2:
  MarkupCompilePass2 successfully generated BAML or source code files.
CleanupTemporaryTargetAssembly:
  Deleting file "Debug\AnyCPU\MyWpfAssembly.dll".
##[error]C:\Windows\Microsoft.NET\Framework\v4.0.30319\Microsoft.WinFx.targets(480,10): Error MSB3061: Unable to delete file "Debug\AnyCPU\MyWpfAssembly.dll". The process cannot access the file 'C:\w\4\s\src\MyWpfAssembly\Debug\AnyCPU\MyWpfAssembly.dll' because it is being used by another process.
C:\Windows\Microsoft.NET\Framework\v4.0.30319\Microsoft.WinFx.targets(480,10): error MSB3061: Unable to delete file "Debug\AnyCPU\MyWpfAssembly.dll". The process cannot access the file 'C:\w\4\s\src\MyWpfAssembly\Debug\AnyCPU\MyWpfAssembly.dll' because it is being used by another process. [C:\w\4\s\src\MyWpfAssembly\MyWpfAssembly.csproj]
Done Building Project "C:\w\4\s\src\MyWpfAssembly\MyWpfAssembly.csproj" (default targets) -- FAILED.

Other customers have reported the same issue, ex. https://social.msdn.microsoft.com/Forums/en-US/6360a1e3-a269-457c-a763-fb415b8bbf14/quotbecause-it-is-being-used-by-another-processquot-error-has-me-dead-in-the-water?forum=msbuild

---

Original Comments

Feedback Bot on 6/11/2021, 08:22 AM:

We have directed your feedback to the appropriate engineering team for further evaluation. The team will review the feedback and notify you about the next steps.

Chaitanya Mangalagiri [MSFT] on 8/12/2021, 00:50 AM:

(private comment, text removed)

---

Original Solutions

(no solutions)

[timmi-on-rails]

Can you rule out, that a virus scanner or windows defender might be locking the file? In my case it is working relyably on one machine, but not on the other.

[luy-ce]

This failure occurs on our build agents, which as you can imagine, are set up to not have any external processes that may lock the file.

[Belpaire]

It also happens on our build servers, I tried looking into it myself in the winfx targets file, as well as into our msbuild binlogs. But I was not able to pinpoint a culprit as of yet unfortunately. Edit: Is it possible that the MarkupCompilePass2 task sometimes keeps the file locked? I don't immediately see what else could be causing a lock to this file.

[timmi-on-rails]

Here comes a workaround that solves my (very similar) problem.

After the MarkupCompilePass2 I am even unable to manually delete the DLL that the CleanupTemporaryTargetAssembly also fails to delete with a Access to the path ... is denied. error. I was unable to determine the reason for the access denied error. The command line tool https://docs.microsoft.com/en-us/sysinternals/downloads/handle showed no open file handles. However I am running the build process inside a windows docker container, so maybe the host is locking the file.

The good thing is, I am still able to manually move the problematic DLL file. So I added the following target to the project file. Make sure to replace the file name.

<Target Name="Hack" BeforeTargets="CleanupTemporaryTargetAssembly">
  <Exec Command="move Debug\AnyCPU\MyWpfAssembly.dll Debug\AnyCPU\MyWpfAssembly.txt" />
</Target>

Tell me, if this works for you as well? There is still a chance that my problem is not connected to yours.

[Belpaire]

@timmi-on-rails I'd rather not implement something that is this implementation dependent, as underlying changes in wpf could easily break this solution if it even works in the first place. I did some digging in the source code and only thing I saw that could perhaps be related to what we see is https://github.com/dotnet/wpf/blob/e4ee239b63e7e0fe64b3bada60234c7595158a0d/src/Microsoft.DotNet.Wpf/src/PresentationBuildTasks/MS/Internal/MarkupCompiler/MarkupCompiler.cs#L205-L230, could be that somehow the compiler is locking this assembly with this "ReflectionHelper", but honestly someone on the wpf team should really look into this issue because we're not really providing any structural solutions ourselves I'm afraid.

[adamlett]

I am also suffering from this issue, we have a large solution that contains many projects (C#, managed C++ and native C++ projects) and yet the only projects that suffer from this 'Unable to delete file' error are those that have just completed the 'MarkupCompilePass2' build task. It only happens intermittently however and sometimes more often than others

Initially we thought it could be an antivirus or similar locking the files, but this does not seem to be the case as I'd expect that to affect other projects too and not just the ones that undergo a second pass compile

Thanks for the workaround @timmi-on-rails, I'll see if it works for us too

[adamlett]

Unfortunately the workaround @timmi-on-rails provided didn't work in my case (it was unable to move the files as well as being unable to delete them), however it put mt on the right track and I have a workaround that seems to be working for us

I created a 'Directory.Build.props' at the root of our solution, so that it will be applied to all projects. The content of it is below:

<Project>
  <!-- To avoid the intermittent error MSB3061: Unable to delete file "obj\Release\<Assembly>.dll". The process cannot access the file.
       The failure occurs in the 'CleanupTemporaryTargetAssembly' target, so we launch a task before that which checks if the file is locked,
       and attempts to wait for it to unlock (up to a maximum time).
       The issue is described here: https://github.com/dotnet/wpf/issues/5255 -->

       <!-- Note: This 'Directory.Build.props' file is automatically used by all projects. https://docs.microsoft.com/en-us/visualstudio/msbuild/customize-your-build?view=vs-2022 -->

  <Target Name="MSB3061_Workaround" BeforeTargets="CleanupTemporaryTargetAssembly">
    <PropertyGroup>
      <TemporaryAssemblyPath Condition="Exists('obj\$(Configuration)\$(AssemblyName).dll')">obj\$(Configuration)\$(AssemblyName).dll</TemporaryAssemblyPath>
      <TemporaryAssemblyPath Condition="Exists('obj\$(Platform)\$(Configuration)\$(AssemblyName).dll')">obj\$(Platform)\$(Configuration)\$(AssemblyName).dll</TemporaryAssemblyPath>
    </PropertyGroup>

    <WaitForLockedFile FullPath="$(TemporaryAssemblyPath)" MaxWaitTimeMillis="60000"/>
  </Target> 

  <UsingTask TaskName="WaitForLockedFile" TaskFactory="CodeTaskFactory" AssemblyFile="$(MSBuildToolsPath)\Microsoft.Build.Tasks.v4.0.dll" >
    <ParameterGroup>
      <FullPath Required="true" />
      <MaxWaitTimeMillis  ParameterType="System.Int32" Required="true" />
    </ParameterGroup>
    <Task>
        <Using Namespace="System.IO"/>
        <Using Namespace="System.Threading"/>
        <Code Type="Fragment" Language="cs">
        <![CDATA[
            int timeBetweenTries = 100;
            int maxTries = MaxWaitTimeMillis / timeBetweenTries;

            for (int numTries = 0; numTries < maxTries; numTries++) {
                FileStream fs = null;
                try {
                    fs = new FileStream (FullPath, FileMode.Open, FileAccess.ReadWrite);
                    fs.Dispose();
                    Log.LogMessage(MessageImportance.High, "Success! File " + FullPath + " is accessible and not locked. (" + numTries + " retries were required.)");
                    return true;
                }
                catch (IOException) {
                    if (fs != null) {
                        fs.Dispose ();
                    }
                    Thread.Sleep (timeBetweenTries);
                }
            }
            Log.LogError("Failed to access file '"  + FullPath + "'. It remains locked by another process despite waiting " + MaxWaitTimeMillis + "ms. (WaitForLockedFile)");
            return false;
        ]]>
        </Code>
    </Task>
  </UsingTask>

</Project>

It essentially just keeps retrying until the file is no longer locked, or until the max wait timeout is reached. In our case our build only seems to keep it locked for a few hundred milliseconds at maximum and so it only needs to retry 0 to 5 times

[t-johnson]

@adamlett I've applied your workaround to avoid random build failures on our build host. Seems to be good for me! I made a small change to support EXE and DLL outputs, as well as using $(IntermediateOutputPath) to simplify:

      <TemporaryAssemblyPath Condition="Exists('$(IntermediateOutputPath)$(AssemblyName).dll')">$(IntermediateOutputPath)$(AssemblyName).dll</TemporaryAssemblyPath>
      <TemporaryAssemblyPath Condition="Exists('$(IntermediateOutputPath)$(AssemblyName).exe')">$(IntermediateOutputPath)$(AssemblyName).exe</TemporaryAssemblyPath>

[MVAnand]

Hi, I have also across the same issue inside the Windows Docker Container. Getting the same error saying access is denied to the path. I tried solution suggested by @adamlett and getting the error : Failed to access file 'obj\x86\Release\XXXX.dll'. It remains locked by another process despite waiting 60000ms. (WaitForLockedFile).

I can successfully project outside the container, but this issue is coming with Docker container only. I have reported an issue https://github.com/microsoft/dotnet/issues/1381

https://learn.microsoft.com/en-us/answers/questions/1283264/docker-container-build-(cleanup-temporary-target-a

Any input would be a great help.

[Foitn]

Anyone found a solution?

[mattnorflus]

We (VSEng) are seeing this now in our build agents. We are setting up a new build environment to validate a VS build. This environment installs a test build of VS to an agent and tries to build the VS repo using the build tools of the test build. The intention is to verify that the VS product can build the VS repo. It is possible a change in the VS product may be surfacing this error but we have not been able to find a root cause. We believe this issue is related to our error at this time due to the similarities with what others have shared.

As other have mentioned above, we do not experience this problem on most of our agents, only this new build environment. The error occurs intermittently in different projects from build to build. The only common element of the failures is that CleanupTemporaryTargetAssembly fails to delete the binary after running MarkupCompilePass2. The above workarounds may not be simple for us to implement due to the size of the VS repo. It would be great to get more insight into the problem and maybe a fix? Our binlogs are large but can be found as artifacts for these builds:

https://dev.azure.com/devdiv/DevDiv/_build/results?buildId=8792392&view=logs&j=018464ba-1ca1-5561-2761-cd93078ac78a&t=3574efd9-071a-5092-469d-6ae99dc3b0cf https://dev.azure.com/devdiv/DevDiv/_build/results?buildId=8790328&view=logs&j=018464ba-1ca1-5561-2761-cd93078ac78a&t=3574efd9-071a-5092-469d-6ae99dc3b0cf https://dev.azure.com/devdiv/DevDiv/_build/results?buildId=8790291&view=logs&j=018464ba-1ca1-5561-2761-cd93078ac78a&t=3574efd9-071a-5092-469d-6ae99dc3b0cf

Feel free to ping me on teams if you are unable to view the builds and need to see the log for debugging.

[kyle-wilt-waters]

Could it be due to the implementation of MarkupCompilePass1 in PresentationBuildTasks? There is code here that creates an AppDomain, runs a compilation step, and then later unloads that AppDomain in the thread pool which effectively swallows any exceptions by default. So if the unload fails or takes a while and that AppDomain happens to include the dll of interest you'll hit this problem. This assumes AppDomain.Unload actually blocks until the domain is unloaded in the first place which isn't 100% clear to me based on the documentation and implementation I can see in 4.8.

There's another task named MarkupCompilePass2 which also creates an AppDomain for additional operations but it doesn't run the AppDomain.Unload asynchronously.