Shadow copying for ASP.NET Core + IIS feedback

[Rick-Anderson]

Leave feedback for shadow copying for ASP.NET Core + IIS in this issue. @adityamandaleeka

What's new in 6.0 - Shadow copying in IIS

[ScottRFrost]

The sample web.config in the documentation at https://docs.microsoft.com/en-us/aspnet/core/release-notes/aspnetcore-6.0?view=aspnetcore-6.0#additional-resources is incorrect.

"AspNetCoreModulev2" must have a capital V in V2 or you'll get the error:

HTTP Error 500.21 - Internal Server Error
Handler "aspNetCore" has a bad module "AspNetCoreModulev2" in its module list

That said, even after I correct that error, I just get a 500 error with no text. Tested with .NET 6 hosting bundle installed on Windows Server 2016 Datacenter. There is also no log file generated when uncommenting the debugFile and debugLevel settings.

UPDATE: I can also reproduce the 500 w/o an error message or logs being created on my locally installed IIS on Windows 11 with VS2022 installed. I installed .NET 6 hosting bundle after I reproduced it, just in case, still getting 500.

My entire web.config:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <location path="." inheritInChildApplications="false">
    <system.webServer>
      <handlers>
        <add name="aspNetCore" path="*" verb="*" modules="AspNetCoreModuleV2" resourceType="Unspecified" />
      </handlers>
      <aspNetCore processPath="dotnet" arguments=".\MyAppNameHere.dll" stdoutLogEnabled="false" stdoutLogFile=".\logs\stdout">
        <handlerSettings>
          <handlerSetting name="experimentalEnableShadowCopy" value="true" />
          <handlerSetting name="shadowCopyDirectory" value="../ShadowCopy/" />
          <handlerSetting name="debugFile" value=".\logs\aspnetcore-debug.log" />
          <handlerSetting name="debugLevel" value="FILE,TRACE" />
        </handlerSettings>
      </aspNetCore>
    </system.webServer>
  </location>
</configuration>

[Deeeej]

This feature has been a life-saver for us. Simply put, since moving our web application to windows server 2022 and IIS 10, the traditional <EnableMsDeployAppOffline>True</EnableMsDeployAppOffline> in our webdeploy publish profile simply doesnt work. I mean, I can see the app_offline.htm going into the application directory, but we still get the classic "cant deploy, file is locked".

We have not been able to deploy to production without creating significant downtime, i.e. turn off the app pool and website in IIS, wait about 2 minutes for the file locking to stop, then publish the app.

If this feature were no to make it into the full .net core it would be really disappointing. Put another way, this setting gives us developers another option for deployment, and its not like there is a huge array of choice, I can only speak from my personal perspective and that of our company, but this feature is the single most important feature of ,net core, as being able to deploy apps with zero downtime (or at least virtually zero) is more important than an other feaure.

It could be argued "well, you just need to fix the issue with EnableMsDeployAppOffline", but thats the point, this shadow copying option means we have another choice.

Please keep it in, from me, other devs that havent discovered the same issues as me and from all devs that enjoy a range of choices of deployment settings.

ps. thank you for this feature, it saved us.

[ScottRFrost]

@Deeeej how did you get it to work? I can't get it to work even in the most trivial scenario of publishing a blank site. Can you share your web.config file with us?

[Deeeej]

<handlerSettings>
    <handlerSetting name="experimentalEnableShadowCopy" value="true" />
    <handlerSetting name="shadowCopyDirectory" value="../ShadowCopyDirectory_site1/" />
</handlerSettings>

Are the required settings but you need to add the correct permissions to the 'shadowCopyDirectory', I gave it the same permissions as the wwwroot folder, but if you are having problems add "Everyone" to the security of the folder, with 'Full Control' temporarily to see if this helps. Remember this only works with .net 6.

[ScottRFrost]

That got me pointed in the right direction @Deeeej ! Thank you!

Based on your comment, I manually created C:\inetpub\Shadow and gave IIS_IUSRS user full control of that folder.

However, I also had ANOTHER problem. The aspNetCore attributes processPath="%LAUNCHER_PATH%" arguments="%LAUNCHER_ARGS%" were not working for me for some reason, and I had to explicity write them out.

My new WORKING web.config:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <location path="." inheritInChildApplications="false">
    <system.webServer>
      <handlers>
        <remove name="aspNetCore"/>
        <add name="aspNetCore" path="*" verb="*" modules="AspNetCoreModuleV2" resourceType="Unspecified" />
      </handlers>
      <aspNetCore processPath="C:\Program Files\dotnet\dotnet.exe" arguments=".\MyAppNameHere.dll" stdoutLogEnabled="false" stdoutLogFile=".\logs\stdout">
        <handlerSettings>
          <handlerSetting name="experimentalEnableShadowCopy" value="true" />
          <handlerSetting name="shadowCopyDirectory" value="../Shadow/"  />
        </handlerSettings>
      </aspNetCore>
    </system.webServer>
  </location>
</configuration>

A new folder named "0" was automatically created in the C:\inetpub\Shadow folder and it appears to be working as intended now. Presumably that folder is created so you an use name="shadowCopyDirectory" value="../Shadow/" on multiple sites, but I haven't tried it yet.

@Rick-Anderson - It may be worth adding something about manually creating the Shadow folder and assigning it proper permissions to the documentation, in case other users run into this same issue.

[alanhoman]

This is a mission critical feature for my company, and we are using it on all of our production servers. I certainly hope this becomes a permanent .NET 7 feature and not just experimental.

With this feature I can now release updates throughout the day knowing that it will not cause any failed client transactions, timeouts, long delays, or unexpected disconnect errors. We have a load balancer for our web servers, and even with it, there are drainstop issues that occasionally cause failed client transactions during a deployment. We have opted to use the experimentalEnableShadowCopy instead and this has proven to be the most reliable, consistent, seamless, and simplest way to deploy updates.

[alanhoman]

I also 100% agree with everything @Deeeej stated above.

[kendallb]

We are migrating to ASP.NET Core from MVC 5 and the stepping stone for us to get stuff working on ASP.NET Core 2.2 layered on top of .NET 4.8. Does anyone know if this ONLY works with .NET 6 application files, or only with the .NET 6 module support (which is what we installed our servers). We are in production with the first (tiny) web site using ASP.NET Core, and instantly ran into this problem deploying our code.

We have our own deployment pipeline, so we currently dump app_offline.htm into the root of the application folder and it seems to work, but I also discovered that also takes down regular .NET web sites (which do not need it as they are shadow copied already). I would much prefer to rely on shadow copies working, but if this is still experimental and may not become a functional feature, I might ignore it for now.

Long term, I would love to see this be supported and ideally the shadow copy directory would NOT need to be specified. It should just work out of the box like it does today with regular ASP.NET web sites. That feature has been extremely useful for us. All our web sites except our image server are load balanced, so we deploy them one at a time while they are out of rotation and warm them up, so having downtime is not a big issue (locked files however is). But we do have one site (still .NET 4.8 and soon to be .NET Core) for serving images that is not yet load balanced, and when it goes down for upgrades we do get production level errors. Shadow copies on that one help mitigate most of the errors we see in production.

I do plan to load balance that one shortly (need to move images to a shared NFS server) so it won't be a big issue in the long term, but even if we are able to take the sites offline, having the site files be updatable without needing to shut the site down is a huge win IMHO. So I hope this feature becomes a standard feature and not experimental.

[kbalint]

Currently rewriting and redeploying a 100+ domain project from .net 4.8 to .net 6. The shadow copy feature is a must! Without it this project would be unmanageable.

[kendallb]

100%. It needs to be a core part of .NET Core (pun intended :) ).

[sthkn]

This feature is such a relief to have back!

Few questions/observations with my testing:

1. Can the numbered folders be named their source folder's name + "_shadow" or something? We could set it up so that each application has it's own shadow copy directory, but that adds another layer unnecessary developer interaction of making sure files and folders are in their correct places with correct permissions. A problem does occur when two applications point to the same shadow copy directory, where a website may display a different site:

   1. Create an IIS Server with 2 websites: Site1 & Site2
   2. Publish a test application to Site1
   3. Modify the Home/Index view and publish to Site2
   4. Recycle Site1 app pool
   5. Refresh Site1 in the web browser. Site1 now displays Site2.

2. Can shadow copy be disabled when the environment is development?

   • I would wager most developers will elect to use runtime compilation for razor, so shadow copying would probably make things more confusing for devs migrating or getting started with .NET.
   • When enabled in development (using IIS pointing to the project's folder, not publish folder), folders are created for app pool start up and shutdown, but are never deleted. I tried giving "Everyone" and literally everyone Full Control of the ShadowCopyDirectory folder, but the folders are still never deleted on my local machine. In the log file (but nothing in Event Viewer), it produces the error:

     [aspnetcorev2_inprocess.dll] Exception 'remove_all: Access is denied.: "C:\inetpub\wwwroot\ShadowCopyDirectory\1"' caught at D:\a_work\1\s\src\Servers\IIS\AspNetCoreModuleV2\InProcessRequestHandler\inprocessapplication.cpp:195

[Levyrii]

Hi，I found this feature doesn't work. Here is my configuration in web.config

<system.webServer>
  <handlers>
    <add name="aspNetCore" path="*" verb="*" modules="AspNetCoreModuleV2" resourceType="Unspecified" />
  </handlers>
  <aspNetCore processPath="dotnet" arguments=".\WebApplication3.dll" stdoutLogEnabled="false" stdoutLogFile=".\logs\stdout" >
    <handlerSettings>
      <handlerSetting name="experimentalEnableShadowCopy" value="true" />
      <handlerSetting name="shadowCopyDirectory" value="../ShadowCopy/" />
    </handlerSettings>
  </aspNetCore>
</system.webServer>

After using the above configuration, A new folder named "0" was automatically created in the "ShadowCopy" folder. However,as soon as I try to overwrite the file “WebApplication3.dll” to the publish folder，the file is still locked

[HaoK]

@LevyTerritory what versions of asp.net core/IIS/Windows are you using?

[Levyrii]

@LevyTerritory what versions of asp.net core/IIS/Windows are you using? asp.net core : 6.0.3 IIS : 10.0.19041.1586 Windows : Win10 Professional 21H1(19043.1586)

[HaoK]

What version of ANCM do you have under C:\Program Files\IIS\Asp.Net Core Module\V2 should have something that looks like a version for example 17.0.22077

[Levyrii]

What version of ANCM do you have under C:\Program Files\IIS\Asp.Net Core Module\V2 should have something that looks like a version for example 17.0.22077 this version of ANCM is 16.0.22055

[HaoK]

@LevyTerritory can you outline exactly what your workflow is for when you are running into the dll being locked, i.e. VS f5 or what exactly are you doing in your dev scenario

[Levyrii]

@HaoK I have found the cause of the problem. It need to set hostingModel="InProcess" Thank you for your help

[SpeednetGroup]

I'm going to chime in with the others who have said this feature is indispensable in light of normal deployment with app_offline still failing with locked files. If you have no way of fixing the locked file error, then please make this feature permanent.

That said, it is still a little wonky, causing a brief period immediately after deployment when the server can return a 503 error. So the ultimate solution it would seem would be to combine the app_offline deployment with the shadow copy.

I tried doing that, enabling both shadow copy and the <EnableMsDeployAppOffline> option in Preview.pubxml. It doesn't really help.

The problem is that the app_offline file is removed from the server before whatever "cutover" occurs to the new shadow-copied files, so there is still that brief period of time when the server can return 503 errors. If whatever script you're running could delay removal of app_offline until after the cutover occurs, I think we can make this thing seamless.

Another overall solution that could fix all these deployment problems? Improve the VS Publish feature to allow us to script actions at different stages of deployment. I'm sure the community could build a ton of useful actions for all the different scenarios that are causing problems.

Finally, one more thing that drives me absolutely batty: I have no ability to modify the app_offline.htm file that gets deployed with the option. That one would be so simple for you all to add an option for in VS. Just let me set the file path to use for app_offline.htm during deployment -- maybe even let me vary the path by host environment name.

[kendallb]

We wrote our own deployment tool using SSH to copy just the changed files along with the app_offline.html file with a custom message. We only have one server not load balanced (image server) and yeah we get the 503 issues also during the transition. My plan is to load balance the image server also with a shared file sever for the images to avoid this issue :(

[petr-horny-bsshop]

If any file in the bin folder is changed, AspNetCoreModuleV2 will restart the site after a certain period of time. We would need the following:

1. To have an option to set after what time the restart occurs.
2. Have the option to turn off change tracking (and subsequent restarts) completely.

Point 2 is important to us for the following reason. We host sites from a network path repository. If there is a brief network outage, the module detects the change and restarts the sites. This behavior is inappropriate for us. We have IIS configured so that it does not monitor changes, i.e. ASP.NET Framework sites do not restart.

[ScottRFrost]

25499 mentioned by @Rick-Anderson indicated that this feature is considered production and no longer experimental in .NET 7! This is great news! Thank you @Rick-Anderson !

[Rick-Anderson]

Closing as Shadow copy is now supported.

@HaoK lots of great comments here to review.

[Nuklon]

Is there any way to exclude some paths from shadow copy?

[HaoK]

Is there any way to exclude some paths from shadow copy?

Not currently, what is the situation where you feel this is necessary?

[Nuklon]

Is there any way to exclude some paths from shadow copy?

Not currently, what is the situation where you feel this is necessary?

It's copying the entire folder here rather than just the assemblies (including wwwroot and others). When you run Umbraco CMS for example, it creates a lot of temp and cache files so there's a lot of files that get copied. Also any images/videos/etc you add to CMS are added to wwwroot. All in all, this is very slow as there are a lot of (small) files. My current Umbraco folder has more than 11k files and copying takes a few minutes, whereas if I could exclude the wwwroot, umbraco folder, and App_Plugins, it only takes 5 seconds or so. I don't really understand why some of these folders are copied as the current directory and ASPNETCORE_IIS_PHYSICAL_PATH seems to be set to the original folder.

[RataplanNL]

Another vote to allow certain paths to be excluded. A simple file uploader with ~5GB of files shouldn't be copied every time a binary changes. Or at least always exclude wwwroot. This folder is probably never used to store binaries and is pointless to copy.

[ctorx]

Part of our deploy process is to apply the app_offline.htm, wait 15 seconds, then start the copy of the new files. This worked perfectly from dotnet v2.2 through v5. With v6 we started seeing issue with file locking and have been struggling to find a work around. Today I found the shadowcopy solution and it seems to be working. Finally.

What's most important is that we have a reliable way to deploy new files and NEVER run into a file locking issue. I think most of us can settle for minimal downtime to spool down a process in IIS and release file locks but it needs to be reliable and predictable and work from here on out. We don't want to continue to have to change our deploy processes with newer versions of dotnet.

[RichardD2]

Not sure if this is the right place for this, but we seem to be running into an issue with shadow copying using .NET 6.x, and I'm at a loss to diagnose it.

We have a Windows Server 2019 VM hosting (currently) 15 instances of an API, all based on identical code. They're all configured with separate AppPools, all set to "no managed code", and all configured for shadow-copying.

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <location path="." inheritInChildApplications="false">
    <system.webServer>
      <handlers>
        <add name="aspNetCore" path="*" verb="*" modules="AspNetCoreModuleV2" resourceType="Unspecified" />
      </handlers>
      <aspNetCore processPath=".\Assembly.exe" stdoutLogEnabled="false" stdoutLogFile=".\logs\stdout" hostingModel="inprocess">
        <handlerSettings>
          <handlerSetting name="experimentalEnableShadowCopy" value="true" />
          <handlerSetting name="shadowCopyDirectory" value="../shadow/api/" />
          <!-- Only enable handler logging if you encounter issues-->
          <!--<handlerSetting name="debugFile" value=".\logs\aspnetcore-debug.log" />-->
          <!--<handlerSetting name="debugLevel" value="FILE,TRACE" />-->
        </handlerSettings>
      </aspNetCore>
    </system.webServer>
  </location>
</configuration>

ANCM version is 16.0.22232.9.

Mostly this works fine. But for the last two months, when the server reboots to install updates, one AppPool seemingly chosen at random ends up with its shadow copy being corrupted. Specifically, the .deps.json file is filled with nulls, and the main assembly .dll is 0kb in size. There are three errors logged in the application event log by the ANCM:

1027: Unable to locate application dependencies. Ensure that the versions of Microsoft.NetCore.App and Microsoft.AspNetCore.App targeted by the application are installed.

1027: Could not find 'aspnetcorev2_inprocess.dll'. Exception message: A JSON parsing exception occurred in C:\inetpub\Instance\shadow\api\99\Assembly.deps.json, offset 0 (line 1, column 1): The document is empty. Error initializing the dependency resolver: An error occurred while parsing: C:\inetpub\Instance\shadow\api\99\Assembly.deps.json

1010: Failed to start application '/LM/W3SVC/9/ROOT/api', ErrorCode '0x8000ffff'.

After stopping the affected AppPool, deleting the shadow folder, and starting the AppPool again, everything springs back into life.

There don't appear to be any other relevant events in the event log around the time of the restart.

I don't know whether this is a bug in the .NET 6 shadow copying option, or a problem with the VM, or whether something else is interfering when the server restarts.

Has anyone else seen this, or have any suggestions on how to diagnose and/or fix it? (The debug file won't help, since the issue occurs before the application has a chance to start.)

[ctorx]

We ran into a different issue. The shadow copy would often be missing DLLs that were in the original folder, causing runtime errors for missing assemblies. Not sure if it's related to what you're experiencing, but we had to disable the feature.

On Thu, Nov 10, 2022 at 1:22 AM Richard Deeming @.***> wrote:

Not sure if this is the right place for this, but we seem to be running into an issue with shadow copying using .NET 6.x, and I'm at a loss to diagnose it.

We have a Windows Server 2019 VM hosting (currently) 15 instances of an API, all based on identical code. They're all configured with separate AppPools, all set to "no managed code", and all configured for shadow-copying.

<?xml version="1.0" encoding="utf-8"?>

ANCM version is 16.0.22232.9.

Mostly this works fine. But for the last two months, when the server reboots to install updates, one AppPool seemingly chosen at random ends up with its shadow copy being corrupted. Specifically, the .deps.json file is filled with nulls, and the main assembly .dll is 0kb in size. There are three errors logged in the application event log by the ANCM:

1027: Unable to locate application dependencies. Ensure that the versions of Microsoft.NetCore.App and Microsoft.AspNetCore.App targeted by the application are installed.

1027: Could not find 'aspnetcorev2_inprocess.dll'. Exception message: A JSON parsing exception occurred in C:\inetpub\Instance\shadow\api\99\Assembly.deps.json, offset 0 (line 1, column 1): The document is empty. Error initializing the dependency resolver: An error occurred while parsing: C:\inetpub\Instance\shadow\api\99\Assembly.deps.json

1010: Failed to start application '/LM/W3SVC/9/ROOT/api', ErrorCode '0x8000ffff'.

After stopping the affected AppPool, deleting the shadow folder, and starting the AppPool again, everything springs back into life.

There don't appear to be any other relevant events in the event log around the time of the restart.

I don't know whether this is a bug in the .NET 6 shadow copying option, or a problem with the VM, or whether something else is interfering when the server restarts.

Has anyone else seen this, or have any suggestions on how to diagnose and/or fix it? (The debug file won't help, since the issue occurs before the application has a chance to start.)

— Reply to this email directly, view it on GitHub https://github.com/dotnet/AspNetCore.Docs/issues/23733#issuecomment-1309996000, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACT5Z4PNHSB36HGUKCXFEDWHS5GRANCNFSM5HKO4FUA . You are receiving this because you commented.Message ID: @.***>

[adityamandaleeka]

@RichardD2 Thanks for the issue report. I've filed a dedicated issue for that here: https://github.com/dotnet/aspnetcore/issues/45017

[MrZander]

Is there any way to exclude some paths from shadow copy?

Not currently, what is the situation where you feel this is necessary?

It's copying the entire folder here rather than just the assemblies (including wwwroot and others). When you run Umbraco CMS for example, it creates a lot of temp and cache files so there's a lot of files that get copied. Also any images/videos/etc you add to CMS are added to wwwroot. All in all, this is very slow as there are a lot of (small) files. My current Umbraco folder has more than 11k files and copying takes a few minutes, whereas if I could exclude the wwwroot, umbraco folder, and App_Plugins, it only takes 5 seconds or so. I don't really understand why some of these folders are copied as the current directory and ASPNETCORE_IIS_PHYSICAL_PATH seems to be set to the original folder.

Are there any plans for this option? We have some large dependencies that are downloaded the first time the application runs that I would like to exclude from being shadow copied.

[andy1547]

Feedback:

First thanks for this feature, without this we would have abandoned migrating from ASP.NET WebAPI v2 to ASP.NET Core MVC. We deploy regularly with the requirement of minimal downtime defined by SLAs. We found that in alot of cases the app_offline.htm approach didn't work (for example doing a Thread.Sleep(TimeSpan.FromHours(1)) in a controller method, with the only reliable fix being to stop/start the IIS server, incurring a significant amount of downtime.

We had to disable the DeleteExistingFiles option when enabling this setting which isn't ideal, not sure why it tries to delete the folder rather than just doing a diff and deleting files missing in src: C:\Program Files\dotnet\sdk\7.0.304\Sdks\Microsoft.NET.Sdk.Publish\targets\PublishTargets\Microsoft.NET.Sdk.Publish.FileSystem.targets(67,5): Error MSB3231: Unable to remove directory "../../VirtualDirectoryRoot/path1/api/". The process cannot access the file 'C:\VirtualDirectoryRoot\path1\api' because it is being used by another process.

From a developer perspective, having to create and set permissions on a shadow folder increases the barrier of entry, so ideally this would be built-in (like ASP.NET).

Are there any differences between using this feature with .NET 6 and .NET 7? We currently using .NET 6 due to it being a LTS release, however if there's bug fixes that have only been applied to .NET 7 we'd migrate over.

We have different patch versions of both the .NET 6 and 7 SDK/runtime installed on each developer machine, when targeting .NET 6 we've have found on some machines we need to enable the experimental setting, and other times the .NET 7+ setting, so we've restored to enabling both just in case.

[tkf144]

Unfortunately without the exclude option mentioned above, +7GB of .angular/cache, node_modules etc get copied on every build in the codebases I'm working in, making this unusable. A shame.

[Gruski]

I made all the changes to web.config outlined in the documentation for v7.0 .net Core

<handlerSettings>
    <handlerSetting name="enableShadowCopy" value="true" />
    <handlerSetting name="shadowCopyDirectory" value="../ShadowCopy/" />

    <handlerSetting name="debugFile" value="aspnetcore-debug.log" />
    <handlerSetting name="debugLevel" value="FILE,TRACE" /> 
</handlerSettings> 

Pointed my IIS app root to bin/net.7.0 of project on dev box Added bin/ShodowCopy folder with full permissions given to to the IIS ApplicationPool identity (IIS AppPool\MyAppName). It's not putting any files in there and build complains that flies are locked in bin/net.7.0. So basically no difference.

Also logging doesn't work. No aspnetcore-debug.log file created anywhere.

[tbayart]

Hi, i'm unable to make shadow copy working on .Net 6 web API I enabled debugFile but nothing in its content is related to shadow copy or the shadow copy folder Here is my web.config

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <location path="." inheritInChildApplications="false">
    <system.webServer>
      <handlers>
        <remove name="aspNetCore"/>
        <add name="aspNetCore" path="*" verb="*" modules="AspNetCoreModuleV2" resourceType="Unspecified" />
      </handlers>
      <aspNetCore processPath="dotnet" arguments="2024-02-09\Gateway.dll" stdoutLogEnabled="false" stdoutLogFile=".\logs\stdout" hostingModel="inprocess">
        <handlerSettings>
          <handlerSetting name="experimentalEnableShadowCopy" value="true" />
          <handlerSetting name="shadowCopyDirectory" value="../ShadowCopy/" />
          <handlerSetting name="debugFile" value="e:\inetpub\aspnetcore-debug.log" />
          <handlerSetting name="debugLevel" value="FILE,TRACE" />
        </handlerSettings>
      </aspNetCore>
    </system.webServer>
  </location>
</configuration>

I have no access to the IIS server itself, only access to inetpub throught a SMB share

[MrZander]

@tbayart Try <handlerSetting name="enableShadowCopy" value="true" />

[tbayart]

@tbayart Try <handlerSetting name="enableShadowCopy" value="true" />

I tried this at first but with no result. Unfortunately i lack of time to investigate yet