Hosting multiple instances of an app published with PublishSingleFile and IncludeNativeLibrariesForSelfExtract does not work as expected.

[qcc-na]

Description

When compiling a web project to a single file application with IncludeNativeLibrariesForSelfExtract you can only run one instance of the application in IIS. If you try to run a second instance, it will try to extract from the same directory and fail to do so. This is contrary to the documented behavior I've read in one of dotnet's design documents. Specifically the linked document says,

Check if the [extraction directory](Extraction Location) exists for this app: If not, perform a new extraction: Extract appropriate files to a temporary location // . This ensures that concurrent launches of the app do not interfere with each other.

This isn't the case in my experience.

If you remove the IncludeNativeLibrariesForSelfExtract option this problem goes away.

  <PropertyGroup>
    <TargetFramework>net5.0</TargetFramework>
    <GenerateAssemblyInfo>false</GenerateAssemblyInfo>
    <AspNetCoreHostingModel>OutOfProcess</AspNetCoreHostingModel>
    <PublishSingleFile>true</PublishSingleFile>
    <SelfContained>true</SelfContained>
    <RuntimeIdentifier>win-x64</RuntimeIdentifier>
    <PublishReadyToRun>true</PublishReadyToRun>
    <IncludeNativeLibrariesForSelfExtract>true</IncludeNativeLibrariesForSelfExtract>
  </PropertyGroup>

Reproduction Steps

0. Open Visual Studio Version 16.10.3

1. File -> New -> Project -> ASP NET Core Web App -> Next -> .Net 5.0 -> Create

2. Paste this into the project file.

  <PropertyGroup>
    <TargetFramework>net5.0</TargetFramework>
    <GenerateAssemblyInfo>false</GenerateAssemblyInfo>
    <AspNetCoreHostingModel>OutOfProcess</AspNetCoreHostingModel>
    <PublishSingleFile>true</PublishSingleFile>
    <SelfContained>true</SelfContained>
    <RuntimeIdentifier>win-x64</RuntimeIdentifier>
    <PublishReadyToRun>true</PublishReadyToRun>
    <IncludeNativeLibrariesForSelfExtract>true</IncludeNativeLibrariesForSelfExtract>
  </PropertyGroup>

3. Folder publish
4. Copy to c:\inetpub\wwwroot\testproject1
5. Copy to c:\inetpub\wwwroot\testproject2
6. Convert each of these to web application in their own application pool
7. Attempt to view each of these applications
8. One will succeed and one will show a hosting error.
9. Remove <IncludeNativeLibrariesForSelfExtract>true</IncludeNativeLibrariesForSelfExtract> from the project file.
10. Republish to respective directories
11. Both applications work

Expected behavior

The expected behavior is for both instances of the application to work properly when <IncludeNativeLibrariesForSelfExtract>true</IncludeNativeLibrariesForSelfExtract> is in the project file prior to publishing.

Actual behavior

The actual behavior is only a single instance working properly when <IncludeNativeLibrariesForSelfExtract>true</IncludeNativeLibrariesForSelfExtract> is included in the project file prior to publishing.

Regression?

I do not know if this worked in prior releases as this was my first attempt.

Known Workarounds

Removing <IncludeNativeLibrariesForSelfExtract>true</IncludeNativeLibrariesForSelfExtract> from the project file prior to publishing enables both applications to work but does NOT produce a single file build output.

Renaming one of the .exe files and it's key in the associated web.config file allows this to work.

Configuration

Visual Studio: Microsoft Visual Studio Professional 2019 Version 16.10.3

OS: Edition Windows 10 Pro Version 21H1 Installed on ‎5/‎20/‎2021 OS build 19043.1237 Experience Windows Feature Experience Pack 120.2212.3530.0

Other information

No response

[ghost]

Tagging subscribers to this area: @agocke, @vitek-karas, @vsadov See info in area-owners.md if you want to be subscribed.

Issue Details

---

### Description When compiling a web project to a single file application with `IncludeNativeLibrariesForSelfExtract` you can only run one instance of the application in IIS. If you try to run a second instance, it will try to extract from the same directory and fail to do so. This is contrary to the documented behavior I've read in one of [dotnet's design documents](https://github.com/dotnet/designs/blob/main/accepted/2020/single-file/extract.md). Specifically the linked document says, > Check if the [extraction directory](Extraction Location) exists for this app: If not, perform a new extraction: Extract appropriate files to a temporary location // . This ensures that concurrent launches of the app do not interfere with each other. This isn't the case in my experience. If you remove the IncludeNativeLibrariesForSelfExtract option this problem goes away. ``` net5.0 false OutOfProcess true true win-x64 true true ``` ### Reproduction Steps 0. Open Visual Studio Version 16.10.3 1. File -> New -> Project -> ASP NET Core Web App -> Next -> .Net 5.0 -> Create 2. Paste this into the project file. ``` net5.0 false OutOfProcess true true win-x64 true true ``` 3. Folder publish 4. Copy to c:\inetpub\wwwroot\testproject1 5. Copy to c:\inetpub\wwwroot\testproject2 6. Convert each of these to web application in their own application pool 7. Attempt to view each of these applications 8. One will succeed and one will show a hosting error. 9. Remove `true` from the project file. 10. Republish to respective directories 11. Both applications work ### Expected behavior The expected behavior is for both instances of the application to work properly when ` true` is in the project file prior to publishing. ### Actual behavior The actual behavior is only a single instance working properly when ` true` is included in the project file prior to publishing. ### Regression? I do not know if this worked in prior releases as this was my first attempt. ### Known Workarounds Removing `true` from the project file prior to publishing enables both applications to work but does NOT produce a single file build output. Renaming one of the .exe files and it's key in the associated web.config file allows this to work. ### Configuration Visual Studio: Microsoft Visual Studio Professional 2019 Version 16.10.3 OS: Edition Windows 10 Pro Version 21H1 Installed on ‎5/‎20/‎2021 OS build 19043.1237 Experience Windows Feature Experience Pack 120.2212.3530.0 ### Other information _No response_

Author:	qcc-na
Assignees:	-
Labels:	`area-Single-File`, `untriaged`
Milestone:	-

[agocke]

@VSadov could you take a look?

[VSadov]

I will take a look.

[VSadov]

The extraction mechanism did not change much, so it may not be a new issue, just a new scenario.

@qcc-na - Is this with 5.0? What kind of error is produced?

[qcc-na]

HTTP Error 502.5 - ANCM Out-Of-Process Startup Failure
Common solutions to this issue:
The application process failed to start
The application process started but then stopped
The application process started but failed to listen on the configured port
Troubleshooting steps:
Check the system event log for error messages
Enable logging the application process' stdout messages
Attach a debugger to the application process and inspect
For more information visit: https://go.microsoft.com/fwlink/?LinkID=808681

Attached Event Logs (cleared before accessing error page): eventlog.zip

If this file isn't sufficient let me know and I can start pasting the logs in here.

Clarification: Yes .net 5.0

[qcc-na]

@VSadov Any info?

[revuniversal]

Any word on this?

[VSadov]

I will try to take a look today. It is a very strange error. You can certainly run multiple copies of a singlefile app. Something must be different for how web apps are being run.

[VSadov]

If I run two copies of the same webapp as-is (without IIS, which I do not have on my machine), I do not see any issues.

Both apps run and the native files are extracted into a temp folder that corresponds to the app %TEMP%\.net\WebApplication1\nreshoqb.fjm in my case

There is nothing special to web apps when running in this mode. I will try reproducing with IIS, but it may take time since I am not very familiar with IIS.

I wonder - can it be some permissions conflict? How does IIS runs application pools? Do they run with limited permissions or under some special user?

[VSadov]

Another thing to try is to set environment variable COREHOST_TRACE to 1 and capture the output.

If you can do this faster than me, it could be helpful with figuring what is going on.

I suspect it is not a problem with the singlefile model itself, but with the app model in IIS - like the first app extracts files, but the next one does not have permissions to use them.

[qcc-na]

I changed the account in both AppPools to "LocalSystem" and both instances were able to load. I think you are correct about this being a permissions issue. Do you think it is an issue that 2 separate instances are extracting to the same folder? The document seems to indicate that process-id is involved but it doesn't seem like it is. Is it the process id of IIS?

[VSadov]

Do you think it is an issue that 2 separate instances are extracting to the same folder?

That is how it should work. Once the files are extracted, they will be reused if the app runs again or if another copy of the app is launched.

The extraction location is specific to the application and is under %TEMP%, which is different for different users, so there are no problems with permissions. Different users see different extraction directories. Under IIS it seems permissions may be configured in various ways including to not match the user account that starts the process and that could cause issues with extraction.

I do not think this is actionable on the singlefile app model side, but may be worth mentioning in documentation as what can happen under IIS and what can be done about that.

[vitek-karas]

@jkotalik for the ASP.NET IIS hosting side of things.

Maybe this could be worked around by setting the DOTNET_BUNDLE_EXTRACT_BASE_DIR to point to a directory with the right permissions (the default value for this is effectively %TEMP%\.net).

Maybe this should be done in the ASP.NET module to make it work for all apps... but I don't know the details around the permissions and so on.

It would still be interesting to capture the host trace for the failure. Set COREHOST_TRACE=1 and COREHOST_TRACE=host.txt and it will trace into the specified file (no need to setup IIS to capture stderr if that's too problematic).

[BrennanConroy]

@HaoK instead of Justin.

[qcc-na]

I think maybe there is a gap somewhere here. I'm not an IIS expert myself, but my understanding is that application pools are a security measure that enforce isolation between applications on a server. Are both of these processes running from the same folder given 2 application pools with 2 different application pool identities?

By default an app pool runs under the "ApplicationPoolIdentity" user, which according to this link about application pool identities:

An application pool identity allows you to run an application pool under a unique account without having to create and manage domain or local accounts. The name of the application pool account corresponds to the name of the application pool.

@VSadov seems correct in that the user doesn't seem to have any impact on the extraction directory, but I'd argue that it should given the context of IIS and an application pool and application pool identity.

@vitek-karas I will set these environment variables and try to get you a trace today if possible.

[qcc-na]

@vitek-karas Where does this "host.txt" file end up being created?

[vitek-karas]

The COREHOST_TRACEFILE=path let's you set an absolute path in which case it goes there. If it's a relative path, it is relative to the current directory (no idea what that is for IIS worked processes though).

Regarding the "isolation" part of the app pools - the extraction process simply uses the GetTempPath Win32 API - which typically returns the value of %TEMP%. If the two accounts point to the same temp folder, that might be a problem, especially if the access rights are such that files created by one account can't be accessed by the other. The extraction directory doesn't change based on PID, it is a constant path relative to TEMP for any version of the app.

[qcc-na]

I cannot find a way to make this file appear. I tried these combinations and can't find the files. I restarted the vm and IIS in-between each attempt. I set these environment variables from the user interface menu located in: C:\Windows\System32\SystemPropertiesAdvanced.exe

COREHOST_TRACE=1
COREHOST_TRACEFILE=C:\hosttrace.txt

COREHOST_TRACE=1
COREHOST_TRACEFILE=C:\users\me\hosttrace.txt

COREHOST_TRACE=1
COREHOST_TRACEFILE=hosttrace.txt

[vitek-karas]

I honestly don't know how to set this up in IIS. The variables should work though... but I don't know how to verify if they are set or not.

[qcc-na]

Here is some more information that might be useful to those not familiar from: https://docs.microsoft.com/en-us/iis/manage/configuring-security/application-pool-identities

Securing Resources Whenever a new application pool is created, the IIS management process creates a security identifier (SID) that represents the name of the application pool itself. For example, if you create an application pool with the name "MyNewAppPool," a security identifier with the name "MyNewAppPool" is created in the Windows Security system. From this point on, resources can be secured by using this identity. However, the identity is not a real user account; it will not show up as a user in the Windows User Management Console.

Compatibility Issues with Application Pool Identities ... User Profile IIS doesn't load the Windows user profile, but certain applications might take advantage of it anyway to store temporary data. SQL Express is an example of an application that does this. However, a user profile has to be created to store temporary data in either the profile directory or in the registry hive. The user profile for the Network Service account was created by the system and was always available. However, with the switch to unique Application Pool identities, no user profile is created by the system. Only the standard application pools (DefaultAppPool and Classic .NET AppPool) have user profiles on disk. No user profile is created if the Administrator creates a new application pool. . However, if you want, you can configure IIS application pools to load the user profile by setting the LoadUserProfile attribute to "true".

Another workaround noted.

Given this information, the question is whether this default IIS hosting scenario should work given the presence of <IncludeNativeLibrariesForSelfExtract> flag in the project file, because it DOES work without that setting. Part of me thinks it SHOULD be supported, and I think maybe basing the extraction based on %TEMP% alone was a mistake, but I understand the baggage here from IIS.

[vitek-karas]

Honestly this is sounding a lot like a similar problem we had with single-file on Linux in systemd. That's another environment where the app is started under a special user/security environment - and for example the systemd doesn't define the standard $HOME. On Windows this would similar to an environment where there's not %TEMP%. Right now our answer to these cases is that the single-file itself should not try to solve these - as the solution is specific to those special environments.

So I could image that we solve this be having the ASP.NET IIS module set the DOTNET_BUNDLE_EXTRACT_BASE_DIR before starting the child process in such a way that it's not colliding between application pools.

[qcc-na]

That seems logical to me.

[agocke]

Moving to ASP.NET, since it sounds like this would need a change in the IIS hosting module.

[ghost]

We've moved this issue to the Backlog milestone. This means that it is not going to be worked on for the coming release. We will reassess the backlog following the current release and consider this item at that time. To learn more about our issue management process and to have better expectation regarding different types of issues you can read our Triage Process.