Open EklipZgit opened 1 year ago
I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.
Tagging subscribers to this area: @dotnet/ncl, @vcsjones See info in area-owners.md if you want to be subscribed.
Author: | EklipZgit |
---|---|
Assignees: | - |
Labels: | `area-System.Net.Security`, `untriaged` |
Milestone: | - |
The behavior you describing comes from X509Chain
. AFAIK it would cache the certificates as long as the user's home is writable. If you have simple app, you can try it with strace
@EklipZgit. You can probably reproduce it even without doing any ssl.
@bartonjs is expert on this.
Tagging subscribers to this area: @dotnet/area-system-security, @vcsjones See info in area-owners.md if you want to be subscribed.
Author: | EklipZgit |
---|---|
Assignees: | - |
Labels: | `area-System.Security`, `untriaged` |
Milestone: | - |
Etw logs seem to show this entry: ExceptionMessage="The owner of '/home/dotnet/.dotnet/corefx/cryptography/x509stores/ca' is not the current user"
Which is odd because dotnet clearly created this directory and wrote stuff to the directory next to it. There's nothing special in our containers, and the entirety of omitted parts of my dockerfile are in the build/publish image steps, not the base/final images, so nothing dotnet gets run in that image besides entry point.
Took me a bit but I got it running with strace, these were the changes I made
(stop installing the workaround intermediate ca cert, installing strace as root, go back to running strace as root (wont run under dotnet user) + telling strace to run the command under the dotnet user instead).
Verified problem reproducing during this time, so the strace run-as-user worked correctly:
Strace log file: strace.log
For what its worth, I don't see anything in there that mentions part of that directory structure, but I don't really know what you're looking for in here anyway.
Can you check what UID is dotnet
user in the container and check directory ownership all the way to home?
Hopefully @bartonjs will chime in with some more insight. I don't know if or why the X509Chain would care but I;'ve seen something like this while about with openssh.
how would I check directory ownership all the way to home? Is there something specific you'd like run? Note that I don't have SSH access to the container in Azure and I've not bothered to set up a local repro since the cert is injected via azure (although I have the raw binary now of what Azure injects so I could in theory probably set up a full local repro, just haven't bothered).
If theres some dotnet way to grab the info you need, I already have a debug endpoint to recursively walk the file system and write stuff back out (but note, that's running as the dotnet user), that's probably easier for me than any shell command + setting up the container repro locally if that works. If shell command as root is only option though, I can set that up. Just need to know what you're interested in (as a non linux user I have no idea what you're looking for)
I was thinking about good old ls -al
but that won't work without SSH ;(
Let's wait for @bartonjs to avoid unnecessary work.
If you can run the Docker image locally, you should be able to just do something like docker exec -it <containerid> /bin/bash
against the already running container to have a shell to snoop around inside. ls -al /home/dotnet/.dotnet/corefx/cryptography/x509stores/
will presumably show the directory owner to be something other than your "dotnet" user. (If that shows as "dotnet", then geteuid()
said you were being someone else)
As for why CRLs would be cached but certs not... certs in the cert store might have associated private keys, so the cert store treats everything as confidential. CRLs don't really have user secrets in them, so we don't use as stringent of checks before allowing them to be written to disk.
So we're trying to reproduce in a local container, and it actually works fine so the assumption is that something that Azure AppService / Kudu modifies the container with is what is causing the issue. The way I set up strace above still doesn't include any of the stuff we're looking for, even locally, so I must just be missing some strace params or something (any suggestions, if either of you still care?)
I got an endpoint in the appservice running ls
commands (as dotnet user).
total 0
drwxrwxrwx 2 nobody nogroup 0 Nov 22 22:20 .
drwxrwxrwx 2 nobody nogroup 0 Nov 22 22:20 ..
drwxrwxrwx 2 nobody nogroup 0 Nov 22 22:20 ca
Is what we have for ls -al /home/dotnet/.dotnet/corefx/cryptography/x509stores/
so indeed, the owner is not what is expected.
Azure AppServices must do something to map the filesystem out for Kudu to read outside the container but I'm not sure what the implications of that are on the access. Given nobody nogroup
above I imagine this might be a more likely culprit
https://learn.microsoft.com/en-us/archive/blogs/waws/things-you-should-know-web-apps-and-linux#you-cannot-change-permissions-on-the-home-directory-when-persisting-storageapplies-to-web-app-for-containers
which I'm not 100% sure applies to our appservice deployment -- still need to poke around and see how this is set up -- but the section right above that says the default is true. So by default in Azure AppServices hosting Linux containers, you have no control over the permissions in the /home directory. Maybe /home isn't a safe place for this cache to be stored if intending to play nice with Azure AppServices?
I made sure we were following the standard guidelines for running aspnet webapi on linux as non-root and saw no mention of this as a pitfall.
From a behavior perspective, perhaps private key-less ca certs (which anything from AIA would be?) shouldn't be subject to the same write-checks that other forms of cert writing are? Different store, or something? I guess, regardless of whatever weird thing Azure AppServices is doing to this container, the behavior we're getting still seems a bit questionable. Even a terminating error would probably have been preferable vs discovering this when Sectigo goes down and every request we make suddenly skyrockets to 30 seconds of timeouts trying to redownload the cert until they come back up (2 cert downloads per call for whatever reason, 15 second timeout on each is the behavior we saw).
I'll keep poking around and try to sort out specifically what Azure AppServices is doing that changes these permissions.
is there anything else in that directory? That + timestamp may be hint what is creating it. running as nobody
is not that unusual but HOME
must be probably set for dotnet
already.
You can try to remove it as last step of the image creation. That would give hint if this happens during building the image or on first run.
I also often set
DOTNET_CLI_TELEMETRY_OPTOUT=1
DOTNET_SKIP_FIRST_TIME_EXPERIENCE=1
to avoid interference with base function.
BTW when fiddling with strace
make sure you pass -f
flag. Based on the discussion this may not find any more info but it is great tool for cases like this.
I just edited the above comment (had a bad SO link originally), but check out https://learn.microsoft.com/en-us/archive/blogs/waws/things-you-should-know-web-apps-and-linux#you-cannot-change-permissions-on-the-home-directory-when-persisting-storageapplies-to-web-app-for-containers this section and the section right above it.
I tried playing with this directory during image creation and it certainly didn't exist during the docker build, but given the above info it looks like by default it just persists between container deployments and always has 777 permissions, even if you try to change them. The runtime and Azure AppServices appear to have conflicting assumptions about what can be done with the /home directory.
FYI: Those ENVs should have no effect since they are SDK oriented and nothing in the SDK should be affecting the final ASP.NET images.
I'll ask someone in app service for help. Thanks for the report! This is quite interesting.
Description
The quick overview:
FROM mcr.microsoft.com/dotnet/aspnet:6.0 AS base
httpClientBuilder.UseClientCertificate(lstCertThumbprint);
Reproduction Steps
Hopefully this is reasonably easy for someone to reproduce who is familiar with the X509 parts of the runtime. I don't have time to sort out how to try to reproduce this behavior with a fake CA cert with AIA data generated via openSSL so hopefully this is easy to reproduce.
asp net 6 web api. linux docker container. This is the entire docker file with the exception of a swagger generator util and unit test run steps for brevity.
In the web api itself, here's the relevant bits of the csproj for reference:
Program.cs logging, shouldn't really matter but including just in case somehow relevant:
And in its Startup.cs, 3 different methods of setting up HttpClients with client certificates all repro the issue: ConfigureServices
We inject the cert into the Azure AppService via the out of the box Azure AppService private cert upload https://learn.microsoft.com/en-us/azure/app-service/configure-ssl-certificate?tabs=apex%2Cportal
Note that the top level CA in the chain of the cert should be in the linux containers ca store, however the mid-level intermediate in the chain should NOT be, and the cert must include AIA information in its x509v3 chain with which to download the DER of the intermediate cert in order to fully reproduce our scenario.
Then just use some of those http clients to make some mutual TLS calls. If you monitor traffic from the container via AppInsights, you'll see the dependency calls going out to the AIA target url. I assume you could also capture traffic with Fiddler or something instead.
Expected behavior
When running as non-root, dotnet should download the missing cert chain via AIA and save it in the local dotnet /ca/ store just like it does when running as root, and should not continue to redownload the chain for each subsequent request.
Actual behavior
After changing the user our dotnet api was running under from root to non-root, we noticed a massive uptick (from basically zero to >3million per day) in dependency calls to http://crt.sectigo.com/SectigoRSAOrganizationValidationSecureServerCA.crt when making a mutual TLS call to our internal gateway (which has a server SSL cert with the same chain as the mutual TLS cert we send). Every single outbound https call to our gateway would make two GET requests to this sectigo URL. We had absolutely no references to that url ourselves so we began digging and found that that URL comes from the Sectigo certificates AIA section, and from there learned about how AIA is used to download incomplete chains.
We converted the service back to running as root user, and found that it made the download request twice on first http call and then never makes it again for the life of the container (still why twice? two chain paths?). After digging through dotnet runtime issues and various other places we found that on linux the place where dotnet is supposed to be storing the just-in-time AIA chains is here and I added some debug endpoints to dump the internals of the container as we don't have SSH enabled for security reasons:
(note /root/ when running as root, otherwise the users home when running as non-root).
Here you can see that we have two revocations cached in the /crls/ folder and 4 pfx's cached in the /ca/ folder.
When running as non-root, dotnet still creates this directory structure including the /ca/ folder (under the non-root users home dir), and populates the two /crls/ entries (so clearly it has write access to this directory structure). However, the /ca/ folder remains empty. Etw logs seem to show this entry:
ExceptionMessage="The owner of '/home/dotnet/.dotnet/corefx/cryptography/x509stores/ca' is not the current user"
Since both the gateways SSL cert and the client mutual TLS cert use the same offending chain, we eliminated the mutual TLS cert from the equation by ceasing to send it and removing the requirement for it from the gateway, and the sectigo calls vanished entirely so we can say for certain that it is the mutual TLS client cert that triggers this behavior (and not the target servers SSL cert).
Presumably something is going on that's causing dotnet to be unable / refuse to cache these missing parts of the cert chain that dotnet wants to send to the gateway as part of the mutual TLS handshake and is redownloading them (twice..?) from the AIA information in every request when running as non-root.
Once I understood the problem domain enough I was able to work around the issue by explicitly installing the .crt version of http://crt.sectigo.com/SectigoRSAOrganizationValidationSecureServerCA.crt (which is actually a DER, not crt, not relevant to this bug as .NET doesn't care about the extension when parsing but mentioning in case someone looks at it and ends up trying to use / test it as a CRT when it is in fact a DER. The equivalent CRT can actually be obtained from here https://support.sectigo.com/articles/Knowledge/Sectigo-Intermediate-Certificates ). With the intermediate ca cert explicitly imported into the linux ca store, we now get 0 calls to sectigo (vs 2 calls on first http call when running as root, and vs 2 calls per one http request when running as non-root). So now notably I'm assuming the reason many people haven't run into this before is likely because A) they're running as root, or B) the mutual TLS certs they're using have the full chain on the box (or have the full chain self contained in the cert they use? Not sure if that's a normal thing).
So while I have worked around the issue, I couldn't find any documentation of others running into the same thing, closest relative seems to be this, https://github.com/dotnet/runtime/issues/29653 which I learned a lot of useful info from. At a glance I'm guessing the problem is around here https://github.com/dotnet/runtime/blob/983c8f239a98812498d874ed36b7001bd764fdfe/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/X509Certificates/OpenSslDirectoryBasedStoreProvider.cs or here https://github.com/dotnet/runtime/blob/af263f7a2b0a309b5ac79ad92f4f7217da906b78/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/X509Certificates/OpenSslX509ChainProcessor.cs#L268 but I'm far out of my comfort zone at this point so it's just pure speculation.
Regression?
Dont know.
Known Workarounds
Install the intermediate certs into the container with Docker
so that dotnet doesn't need to try to download them from AIA.
Configuration
.NET 6
Debian 11, per the 6.0 tag here: https://hub.docker.com/_/microsoft-dotnet-aspnet#:~:text=6.0.13%2Dbullseye%2Dslim%2Damd64%2C%206.0%2Dbullseye%2Dslim%2Damd64%2C%206.0.13%2Dbullseye%2Dslim%2C%206.0%2Dbullseye%2Dslim%2C%206.0.13%2C%206.0
unrelated to port 8080 change, running as root port 8080 does not reproduce the issue.
Other information
No response