Feature Request: Plans to support client authentication?

[mvromer]

Is your feature request related to a problem? Please describe.

I'm currently setting up an ACA environment where a small number of teams can deploy their container apps independently. I would like to use a non-premium SKU of Azure API Management to expose a client-friendly API to those container apps that enable ingress, and I would like to ensure that clients can only connect to those services by going through the APIM gateway.

Describe the solution you'd like.

It would be nice if I could configure ingress on my ACA environment so that it authenticates client connections via client certificates/mTLS. That way I could configure APIM to present a client certificate to my ACA environment backend, and my ACA environment would only allow connections from those clients that presented one of the allowed certificates.

Describe alternatives you've considered.

I first thought about seeing if I could setup IP restrictions on my ACA environment so that it would only allow connections from my APIM backend. But it seems like ACA doesn't allow setting up IP restrictions in this way at this time. Also, I think this would preclude me from using the consumption tier of APIM since your APIM service then doesn't have dedicated IPs you can use for setting IP restrictions.

That's when I figured something akin to mTLS would be a good alternative. It was similar to how I secured communication and access from APIM to a Service Fabric cluster I was building last year for hosting containers. In lieu of ACA directly supporting client authentication on ingress, I think I can achieve this by deploying an Azure App Gateway that does the client authentication. App Gateway can then forward authenticated requests to my ACA apps assuming I deployed the environment to my own VNET and used internal ingress on exposed apps. However, this is obviously more things for me to manage, and being able to manage less Azure infrastructure and reduce the cost of running container workloads was one of the appeals of ACA for me.

Would this (or something like it) be something that would make its way onto the ACA roadmap at some point?

[quality-leftovers]

It would be really great, if we could get the x-forwarded-client-cert header forwarded to the app. Could that be in the near future?

Currently when you connect to a container app via mTLS you will receive the following request in your app

$ curl -k --cert ./my.cert.pem --key ./my.key.pem -X POST -d "hi" https://echo.myapp.northeurope.azurecontainerapps.io/

{
  "path": "/",
  "headers": {
    "host": "echo.myapp.northeurope.azurecontainerapps.io",
    "user-agent": "curl/7.77.0",
    "accept": "*/*",
    "content-length": "2",
    "content-type": "application/x-www-form-urlencoded",
    "x-forwarded-for": "84.60.210.127",
    "x-envoy-external-address": "84.60.210.127",
    "x-request-id": "e66a1d63-5a57-4e37-b0ae-20c0a6a7e690",
    "x-envoy-expected-rq-timeout-ms": "1800000",
    "x-k8se-app-name": "echo--jf0zyv3",
    "x-k8se-app-namespace": "k8se-apps",
    "x-k8se-protocol": "http1",
    "x-k8se-app-kind": "web",
    "x-ms-containerapp-name": "echo",
    "x-ms-containerapp-revision-name": "echo--jf0zyv3",
    "x-arr-ssl": "true",
    "x-forwarded-proto": "https"
  },
  "method": "POST",
  "body": "hi",
  "fresh": false,
  "hostname": "echo.myapp.northeurope.azurecontainerapps.io",
  "ip": "84.60.210.127",
  "ips": [
    "84.60.210.127"
  ],
  "protocol": "https",
  "query": {},
  "subdomains": [
    "northeurope",
    "myapp",
    "echo"
  ],
  "xhr": false,
  "os": {
    "hostname": "echo--jf0zyv3-85ccd545d8-tdsnl"
  },
  "connection": {}
}

[ahmelsayed]

/cc @zhenqxuMSFT for FYI

[kzhen]

+1

As mentioned above by @quality-leftovers if the x-forwarded-client-cert header was sent to Container Apps this would allow us to run APIs that implement mTLS authentication.

[onionhammer]

Definitely need more guidance / documentation on how to protect ACA as a backend.. really Azure has a TON of well documented solutions for creating gateways (APIGW, Frontdoor, APIM, etc), but there isn't a ton of documentation on how to secure backends and restrict them to ONLY those gateways (that AREN'T premium & cost $1500/month).

Googling this comes up pretty dry. Basic security shouldn't cost an arm and a leg.

[ahmelsayed]

You should be able to set clientCertificateMode on the ingress property. See schema here and an example here

or

{ 
  "properties": {
    "configuration": {
      "ingress": {
        "clientCertificateMode": "require | accept | ignore"
      }
    }
  }
}

It's not yet in the cli as an explicit option, but you can patch your app with it in the cli using

# get your app ARM id if you don't have it handy
APP_ID=$(az containerapp show -n app2 -g vnet-container-apps  --query id --output tsv)

# Patch clientCertificateMode property on the app
az rest --method patch \
    --url "https://management.azure.com/$APP_ID?api-version=2022-10-01" \
    --body '{"properties": {"configuration": {"ingress": {"clientCertificateMode": "require"}}}}'

The with require, the client must pass a certificate. with accept it's optional. If a client passes a certificate it'll be made available to the app in X-Forwarded-Client-Cert header in a semicolon separated list

Hash=....;Cert="-----BEGIN CERTIFICATE-----....";Chain="-----BEGIN CERTIFICATE-----...";

[Erikvdv]

@ahmelsayed I have made this update and I do see my requests blocked when I set the mode to "require" when I don't include the client cert. However, when I do use the client cert I don't see a request header with the certificate info

Update: apparently it is required to have a chain in the cert as well. A very basic self signed cert will not be forwarded

[zhenqxuMSFT]

@Erikvdv I just tested with a self signed cert with no chain and it works. Could you send an email to acasupport@microsoft.com with your app name, environment name, as well as how you make calls to your app so that we can take a look? Note, we may make some test calls to your app fqdn during investigation.

[Erikvdv]

@zhenqxuMSFT I just did another check and it is working for me now. Don't know if something has changed on your end. But when I generate a certificate like this, it is passed through now: FILENAME="acatestcert" openssl req -new -newkey rsa:2048 -days 365 -nodes -x509 -keyout $FILENAME.key -out $FILENAME.crt

Any idea when there will be official bicep support for 2022-10-01? it works but the template is not yet known.

[zhenqxuMSFT]

@Erikvdv We do not have changes on our side but it's good to know it works for you now. If you see same issue again, kindly let us know :-)

For bicep thing, I am not 100% sure but you may try it out now. The 2022-10-01 api version is already available on ARM.

[ahmelsayed]

2022-10-01 api should be released and the clientCertificateMode is documented here https://learn.microsoft.com/en-us/azure/templates/microsoft.app/2022-10-01/containerapps?pivots=deployment-language-bicep#ingress

[andrekiba]

It would be nice to have a better documentation about this. Tried with a self signed certificate, it works from a caller in .net core/net 6.0 but always fails in .net full framework (4.5 or 4.8)…seems that during the tls handshake the certificate is not sent at all…so then nothing is present in the x-forwarded-client-cert that is missing. Any idea?

[ahmelsayed]

If I remember correctly, you might need to store the certificate in a Windows Certificate Store to use it with .NET Framework 4.5+ HttpClients.

[andrekiba]

@ahmelsayed the certificate is inside the store, I read it from the store. If I deploy the same containerized .net 6 api (the one that is now on ACA) using a linux azure app service...it works also from .net full framework. In that case the header where I can find the certificate information is the standard one, X-ARR-ClientCert. The deployed docker container image is exactly the same for both, ACA and web api on app service... so I think something different (Envoy vs Gateway in front of app service?) happens during the tls handshake before reaching the api code

[andrekiba]

@ahmelsayed could you please do a test with a self signed certificate, calling an api (.net 6, 7 or core) deployed on ACA from a .net framework 4.5 or 4.8 console application?

[andrekiba]

@ahmelsayed with the trace enabled on caller one thing that I can notice during the tls handshake is this

ACA:

System.Net Information: 0 : [13544] SecureChannel#41622463 - We have user-provided certificates. The server has specified 127 issuer(s). Looking for certificates that match any of the issuers.
System.Net Information: 0 : [13544] SecureChannel#41622463 - Left with 0 client certificates to choose from.

App Service:

System.Net Information: 0 : [34872] SecureChannel#41622463 - We have user-provided certificates. The server has not specified any issuers, so try all the certificates.
System.Net Information: 0 : [34872] SecureChannel#41622463 - Left with 1 client certificates to choose from.
System.Net Information: 0 : [34872] SecureChannel#41622463 - Selected certificate: [Version] V3

During the SERVER HELLO step of the handshake seems that the app service is more permissive, it doesn't specify any issuer allowing the caller to provide it's own self signed certificate (and then is the api code in charge to control the info and eventually trust the certificate or not). On the other hand ACA specifies a list of possible issuers (where do those 127 default ones I see come from?) and this automatically deletes the self signed certificate by not finding any trusted issuers...this leads to an anonymous call --> no x-forwarded-client-cert --> 403 forbidden

[ahmelsayed]

I see thanks for the schannel trace logs. Yes, it seems we're advertising a list of acceptable CAs, those 127 issuer(s). You can see them with

$ openssl s_client -connect {appName}.{env}.{region}.azurecontainerapps.io:443

should be the default mariner linux /etc/ssl/certs/ca-certificates.crt bundle.

It'll still pass whatever cert the client actually sends to the app, which seems to be how curl, go and dotnet core, etc do it. But .net 4.8 + schannel seem to require it to match or be empty. The list is meant to just help the client (like a browser) give feedback to the user about which certs might be acceptable, but since we always pass whatever cert to the app, it doesn't make sense to advertise any particular list there.

I'm not sure if there is a workaround for this. Should have a fix for it from the proxy side on the next update.

[andrekiba]

thank you @ahmelsayed, yes please we really need to call the api also from .net full framework. Do you want me to open another issue to track this? In the meantime if you can think of a workaround...when will be the next update of the proxy?

[ahmelsayed]

Sure, feel free to open another issue on this repo linking to this one, and I'll update it once the deployment has rolled out everywhere.

I'm not really sure about a workaround. All the documented SCHANNEL registry settings I can see only refer to a server side fix. Someone from dotnet might know more, but we can have an update start rolling out by end of this week.

[andrekiba]

@ahmelsayed do we have an update?

[torosent]

We are working on a fix but it will take a few weeks to roll it out to all regions

[andrekiba]

ok but is it possible to know when it will be for north europe?

[andrekiba]

@torosent could you please provide confirmation about the roll out? Seems that it works now.

[ahmelsayed]

yes, this should have rolled out to all regions earlier this week.

[zhenqxuMSFT]

@andrekiba today the "require" mode actually behaves exactly same as "accept" mode, which is inconsistent with the name "require". Recently we had an internal discussion on it and plan to restore the "require" behavior to old one - which advertises default Ubuntu trusted CA and rejects connection if no client cert is provided. For you, we suggest you to change mode type to accept, this won't break your functionality because currently require is same as accept. cc @ahmelsayed

[MarekLani]

@zhenqxuMSFT please any update on require vs accept?

[j94305]

The documentation now specifies the client certificate modes available. In order to enforce mTLS, we could say "require". The documentation also specified the headers used to pass the certificate information to the application. Fine. The documentation mentions the relevant client certificates could be self-signed. Perfect.

What the documentation does not mention (and I have done extensive search in all kinds of resources by now): how do you specify the PKCS#12 keystore with the permitted client certificates? There do not seem to be any parameters in the GUI or the CLI to provide the keystore. Can anybody resolve this mystery?

If this would work, we could use the ingress of ACA essentially like an NGINX in front of a bunch of containers, implementing mTLS and passing proper authentication headers to the applications behind.

Thanks much in advance for any pointers.