Feature Request: Azure Front Door ingress for private environment support/example

[madhancock]

Is your feature request related to a problem? Please describe.
I've put this up on the Front Door community also: https://feedback.azure.com/d365community/idea/01498c46-b232-ed11-a81b-000d3ae3db6e

Either documentation on how to correctly use Azure Front Door as a gateway to a private container environment. This seems to be possible using AKS using a private link to the internal load balancer and, as you'll see in the link - I was able to get it working for a short time. But there was also some strange behaviour that I was not able to confirm if it was caused by Front Door or ACA.

Describe the solution you'd like.
Documentation or confirmation it's not currently possible.

Describe alternatives you've considered.
I've tried following examples with AKS, however the origin configuration needed to prevent unexpected behaviours and to ensure ingress functions as expected isn't 100% clear:

Additional context.
Being able to do this would allow for use of WAF features and traffic management/auditing, ensuring those features cannot be bypassed by going direct to the ACA URLs (which would be possible if a public ACA was used as an origin).

[kendallroden]

Thanks for your request. I will dig into this and update when possible

[kendallroden]

@swgriffith

[Ricky-G]

Keen to know if its on the road map, private endpoint support for ACA

[madhancock]

@kendallroden Any update you can share on this?

[kendallroden]

@madhancock we are working on documentation, but is there any way you can elaborate on the "strange behaviors" you were observing?

[Masahigo]

According to MS docs integrating AFD with ACA in internal mode is not officially supported so curious to hear if this works after all. This would definitely simplify the overall architecture required for this kind of setup.

[madhancock]

@madhancock we are working on documentation, but is there any way you can elaborate on the "strange behaviours" you were observing?

I suspect the odd behaviours were to do with propagation. It's entirely possible that one of my iterations worked but unfortunately AFD gives no indicator on how far the latest changes to the configuration have propagated to the AFD network.

Because of this you often get a different response back from AFD every few minutes but you're unsure if it's because you hit a cold box or because what you thought was your change working was actually an earlier change taking effect and a subsequent change has now rendered it inoperative.

@Masahigo the way I think I had this working was with a private link to the internal LB of the ACA environment. Based on the docs you could say it is half supported - but I'd like to see some something more official from MS on this. I've not checked it out yet but #512 seems to implement what I was going for.

@kendallroden It would be reassuring to have a comment on the solution in #512 from @sebafo in a more "official" capacity. Particularly what would be needed to consider that a prod-ready setup.

[Masahigo]

@madhancock What about the traffic routing via Envoy. Is it enough to have just one private endpoint per microservice (in internal mode in ACA) and you can still use the multiple revision mode?

Probably the biggest pain point I've struggled with Azure's private networking in App Service is the fact that once you introduce it, every deployment slot requires its own. I'm hoping to get rid of such restrictions by moving to ACA.

[madhancock]

@madhancock What about the traffic routing via Envoy. Is it enough to have just one private endpoint per microservice (in internal mode in ACA) and you can still use the multiple revision mode?

Probably the biggest pain point I've struggled with Azure's private networking in App Service is the fact that once you introduce it, every deployment slot requires its own. I'm hoping to get rid of such restrictions by moving to ACA.

The private link actually goes direct to the internal LB of the ACA environment. So in theory you only need one private link and that enables you to route to any number of apps via AFD.

Presumably, the origin host header is used to identify the target app which in turn routes the traffic to the appropriate envoy instance within the environment (although I've not had a chance to test this out with multiple apps yet).

ACA is certainly better in this regard than App Service. I'd encourage the use of IaC with any of this.

[Masahigo]

Presumably, the origin host header is used to identify the target app which in turn routes the traffic to the appropriate envoy instance within the environment (although I've not had a chance to test this out with multiple apps yet).

Appreciate the insights! Planning to test this soon.

[maskati]

I am testing this setup at the moment and planning its use in a production scenario, so I would also appreciate clarification on official support.

According to MS docs integrating AFD with ACA in internal mode is not officially supported so curious to hear if this works after all. This would definitely simplify the overall architecture required for this kind of setup.

Private link to load balancers is officially supported, so my main concern is whether connecting a private link service to ACA managed ILB is a supported scenario. From the ACA documentation:

When you deploy an internal or an external environment into your own network, a new resource group prefixed with MC_ is created in the Azure subscription where your environment is hosted. This resource group contains infrastructure components managed by the Azure Container Apps platform, and shouldn't be modified.

Specifically the last part "shouldn't be modified". My understanding is that technically adding a private link service to a load balancer does not modify the load balancer, but it does add an external Azure dependency on that ACA managed load balancer resource. So I suppose official supportability depends on the interpretation of "modified"?

The private link actually goes direct to the internal LB of the ACA environment. So in theory you only need one private link and that enables you to route to any number of apps via AFD.

@madhancock unfortunately AFD does not allow rewriting the request host header in routes or rules, but these must be defined in the origin configuration. Each AFD origin requires its own private endpoint, so it seems that this would therefore require an AFD origin and separate private endpoint for each ACA container app.

[sebafo]

Because my repo (https://github.com/sebafo/frontdoor-container-apps) was mentioned in the related issue some thoughts about a solution. To deploy AFD with a private ACA environment you need to create a private link connection between Front Door and the internal Azure Load Balancer. For that reason you have to create a Private Link Service for the Load balancer. The challenge is now to 'find' the Load Balancer the Private Link Service should be created for and add the ID of the Load Balancer to the Private Link Service resource. In my example I used the default domain of the environment to create the name of the 'auto-generated' resource group, because you need to provide the name and the resource group of the Load Balancer to get the required ID. Bicep example code:

// Create Container Apps Environment
resource environment 'Microsoft.App/managedEnvironments@2022-03-01' = {
  name: environmentName
  location: location

  ...

}

// Get the Default Domain of the ACA environment
var containerAppsEnvironmentDefaultDomain string = environment.properties.defaultDomain

// Split the domain to get the identifier of the ACA environment (the element before the location)
var containerAppsNameIdentifier = split(containerAppsDefaultDomainName, '.')[lastIndexOf(containerAppsDefaultDomainArray, location)-1]

// Use the identifier to 'generate' the resource group name
var containerAppsManagedResourceGroup = 'MC_${containerAppsNameIdentifier}-rg_${containerAppsNameIdentifier}_${location}'

// Get the ID of the Load Balancer
resource loadBalancer 'Microsoft.Network/loadBalancers@2021-05-01' existing = {
  name: 'kubernetes-internal'
  scope: resourceGroup(containerAppsManagedResourceGroup)
}

This is a bit 'hacky' and for that reason I wouldn't call the solution production ready. There might be moving targets in it, like a changed naming convention etc. Perhaps there is already something more elegant that I am just missing?

An easy solution for me would be to provide the name of the resource group, that contains the 'kubernetes-internal' Load Balancer or the ID of this Load Balancer directly as part of an output variable/property after creating the ACA environment.

That name of the RG or the ID can be used to create all relevant Private Link connections afterwards.

[maskati]

@sebafo I am doing something similar. You can actually "find" the correct ILB by searching the associated subscription load balancers for a load balancer having the tag managed environment resource id matching the resource id of your Container Apps environment. This however requires scripting and is not usable in a templated deployment since it requires enumeration of Microsoft.Network/loadBalancers.

[jasonjohnlee]

Hi, this article was published by a Microsoft employee Chris Bellée on Feb 1, 2023. It details a way to connect Front Door to ACA via a Private Link Service resource and the Internal Load balancer that ACA automatically creates. I gave it a try and it does work. Is this official guidance from Microsoft? Specifically, is the Internal Load Balancer on which this architecture depends a stable resource that will not be randomly destroyed/recreated (e.g. security patch, Infra-as-Code/ARM/Terraform update, etc...)?

[jasonjohnlee]

Hi All, FYI I got confirmation from Microsoft via Microsoft Q&A that setting up a private link between Front Door and ACA's internal load balancer is indeed supported. After a week without a response on Github, I posted the question there as I find the responses from MS are very prompt when compared to Github issues. ;)

[maskati]

However, the only official note in the ACA docs regarding this setup is that we are not allowed to touch the managed networking resources such as the ILB. I have got the setup working fine, however I am at the same time hesitant to promote this as a supported architecture without official confirmation, and while having this warning in the docs.

[jasonjohnlee]

Hi @maskati , yes, that exact sentence is what was making me hesitant too. I quoted that exact article in my Microsoft Q&A as well. Even though MS confirmed internally on the Q&A (please look at comments in the "answer" too) that it's fine, I do agree that I'll feel much more comfortable if Microsoft publishes an article formally detailing/confirming this architecture so that could be an action item of this Github issue (I see @cachai2 has assigned it to himself so maybe that's something he can follow up on?). In theory (maybe I'm wrong), but connecting a Private Link to the Load Balancer is not really "modifying" it as it's creating the network tunnel to the Load Balancer rather than changing anything about the Load Balancer itself. I can't think of any other reason why they'd make the networking resources in the "MC_" group visible while hiding everything else (e.g. AKS). I'm more worried about the load balancer getting randomly destroyed and created during a patch or some other update, which will break the Private Link.

[maskati]

I agree regarding private link not actually updating the load balancer, but it does create an Azure dependency. Could creating a private link prevent ACA from updating networking infrastructure? Not sure what happens if you try to delete a load balancer referenced by a private link, would Azure block that thereby breaking ACA maintenance? It all depends on how ACA internally updates/manages the lifecycle of MC resources. Which is why I worry about suggesting it as a supported configuration to customers without official confirmation…

[maskati]

Not sure what happens if you try to delete a load balancer referenced by a private link, would Azure block that thereby breaking ACA maintenance?

I tested this, and Azure does in fact restrict changes to the load balancer when it is attached to a private link service. For example trying to delete the load balancer results in:

Failed to delete load balancer
Error: Cannot delete load balancer since it is referenced by private link service /subscriptions/.../resourceGroups/.../providers/Microsoft.Network/privateLinkServices/...
Please delete private link service prior to deleting load balancer.

Not sure what kind of operations the ACA platform performs on managed resources, but there is at least some risk of breaking ACA processes by adding a private link to the load balancer.

[madhancock]

@kendallroden or @cachai2 - is there any way we can get a solid update on this given we're closing on 8 months since this was raised. It's been great to see the community try and work around this but nothing from the actual ACA team to say "yes solution X is actually fine".

Without that, it's dead in the water. No business is going to deploy something like this to production for 24/7 business critical applications if the workarounds suggested are not recommended by Microsoft Azure.

In summary, I think some have concluded this is doable and these is general stability in the architecture - but the sticking point is the solution violates rules about modification of the ACA resources.

Please can someone from the ACA team have a look at this and determine one of the following:

1. The suggested method is absolutely fine and the action to close this issue is to document it officially.
2. The suggested solution is NOT fine and changes to ACA (and possibly AFD) would be required to properly support this.
3. Other

Thanks!

[onionhammer]

I would love to just see an FDID restriction setting I can put into my bicep - just like in app service.

[maskati]

I would love to just see an FDID restriction setting I can put into my bicep - just like in app service.

That is mostly useful when you can combine it with the AzureFrontDoor.Backend service tag, in which case you can be pretty sure that the request is coming from that specific Azure Front Door resource (since Microsoft controls the service tag associated IP range). On its own FDID restriction is just HTTP header based shared key / basic authentication.

[onionhammer]

I would love to just see an FDID restriction setting I can put into my bicep - just like in app service.

That is mostly useful when you can combine it with the AzureFrontDoor.Backend service tag, in which case you can be pretty sure that the request is coming from that specific Azure Front Door resource (since Microsoft controls the service tag associated IP range). On its own FDID restriction is just HTTP header based shared key / basic authentication.

That goes without saying, yes

[maskati]

The container apps Consumption plan creates a load balancer with a NIC based backend pool. In contrast the new container apps Consumption + Dedicated plan creates a load balancer with an IP based backend pool. Note from load balancer backend pool management limitations section:

Private endpoint resources can't be placed in an IP based backend pool

And indeed trying to create a private link service on such a load balancer results in PrivateLinkServiceIsNotSupportedForIPBasedLoadBalancer. So the Front Door private link to container apps configuration described earlier in this issue will not work in the Consumption + Dedicated plan.

The currently documented and supported way to publish private container apps is to use an Application Gateway. One potential way of doing this using Front Door is to use the still preview Application Gateway private link support to publish an application gateway private frontend IP as a private link service, and then connect to that from Front Door using private endpoint (I have not attempted this yet).

[chriswangelin]

@SophCarp, @cachai2 - I'm following up on @madhancock's comment from May 4th:

It would be hugely appreciated if we could get a response on this! :)

[cachai2]

Hi folks, apologies for the delay here. The recommended path is to use an Application Gateway in front of an internal only container apps and setup private endpoint there to connect with resources such as Azure Front Door privately. This will work for both Consumption only plans as well as workload profiles environments.

That being said, for the Consumption only plan, private endpoints can be placed on the load balancer, and this solution should work fine. @maskati is right that this behavior is different for the Consumption + Dedicated plan structure. We are looking into providing a similar experience for workload profiles environments. Once the experience is the same across environments, we will document it officially.

[GurliGebis]

@cachai2 just to be sure - doesn't this require using the Premium SKU of the AFD? (I'm working on some projects where the Premium SKU is not an option at the moment)

[mmruesch12]

This is something I would like to see as well. In my situation we are using Container Apps at scale and a simple solution for adding the IP range restriction and checking the header would be greatly helpful. It's certainly a cheaper and more easily scalable option.

Has there been any confirmation as to whether or not this is an intended feature for ACA?

[CamiloTerevinto]

Hi folks, apologies for the delay here. The recommended path is to use an Application Gateway in front of an internal only container apps and setup private endpoint there to connect with resources such as Azure Front Door privately. This will work for both Consumption only plans as well as workload profiles environments.

That being said, for the Consumption only plan, private endpoints can be placed on the load balancer, and this solution should work fine. @maskati is right that this behavior is different for the Consumption + Dedicated plan structure. We are looking into providing a similar experience for workload profiles environments. Once the experience is the same across environments, we will document it officially.

Hi @cachai2, has there been any progress on supporting this for workload profile environments? We are looking to do FD->ACA directly through Private Link for separate products on separate subscriptions so using AppGateway is not something I can justify the costs for.

In the meantime, it would be great if this was documented upfront in a central place (like this doc https://learn.microsoft.com/en-us/azure/container-apps/networking) instead of making us Google until we find this issue.

[onionhammer]

Doesnt FD -> ACA private link require a premium front door?

at the very least, the default should be a setting (similar to what azure app service has) which allows you to restrict to a configured FDID header.

https://learn.microsoft.com/en-us/azure/app-service/app-service-ip-restrictions?tabs=azurecli#access-restriction-advanced-scenarios

[fjvela]

Hello!

Any update?

[simonjj]

Closing this and consolidating this as part of #867. Please add any additional comments there.