search

MicrosoftDocs / azure-docs

Open source documentation of Microsoft Azure
https://docs.microsoft.com/azure
Creative Commons Attribution 4.0 International
10.26k stars 21.43k forks source link

Securing egress: Why is whitelisting of public ip of api server required? #107506

Closed cveld closed 1 year ago

cveld commented 1 year ago

We are struggling with the guidance that is provided in the paragraph https://learn.microsoft.com/en-us/azure/aks/limit-egress-traffic#restrict-egress-traffic-using-azure-firewall. As soon as we add the 0.0.0.0/0 rule to our firewall, the service address space becomes unreachable. E.g. AKS Azure cni subnet: 10.128.0.0/16 AKS service address space: 10.0.0.0/16

When on a node, we test this with curl 10.0.0.1, i.e. the kubernetes service in the default namespace, also known as the api server endpoint. On first glance this doesn't seem to have anything to do with the public ip of the api server which is used to connect from the internet.

But, as soon as we whitelist the public ip of the api server in the firewall:

The service address space becomes reachable again! I can't get my head around this. What is the network topology here? Why is this firewall rule required? Maybe some obscure NATting is happing below the covers.

The documentation isn't too clear about this requirement. Inside a purple note block there is a brief comment about this:

For applications outside of the kube-system or gatekeeper-system namespaces that needs to talk to the API server, an additional network rule to allow TCP communication to port 443 for the API server IP in addition to adding application rule for fqdn-tag AzureKubernetesService is required.

The first time I did read this I was assuming it was referring to the private ip from the service address space. I would never have thought it was referring to the public ip of the api server. Maybe it is good to include an az cli commandline for this extra rule as well to make it more explicit.

Some other pointers that reference the same: https://github.com/MicrosoftDocs/azure-docs/issues/61470 https://stackoverflow.com/questions/60565449/no-certificate-returned-from-internal-azure-aks-api-server

Document Details

⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

Naveenommi-MSFT commented 1 year ago

@cveld Thanks for your feedback! We will investigate and update as appropriate.

RamanathanChinnappan-MSFT commented 1 year ago

@cveld

Since this issue isn't directly related to improving our docs, and to gain a better understanding of your issue, I'd recommend working closer with our support team via an [Azure support request] (https://docs.microsoft.com/en-us/azure/azure-portal/supportability/how-to-create-azure-support-request). Or you can leverage our Q&A forum by posting your issue there so our community, and MVPs can further assist you in troubleshooting this issue or finding potential workarounds.

[Teams Q&A forum] (https://docs.microsoft.com/en-us/answers/topics/46488/office-teams-windows-itpro.html) for technical questions about the configuration and administration of Microsoft Teams on Windows. [Microsoft Teams Community forum] (https://answers.microsoft.com/en-us/msteams/forum?sort=LastReplyDate&dir=Desc&tab=All&status=all&mod=&modAge=&advFil=&postedAfter=&postedBefore=&threadType=All&isFilterExpanded=false&page=1

cveld commented 1 year ago

@RamanathanChinnappan-MSFT I tend to disagree. The docs don't mention it too clearly. So I would vote to make it more visible in the documentation. And, it would be great to get some explanation added to the docs. This would have saved us a couple of days of troubleshooting.

RamanathanChinnappan-MSFT commented 1 year ago

@cveld

I've delegated this to @asudbring , a content author, to review and share their valuable insights.

schaffererin commented 1 year ago

reassign:schaffererin

schaffererin commented 1 year ago

Hi, @cveld, thank you for your feedback.

Adding the /0 routing rule starts forwarding all your traffic to the firewall. Without that rule, the firewall is bypassed entirely. When you add the /0 routing rule, the traffic starts to go through the firewall. This is why the rule is required. If your firewall is configured to block the traffic, the traffic will be blocked. If your firewall is configured to not block the traffic, the traffic will be allowed through.

Could you please provide more details regarding your suggestion to improve the docs? What would be the most helpful?

Thanks!

schaffererin commented 1 year ago

As we have not heard back, we will now close this GitHub issue. Thank you! #please-close

mjnovice commented 5 months ago

Can we re-open this with an explanation ? Did you eventually get any answer @cveld ?

cveld commented 5 months ago

Yes. The api server is hosted inside the host control plane (hcp) and must be reachable. You could also go for custom routing instead of "public load balancer" routing.

There is a preview though which enables vnet integration. This makes dropping the public ip much simpler.

Op za 25 mei 2024 02:41 schreef Mayank Jha @.***>:

Can we re-open this with an explanation ? Did you eventually get any answer @cveld https://github.com/cveld ?

— Reply to this email directly, view it on GitHub https://github.com/MicrosoftDocs/azure-docs/issues/107506#issuecomment-2130603425, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABPIYJGMBI57JJYSD6NZ2ZTZD7M5NAVCNFSM6AAAAAAWOTBC5KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMZQGYYDGNBSGU . You are receiving this because you were mentioned.Message ID: @.***>