search

Azure / AKS

Azure Kubernetes Service
https://azure.github.io/AKS/
1.93k stars 293 forks source link

AKS Private Egress (Network isolated cluster) #3319

Open miwithro opened 1 year ago

miwithro commented 1 year ago

For AKS Private Clusters remove all Egress requirements.

ghost commented 1 year ago

@Azure/aks-pm issue needs labels

denniszielke commented 1 year ago

@miwithro can you outline what scenario is covered by this workitem? I would assume that for Airgapped the API server/ ARM/ AAD endpoints have to be reachable from the vnet (private link or vnet injection) but how will worker nodes have the ability to pull mcr containers, microsoft packages and plugins?

miwithro commented 1 year ago

The desire is to have all egress endpoints eliminated, which essentially means bringing things like MCR/AAD/ARM/Package Manager inside of the customer vnet. With MCR we are working on Pull-Through Cache which will allow for an ACR in the customer vnet to pull AKS specific images from MAR, which will eliminate the mcr.microsoft.com and *.data.mcr.microsoft.com egress requirements.

joshcoburn commented 1 year ago

I would also like to see this feature!

khetman-sas commented 1 year ago

I think this feature is imperative to really gain full market potential!

ToniCipriani commented 1 year ago

The desire is to have all egress endpoints eliminated, which essentially means bringing things like MCR/AAD/ARM/Package Manager inside of the customer vnet. With MCR we are working on Pull-Through Cache which will allow for an ACR in the customer vnet to pull AKS specific images from MAR, which will eliminate the mcr.microsoft.com and *.data.mcr.microsoft.com egress requirements.

I assume that will also require at the cluster level for mcr.microsoft.com to be configurable to an ACR? Or will that be accomplished using private link/DNS to redirect mcr.microsoft.com to said ACR with caching?

CocoWang-wql commented 1 year ago

I assume that will also require at the cluster level for mcr.microsoft.com to be configurable to an ACR? Or will that be accomplished using private link/DNS to redirect mcr.microsoft.com to said ACR with caching?

If the image doesn't exist in private ACR, ACR will leverage pull through cache to pull the image from MCR.

ToniCipriani commented 1 year ago

If the image doesn't exist in private ACR, ACR will leverage pull through cache to pull the image from MCR.

That's fine, but what I'm asking is how will an AKS know to connect to ACR. today it is hardcoded to mcr.microsoft.com which resolves to a public endpoint, even if you attempt to edit the YAML to an ACR.

peterbuecker-form3 commented 1 year ago

This looks to be very promising. Is there any ETA for this work?

CocoWang-wql commented 1 year ago

If the image doesn't exist in private ACR, ACR will leverage pull through cache to pull the image from MCR.

That's fine, but what I'm asking is how will an AKS know to connect to ACR. today it is hardcoded to mcr.microsoft.com which resolves to a public endpoint, even if you attempt to edit the YAML to an ACR.

In our original design, AKS connects to the private ACR via private link.

CocoWang-wql commented 12 months ago

Hello @denniszielke @mac-kul @Ashengross @psddp @Besdima @slawekww @kevinharing @etaham @chudytom @klojtas You reacted to this feature and I assume you are interested in it.

We are seeking more details on your requirements, are you folks looking for BYO (bring your own) ACR or AKS-managed ACR as part of this? Pls let me know if you have any interest in private preview test. Thank you :)

slawekww commented 12 months ago

My use case is the following:

  1. private AKS in Spoke-Hub architecture with Network Virtual Appliance, NVA at another subscription/resource group attached to Hub VNET
  2. many BYO ACRs should be used by AKS, ACRs in deployed at different subscriptions/resource groups For now, ACRs are public, plan to have ACR private visible only in Hub network
  3. authentication to ACR via Managed Identity, refresh token each hour and create pull secret in namespaces
  4. do not change any k8s YAML manifests to access ACRs in case of private ACR - I assume that FQDN for private ACR should be different than public ACR FQDN as defining the same FQDNs for public and private ACR would not be possible

I've not get idea of AKS-managed ACR.

joshcoburn commented 12 months ago

Hello @denniszielke @mac-kul @Ashengross @psddp @Besdima @slawekww @kevinharing @etaham @chudytom @klojtas You reacted to this feature and I assume you are interested in it.

We are seeking more details on your requirements, are you folks looking for BYO (bring your own) ACR or AKS-managed ACR as part of this? Pls let me know if you have any interest in private preview test. Thank you :)

Our specific use case, is we have customers that are using Landing Zones (or some other blanket policy enforcement) that does not allow resources direct egress to public internet (even by exception).

As far as ACR, I think there will be a mix of BYO ACR and AKS-managed ACR.

ToniCipriani commented 12 months ago

Hello @denniszielke @mac-kul @Ashengross @psddp @Besdima @slawekww @kevinharing @etaham @chudytom @klojtas You reacted to this feature and I assume you are interested in it.

We are seeking more details on your requirements, are you folks looking for BYO (bring your own) ACR or AKS-managed ACR as part of this? Pls let me know if you have any interest in private preview test. Thank you :)

In our case we are looking at BYO ACR specifically.

Ashengross commented 11 months ago

Hello,

My customers use cases are environment which are all private including ACR (and even use Nexus and JFROG in some of the use-cases) They work BYO ACR but can explore AKS managed ACR if that will solve in no Egress requirements.

Amir Shengross Sr. Specialist, Azure Mobile: +972-54-7181611 @.**@.>

[Microsoft Logo]

From: Calvin C @.> Date: Tuesday, 11 July 2023 at 15:54 To: Azure/AKS @.> Cc: Mention @.***> Subject: Re: [Azure/AKS] Private Egress AKS (Issue #3319)

Hello @denniszielkehttps://github.com/denniszielke @mac-kulhttps://github.com/mac-kul @Ashengrosshttps://github.com/Ashengross @psddphttps://github.com/psddp @Besdimahttps://github.com/Besdima @slawekwwhttps://github.com/slawekww @kevinharinghttps://github.com/kevinharing @etahamhttps://github.com/etaham @chudytomhttps://github.com/chudytom @klojtashttps://github.com/klojtas You reacted to this feature and I assume you are interested in it.

We are seeking more details on your requirements, are you folks looking for BYO (bring your own) ACR or AKS-managed ACR as part of this? Pls let me know if you have any interest in private preview test. Thank you :)

In our case we are looking at BYO ACR specifically.

— Reply to this email directly, view it on GitHubhttps://github.com/Azure/AKS/issues/3319#issuecomment-1630776663 or unsubscribehttps://github.com/notifications/unsubscribe-auth/AUP6O3GXPG2GSKIB7ZLRSTDXPVEI3BFKMF2HI4TJMJ2XIZLTSWBKK5TBNR2WLJDUOJ2WLJDOMFWWLO3UNBZGKYLEL5YGC4TUNFRWS4DBNZ2F6YLDORUXM2LUPGBKK5TBNR2WLJDUOJ2WLJDOMFWWLLTXMF2GG2C7MFRXI2LWNF2HTAVFOZQWY5LFUVUXG43VMWSG4YLNMWVXI2DSMVQWIX3UPFYGLAVFOZQWY5LFVE3TONBSGU2DANRZURXGC3LFVFUGC427NRQWEZLMQKSXMYLMOVS2UMJQGAYTKMBVGI4TBJDOMFWWLKLIMFZV63DBMJSWZLDTOVRGUZLDORPXI6LQMWWES43TOVSUG33NNVSW45FGORXXA2LDOOKIFJDUPFYGLKTSMVYG643JORXXE6NFOZQWY5LFVEYTAOBSHE3DENRZQKSHI6LQMWSWS43TOVS2K5TBNR2WLKRRGQ2DENBZHE3DGM4CUR2HS4DFUVWGCYTFNSSXMYLMOVS2SNZXGQZDKNBQGY4YFJDUPFYGLJLMMFRGK3FFOZQWY5LFVIYTAMBRGUYDKMRZGCTXI4TJM5TWK4VGMNZGKYLUMU. You are receiving this email because you were mentioned.

Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

CocoWang-wql commented 7 months ago

Hello @denniszielke @mac-kul @Ashengross @psddp @Besdima @slawekww @kevinharing @etaham @chudytom @klojtas Thanks for the prompt response. Airgapped cluster attaches with a private Azure container registry. Can you pls let me know in your scenario, do you need to attach different nodepools with different registries, or it's fine that all the nodepools in the same cluster share a same registery.

chudytom commented 7 months ago

From our side single registry for all node pools would be fine.

mac-kul commented 7 months ago

Our current use cases are mostly about not having public mcr.microsoft.com requirement and it's fine to have one registry for all node pools.

denniszielke commented 7 months ago

Agree all node pools pulling from the same registry is a reasonable assumption

vladimir-setka-form3 commented 3 months ago

Hi @CocoWang-wql, could you provide a progress update? Any ETAs?

CocoWang-wql commented 3 months ago

Hello, we are still in design and will update here once we have a valid ETA. Thanks

sumitkute commented 1 month ago

@CocoWang-wql any update this is a major requirement in FSI sector for private egress or lockdown cluster, so called airgapped.

chudytom commented 1 month ago

@CocoWang-wql @miwithro as this feature is currently our most requested feature for AKS, together with @mac-kul we would be happy to take part in the private preview test. You mentioned that option in one of the previous messages.