search

Azure / AKS

Azure Kubernetes Service
https://azure.github.io/AKS/
1.97k stars 308 forks source link

User added labels on kube system disappear unexpectedly over time #879

Closed tomecho closed 4 years ago

tomecho commented 5 years ago

What happened: I added a label to kube-system to select it for NetworkPolicies through kube-router

What you expected to happen: The label to persist forever.

How to reproduce it (as minimally and precisely as possible): Add a label to kube-system, wait some time and it will disappear

Anything else we need to know?:

Environment:

jnoller commented 5 years ago

Duplicate: https://github.com/Azure/AKS/issues/468

tomecho commented 5 years ago

I dont believe this is the same issue @jnoller, these are specifically labels on a namespace resource.

jnoller commented 5 years ago

@tomecho I'll reopen then, note kube-system is a managed resource by aks and user applied changes would not take effect.

slyimperator commented 5 years ago

Hmm, then does that mean that resources in kube-system cannot be safely referenced with namespaceSelector in a networkpolicy resource? Following the doc at https://docs.microsoft.com/en-us/azure/aks/ingress-tls places the nginx ingress controller into the kube-system namespace meaning that a namespace selector would be needed to allow traffic from the ingress reverse proxy to a application in another namespace.

jnoller commented 5 years ago

@slyimperator Yep, kube-system pods are set to reconcile (as you've probably guessed) - discussing with the wider team.

palma21 commented 5 years ago

Hi @slyimperator, @tomecho,

Yes today to achieve that you need to use the pod tags. This has been flagged during this preview phase and we've already scoped adding a AKS-specific tag to the kube-system namespace and pods that will persist, this should make your use case simpler, let us know if it doesn't.

Thanks for the note on the docs. You probably would want to put that on a NS you control instead of kube-system. We'll get that fixed/changed.

garethmorris78 commented 4 years ago

Hi, is there an ETA for this functionality? Not being able to allow ingress/egress traffic to/from kube-system across pod namespaces using a namespaceselector pretty much precludes you from using a simple global network policy in calico i.e. 

apiVersion: projectcalico.org/v3
kind: GlobalNetworkPolicy
metadata:
  name: global-deny-all
spec:
  order: 2000
  types:
  - Ingress
  - Egress

  # egress network rules
  egress:
  # Allow all egress traffic from kube-system.
  - action: Allow
    destination: {}
    source:
      namespaceSelector: name == 'kube-system'

  # ingress network rules
  ingress:
  # Allow all ingress traffic for the kube-system namespace.
  - action: Allow
    destination:
      namespaceSelector: name == 'kube-system'
    source: {}

source:https://octetz.com/docs/2019/2019-04-22-netpol-api-k8s/

This would "whitelist" kube-system traffic and you are then able to apply more granular rules between user namespaces etc.

JasonWhall commented 4 years ago

+1 Have also been affected by this. I've updated my network policies to use one of the pre-existing labels on kube-system namespace when first creating a cluster, which is control-plane: "true". Definitely not guaranteed but has the desired effect for now.

For reference some of the other labels that are present on the namespace:

  labels:
    addonmanager.kubernetes.io/mode: Reconcile
    control-plane: "true"
    kubernetes.io/cluster-service: "true"

However a quick dig shows kubernetes.io/cluster-service: "true" is being deprecated

Example policy:

# Allow Outbound DNS calls
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns-access
spec:
  podSelector:
    matchLabels: {}
  policyTypes:
  - Egress
  egress:
  - to:
    - namespaceSelector:
        matchLabels:
          control-plane: "true" # expects this label to exist
    ports:
    - protocol: UDP
      port: 53

It's also worth pointing out Calico explicitly mention to label the kube-system namespace to use this for targeting network policies on DNS here - https://docs.projectcalico.org/security/tutorials/kubernetes-policy-advanced and other guides/documentation also recommend labelling the namespace as standard practice.

If user defined labels on the namespace are not guaranteed, It would be nice instead to have a guaranteed AKS provided label on the kube-system namespace that we can use for network policy targeting.

palma21 commented 4 years ago

control-plane is currently the label we provide and we won't be removing it. we will be adding a few new more.

we don't have plan allowing to alter kube-system, but open to any additional requests you might have

garethmorris78 commented 4 years ago

As long as the control-plane metadata label is guaranteed never to be altered then it can be used for network policies. It is still less of a risk though to base it on a label that i can configure and control Vs having that dependency on a 3rd party . But understand the need for MSFT to control the Kube-system namespace. I've checked the CHANGELOG in this repo for AKS and label changes are communicated so if there ever was a need to change it I'd know about it ahead of time.

ghost commented 4 years ago

This issue has been automatically marked as stale because it has not had any activity for 60 days. It will be closed if no further activity occurs within 15 days of this comment.

ghost commented 4 years ago

This issue will now be closed because it hasn't had any activity for 15 days after stale. tomecho feel free to comment again on the next 7 days to reopen or open a new issue after that time if you still have a question/issue or suggestion.