Prevent provider configuration from tenant modification

[Gerrit91]

Problem Description

As a provider, we host the Kubernetes control plane of a tenant cluster in the namespace of a provider cluster (Gardener seed <-> shoot architecture, see Gardener docs). This is an effective way for the provider to hide resources from the tenant in order to keep the cluster and the provider infrastructure alive. It cleanly separates responsibilities between provider and tenant.

This is the configuration that should only be configurable through the provider:

• The existence of firewalls for a cluster (the provider is responsible for availability of the cluster firewall and rolling out updates)
• Enabling / disabling IDS (could be a payed feature or have a critical impact on cluster performance, a tenant will be able to configure this only through a provider interface -> shoot spec)
• The prefixes which are considered "internal" in .spec.internalPrefixes (used by the provider for network traffic accounting, tenants should not be able to touch this or even know about this)
• Firewall policies of essential components (e.g. Kubelet -> apiserver, there is no reason that a customer should be able to take down his cluster by mistake)

A tenant should only be able to see / manage the following things on his own (the provider does not need to know about this):

• Deploy own firewall policies (for his applications)
• See firewall drops
• Check firewall status

Possible Implementation

• Pass two kubeconfig files into the firewall controller, reconcile the provider resources from the provider cluster and the tenant resources from the tenant cluster
• Move the Firewall resource into the provider cluster
  • The firewall status still needs to be made available for a customer, either through a web interface in the provider cluster or through a resource in the tenant cluster
  • There shoud be individual names for the firewall resources in order to prepare for firewall HA / firewall rolling upgrades (which could be reconciled from a special firewall-controller-manager in the future)
• Allow the tenant to configuration certain attributes of the Firewall resource through provider interface (shoot spec, e.g. enabling IDS through gardener-extension-provider-metal)

[majst01]

done with the signature

[Gerrit91]

Could be interesting approach again in order to solve #104.