search

kubernetes-sigs / cluster-api

Home for Cluster API, a subproject of sig-cluster-lifecycle
https://cluster-api.sigs.k8s.io
Apache License 2.0
3.5k stars 1.3k forks source link

Http proxy support for egress traffic #3751

Closed hanlins closed 3 years ago

hanlins commented 3 years ago

User Story

As a operator I would like to proxy cluster egress traffic for security compliance As a operator I would like to store proxy credentials in secret(s) and consume them securely As a user I would like to update cluster proxy settings via cluster API As a user I would like to view proxy configurations being applied for my cluster

Detailed Description

In some corporation network, security policies or firewall rules will be applied to block egress traffic to prevent data exfiltration, which will block the traffic for pulling images from external image registry or sending telemetry data. A proxy is required in this scenario to ensure the egress traffic can be managed in a secure way.

Anything else you would like to add:

With current v1alpha3 implementation, we could potentially inject proxy configs to container runtime via the Files hook, however it has following problems:

  1. It's making assumption that kubeadm/cloud-init is being provided.
  2. You need to expose the credentials in the kubeadmconfig object.
  3. It's not supporting reconfiguration. The Files or PreKubeadmCommands hook are making use of cloud-init and it will only take effective on bootstrap. You need to perform a machine deployment rolling upgrade to ensure the change is propagated to each VM within the deployment.

It'll be helpful to add configurations in the cluster API spec to natively support http proxy to fix the issues we have above.

/kind feature

vincepri commented 3 years ago

This will probably need a design, let's see how we can kickstart something.

Regarding point 2, we should be able to use secrets to make sure we don't expose sensitive data https://github.com/kubernetes-sigs/cluster-api/blob/master/bootstrap/kubeadm/api/v1alpha3/kubeadmbootstrapconfig_types.go#L203-L205

/cc @CecileRobertMichon for CAPZ side /kind design

ncdc commented 3 years ago

@hanlins thanks for filing this! Could you please update your description with as many separate user stories that you can think of? For example, I think you could have separate stories for:

Thanks!

hanlins commented 3 years ago

Hi @ncdc,

Thanks for the feedback. For the user stories:

supporting non-kubeadm bootstrap providers (will require a design with proposed contract+API changes for bootstrap providers) supporting things other than cloud-init

We don't actually have a user story for other bootstrap providers. I'm bringing that up just want to make sure we choose the right layer of abstraction if we decide to involve a config (e.g. we could expose a new option specifically for proxy in ClusterNetwork or just KubeadmConfigSpec to make it a provider specific thing). For whichever approach, we need to make adaptions thus we're open to either decision and want to be aligned with upstream.

ensuring any sensitive information (usernames/passwords) are stored securely

👍 , will add

supporting in-place reconfiguration

Yeah this is a nice to have from UX stand point but we know it's not easy and come with a cost. For openshift we notice that what they did is they need to reboot VMs to pick up the new changes, this is clearly not aligned with the immutable model we adopt in cluster API. Sure we can put it down as user story but we're not setting expectation that we have to the VMs in-place 😄

hanlins commented 3 years ago

Hi @vincepri, true we can leverage SecretFileSource to hide the credentials, and this is what we're planning to work on shortly. It's good that we can hide the secrets but at the same time it's difficult for users to read the proxy settings for diagnose. Ideally if we have an option just for proxy, we could add just show the HTTP(S)_PROXY url to the users and hide its credentials. Here I'm making an assumption that the URL itself is not a secret and it's the operator's job to perform security hardening and people will have to be authenticated to get proxy access.

hanlins commented 3 years ago

@ncdc one more thing to add for the bootstrap provider assumption. Our current approach is Linux based and grouping a new option with the bootstrap provider could be easier for the moment but we might need to come back later for windows support. Exposing a new option in Cluster might enforce windows to support the new option.

vincepri commented 3 years ago

/milestone v0.4.0

hanlins commented 3 years ago

Hi @vincepri, this is a draft proposal for the proxy changes we're planning.

nicolehanjing commented 3 years ago

/ok-to-test

fejta-bot commented 3 years ago

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community. /lifecycle stale

fejta-bot commented 3 years ago

Stale issues rot after 30d of inactivity. Mark the issue as fresh with /remove-lifecycle rotten. Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community. /lifecycle rotten

fejta-bot commented 3 years ago

Rotten issues close after 30d of inactivity. Reopen the issue with /reopen. Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-contributor-experience at kubernetes/community. /close

k8s-ci-robot commented 3 years ago

@fejta-bot: Closing this issue.

In response to [this](https://github.com/kubernetes-sigs/cluster-api/issues/3751#issuecomment-820741046): >Rotten issues close after 30d of inactivity. >Reopen the issue with `/reopen`. >Mark the issue as fresh with `/remove-lifecycle rotten`. > >Send feedback to sig-contributor-experience at [kubernetes/community](https://github.com/kubernetes/community). >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.