Updating Cluster network configuration for single stack to dual-stack upgrades

christianang commented 2 years ago

User Story

As an operator I would like to update the cluster network configuration for an existing Cluster resource for upgrading a single stack Cluster to a dual-stack Cluster.

Detailed Description

I'd like to be able to add the cidr range for an additional IP family to spec.clusterNetwork.pods.cidrBlocks and spec.clusterNetwork.services.cidrBlocks e.g If I have a single stack IPv4 cluster with IPv4 cidr ranges I'd like to be able to add IPv6 cidr ranges to the cidrBlocks to upgrade my cluster to a dual-stack cluster.

Updating the cidrBlocks on a Cluster needs to be able to reconcile the KubeadmControlPlane with the additional CIDRs. Right now it seems like the KubeadmControlPlane disallows changes to the clusterConfiguation.networking.{serviceSubnet,podSubnet} and updating the cidrBlocks on a Cluster also does not affect the KubeadmControlPlane.

Anything else you would like to add:

There may be some complexity to orchestrating this because PodCIDRs are cached on a node so you have to drain, delete, re-init the node to change PodCIDRs (from https://github.com/kubernetes/kubernetes/issues/87150#issuecomment-601302199)

/kind feature

fabriziopandini commented 2 years ago

We discussed this in the 20th July office hours and I'm reporting my take from the discussion (feel free to discuss)

unblocking the webhook is the easiest part, but we should understand what this operation entails at infrastructure level and/or at node level
given ^^ we should evaluate if it is worth to go down this path or it will be simpler to create a new cluster and migrate workloads

k8s-triage-robot commented 1 year ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue or PR as fresh with /remove-lifecycle stale
Mark this issue or PR as rotten with /lifecycle rotten
Close this issue or PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot commented 1 year ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue or PR as fresh with /remove-lifecycle rotten
Close this issue or PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

fabriziopandini commented 1 year ago

It is not clear how to address this use case and if there are contributors willing to invest in this effort. but let's keep this around at least till the bot closes it /triage accepted

fabriziopandini commented 1 year ago

/help

k8s-ci-robot commented 1 year ago

@fabriziopandini: This request has been marked as needing help from a contributor.

Guidelines

Please ensure that the issue body includes answers to the following questions:

Why are we solving this issue?
To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points?
Does this issue have zero to low barrier of entry?
How can the assignee reach out to you for help?

For more details on the requirements of such an issue, please see here and ensure that they are met.

If this request no longer meets these requirements, the label can be removed by commenting with the /remove-help command.

In response to [this](https://github.com/kubernetes-sigs/cluster-api/issues/6951): >/help Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

fabriziopandini commented 1 year ago

(doing some cleanup on old issues without updates) /close unfortunately, no one is picking up the task. the thread will remain available for future reference

k8s-ci-robot commented 1 year ago

@fabriziopandini: Closing this issue.

In response to [this](https://github.com/kubernetes-sigs/cluster-api/issues/6951#issuecomment-1483242156): >(doing some cleanup on old issues without updates) >/close >unfortunately, no one is picking up the task. the thread will remain available for future reference Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

dntosas commented 1 year ago

I was also trying to change podCIDR on a cluster, tried to manipulate directly the control-plane config via:

apiVersion: controlplane.cluster.x-k8s.io/v1beta1
kind: KubeadmControlPlane
metadata:
  name: cluster-name
spec:
    clusterConfiguration:
      networking:
        podSubnet: '10.64.0.0/16,100.64.0.0/16'
        serviceSubnet: '10.254.0.0/16'
[...]

Got validation webhook deny:

one or more objects failed to apply, reason: admission webhook "validation.kubeadmcontrolplane.controlplane.cluster.x-k8s.io" denied the request: KubeadmControlPlane.controlplane.cluster.x-k8s.io "cluster-name" is invalid: 
[spec.kubeadmConfigSpec.clusterConfiguration.networking.podSubnet: Forbidden: cannot be modified, 
spec.kubeadmConfigSpec.clusterConfiguration.networking.serviceSubnet: Forbidden: cannot be modified]

I wonder if as first step we should allow this change?

dntosas commented 1 year ago

I'd like to be able to add the cidr range for an additional IP family to spec.clusterNetwork.pods.cidrBlocks and spec.clusterNetwork.services.cidrBlocks e.g If I have a single stack IPv4 cluster with IPv4 cidr ranges I'd like to be able to add IPv6 cidr ranges to the cidrBlocks to upgrade my cluster to a dual-stack cluster.

I can give a try on this if someone can guide where to find this logic on the codebase :D

dntosas commented 1 year ago

Note: While changing podCIDR should be trivial, changing spec.clusterNetwork.services.cidrBlocks maybe also require regenerate apiserver Certificates. Leaving it here for discussion.

killianmuldoon commented 1 year ago

/reopen

(Just to signal discussion is still ongoing on this topic)

k8s-ci-robot commented 1 year ago

@killianmuldoon: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to [this](https://github.com/kubernetes-sigs/cluster-api/issues/6951#issuecomment-1519707881): >/reopen > >(Just to signal discussion is still ongoing on this topic) Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

k8s-ci-robot commented 1 year ago

@killianmuldoon: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to [this](https://github.com/kubernetes-sigs/cluster-api/issues/6951#issuecomment-1519707881): >/reopen > >(Just to signal discussion is still ongoing on this topic) Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

k8s-ci-robot commented 1 year ago

@killianmuldoon: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to [this](https://github.com/kubernetes-sigs/cluster-api/issues/6951#issuecomment-1519707881): >/reopen > >(Just to signal discussion is still ongoing on this topic) Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

k8s-ci-robot commented 1 year ago

@killianmuldoon: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to [this](https://github.com/kubernetes-sigs/cluster-api/issues/6951#issuecomment-1519707881): >/reopen > >(Just to signal discussion is still ongoing on this topic) Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

k8s-ci-robot commented 1 year ago

@killianmuldoon: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to [this](https://github.com/kubernetes-sigs/cluster-api/issues/6951#issuecomment-1519707881): >/reopen > >(Just to signal discussion is still ongoing on this topic) Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

k8s-ci-robot commented 1 year ago

@killianmuldoon: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to [this](https://github.com/kubernetes-sigs/cluster-api/issues/6951#issuecomment-1519707881): >/reopen > >(Just to signal discussion is still ongoing on this topic) Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

k8s-ci-robot commented 1 year ago

@killianmuldoon: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to [this](https://github.com/kubernetes-sigs/cluster-api/issues/6951#issuecomment-1519707881): >/reopen > >(Just to signal discussion is still ongoing on this topic) Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

fabriziopandini commented 5 months ago

/priority backlog

kubernetes-sigs / cluster-api

Updating Cluster network configuration for single stack to dual-stack upgrades #6951

Guidelines