search

rancher / terraform-provider-rancher2

Terraform Rancher2 provider
https://www.terraform.io/docs/providers/rancher2/
Mozilla Public License 2.0
263 stars 228 forks source link

No provision to pass labels to machine pools and nodes #949

Closed PrakashFromBunnings closed 1 year ago

PrakashFromBunnings commented 2 years ago

Hello ,

We have a requirement to have multiple machine pools and deploy different kind of workload to each pools.

I was thinking to apply some labels to these pools , and pass my deployment to have nodeSelector . So that my deployment goes to selected nodeSelector label only .

I cant see any options in terraform modules rancher2_cluser_v2 https://registry.terraform.io/providers/rancher/rancher2/latest/docs/resources/cluster_v2#machine_pools

Does any one have bright idea how can I achieve that?

Thanks

PrakashFromBunnings commented 2 years ago

I can see adding labels option is available in GUI ( in Advance option ). Not sure if I am missing anything in terraform module.

azafairon commented 2 years ago

Hi,

We have tried also to set the labels on machine pools which it seems that is supported: https://github.com/rancher/terraform-provider-rancher2/blob/master/rancher2/schema_cluster_v2_rke_config_machine_pool.go#L143

The above will populate the metadata on MachineDeployment but will not create the nodes with labels as the specs under templates are not correct.

Are we missing something ?

Thanks

m4rCsi commented 2 years ago

Hi,

we ran into this as well and were able to figure it out.

Flow

The flow (backwards) to get labels passed to the eventual nodes:

Where is the Problem

Following through all of this we see:

But we can set some labels?

But at the moment, as pointed out by @azafairon, labels can be set in the Machine Pool through the common labels/annotations mechanism.

However, these labels get assigned to RKEMachinePool.MachineDeploymentLabels , which as the name suggests goes to the labels of the MachineDeployment (i.e. labels of the Resource, not the spec). (Link)

How to fix/make it possible What I don't know if the above behaviour is intended.

Are the (undocumented) labels of the machine_pool resource meant to:

Depending on the above the fix would be to either:

Both are a pretty minimal change. I tried out (solution B), and it worked as intended. (see linked PR)

Kind regards, Marc

luis-garza commented 2 years ago

We are getting the same behavior :cry:

Not sure what is for the labels translated as spec.rkeConfig.machinePools.machineDeploymentLabels in the cluster CRD, but definitely we need to be able to define spec.rkeConfig.machinePools.labels though the provider...

cloudnautique commented 2 years ago

Hi, this is a big gap in the ability to provision clusters in the new provisioning framework. There is no way to allow dedicated Ingress nodes for example. Any chance we will see some traction on #951 ??

Heiko-san commented 2 years ago

As far as I can tell from https://registry.terraform.io/providers/rancher/rancher2/latest/docs/resources/machine_config_v2 (Note: labels and node_taints will be applied to nodes deployed using the Machine Config V2) you are supposed to set node-labels via the machine_config_v2 . But there seems to be a bug preventing you from doing so, see issue https://github.com/rancher/terraform-provider-rancher2/issues/976 .

HarrisonWAffel commented 2 years ago

951 was merged yesterday, so this issue can be tested with the latest commit of the provider.

Test steps:

  1. Create a new .tf file for an RKE2 cluster on any cloud provider
  2. When defining the machine-pool set a value for both the labels and machine_labels fields, the exact value is not important
  3. Bring the cluster up, and import it into Rancher a. This is just done to make inspecting the yaml easier, this verification can also be done with kubectl
  4. Using the Rancher UI navigate to the provisioned machine and view its YAML.
  5. Check the metadata.labels field and ensure the machine_labels value is attached to the machine yaml
  6. On the left hand side of the UI, under Advanced find the MachineDeployment for your newly created cluster.
  7. Ensure that the MachineDeployment metadata.Labels field has the value specified in the labels field within the .tf file.
Josh-Diamond commented 1 year ago

Ticket #949 - Test Results - ✅

With Docker on a single-node instance, using terraform rancher2 provider v1.25.0:

Verified on rancher v2.7.0:

  1. Fresh install of rancher v2.7.0
  2. Using main.tf below, provision requested RKE2 infrastructure w/ both labels and machine_labels defined in machine_pools resource block 
    
terraform {
required_providers {
rancher2 = {
  source  = "rancher/rancher2"
  version = "1.25.0"
}
}
}
provider "rancher2" {
api_url   = var.rancher_api_url
token_key = var.rancher_admin_bearer_token
insecure  = true
}
data "rancher2_cloud_credential" "rancher2_cloud_credential" {
name = var.cloud_credential_name
}
resource "rancher2_machine_config_v2" "rancher2_machine_config_v2" {
generate_name = var.machine_config_name
amazonec2_config {
ami            = var.aws_ami
region         = var.aws_region
security_group = [var.aws_security_group_name]
subnet_id      = var.aws_subnet_id
vpc_id         = var.aws_vpc_id
zone           = var.aws_zone_letter
}
}
resource "rancher2_cluster_v2" "rancher2_cluster_v2" {
name                                     = var.cluster_name
kubernetes_version                       = "v1.24.7+rke2r1"
enable_network_policy                    = var.enable_network_policy
default_cluster_role_for_project_members = var.default_cluster_role_for_project_members
rke_config {
machine_pools {
  name                         = "pool1"
  labels                       = { "jkeslar1" = "true", "remy1" = "false" }
  machine_labels               = { "jkeslarML1" = "true", "remyML1" = "false" }
  cloud_credential_secret_name = data.rancher2_cloud_credential.rancher2_cloud_credential.id
  control_plane_role           = false
  etcd_role                    = true
  worker_role                  = false
  quantity                     = 1
  machine_config {
    kind = rancher2_machine_config_v2.rancher2_machine_config_v2.kind
    name = rancher2_machine_config_v2.rancher2_machine_config_v2.name
  }
}
machine_pools {
  name                         = "pool2"
  labels                       = { "jkeslar2" = "true", "remy2" = "false" }
  machine_labels               = { "jkeslarML2" = "true", "remyML2" = "false" }
  cloud_credential_secret_name = data.rancher2_cloud_credential.rancher2_cloud_credential.id
  control_plane_role           = true
  etcd_role                    = false
  worker_role                  = false
  quantity                     = 1
  machine_config {
    kind = rancher2_machine_config_v2.rancher2_machine_config_v2.kind
    name = rancher2_machine_config_v2.rancher2_machine_config_v2.name
  }
}
machine_pools {
  name                         = "pool3"
  labels                       = { "jkeslar3" = "true", "remy3" = "false" }
  machine_labels               = { "jkeslarML3" = "true", "remyML3" = "false" }
  cloud_credential_secret_name = data.rancher2_cloud_credential.rancher2_cloud_credential.id
  control_plane_role           = false
  etcd_role                    = false
  worker_role                  = true
  quantity                     = 1
  machine_config {
    kind = rancher2_machine_config_v2.rancher2_machine_config_v2.kind
    name = rancher2_machine_config_v2.rancher2_machine_config_v2.name
  }
}
}
}

Rancher specific variable section.

variable rancher_api_url {} variable rancher_admin_bearer_token {} variable cloud_credential_name {}

AWS specific variables.

variable aws_access_key {} variable aws_secret_key {} variable aws_ami {} variable aws_region {} variable aws_security_group_name {} variable aws_subnet_id {} variable aws_vpc_id {} variable aws_zone_letter {}

RKE2/k3s specific variables.

variable machine_config_name {} variable cluster_name {} variable enable_network_policy {} variable default_cluster_role_for_project_members {}


3. Verified - TF plan accurately reflects requested `labels` and `machine_labels` on `machine_pools` resource block
4. Verified - Cluster successfully provisions
5. Verified - Accurate `labels` and `machine_labels` observed in YAML via RancherUI

Screenshots:

`Step 3 - TF Plan`
<img width="320" alt="Screenshot 2022-11-28 at 11 16 16 AM" src="https://user-images.githubusercontent.com/46494969/204354108-87afc749-3b1e-4313-af5e-f52a249435d6.png">

<img width="316" alt="Screenshot 2022-11-28 at 11 15 58 AM" src="https://user-images.githubusercontent.com/46494969/204354121-12847570-10e4-494f-bea3-7dbecca82ab3.png">

<img width="322" alt="Screenshot 2022-11-28 at 11 15 42 AM" src="https://user-images.githubusercontent.com/46494969/204354134-cc7082dc-3958-4265-a5e9-8a7783a2965d.png">

`Step 4`
<img width="1370" alt="Screenshot 2022-11-28 at 11 26 21 AM" src="https://user-images.githubusercontent.com/46494969/204354333-be3a0e69-a151-4e27-b579-d3e3815eefca.png">

`Step 5 - Machine YAML`
<img width="516" alt="Screenshot 2022-11-28 at 11 28 19 AM" src="https://user-images.githubusercontent.com/46494969/204354483-03cad75d-d44e-4def-aade-8673373ec62c.png">

<img width="517" alt="Screenshot 2022-11-28 at 11 28 50 AM" src="https://user-images.githubusercontent.com/46494969/204354501-bfcdc030-2a09-49a0-983a-984d29272ad5.png">

<img width="521" alt="Screenshot 2022-11-28 at 11 29 16 AM" src="https://user-images.githubusercontent.com/46494969/204354515-cb4a62f7-28ad-41a8-afb4-a83898108814.png">

`Step 5 - MachineDeployment YAML`
<img width="283" alt="Screenshot 2022-11-28 at 11 31 51 AM" src="https://user-images.githubusercontent.com/46494969/204354602-7cde8026-2102-40d8-a196-36e634c2df0d.png">

<img width="291" alt="Screenshot 2022-11-28 at 11 32 12 AM" src="https://user-images.githubusercontent.com/46494969/204354613-7c3a5191-ffc4-42a3-8d6e-14d27eb93477.png">

<img width="283" alt="Screenshot 2022-11-28 at 11 32 34 AM" src="https://user-images.githubusercontent.com/46494969/204354622-b2e3f961-1355-42e1-8954-e5aa279b9ba3.png">