Slow ETCD perfomance when terraforming a rke cluster on openstack.

[andersjohansson2021]

Provider version: rancher/rancher2 v1.21.0 Terraform Version: 1.0.0 I have the below terraform plan to terraform a rke cluster in Rancher ontop of openstack. Howerver i noticed that i get a pretty large latency on my etcd nodes when terraform instead of using to gui to create the cluster. I get numbers around the 3-400s in ms on the fsync value where i should be is below 10ms at least. In the cluster that ive as a reference cluster i have number way below 10ms, i usually lay around 1.5 to 2 ms which perfect. The cluster is provisioned on the same hardware and yes i have ssd for the etcd nodes 😄 .

Anyone that can lend me an eye or two see where i go wrong in my terraforming?

helm_release.cilium will be created

• resource "helm_release" "cilium" {

  • atomic = false
  • chart = "cilium"
  • cleanup_on_fail = false
  • create_namespace = false
  • dependency_update = false
  • disable_crd_hooks = false
  • disable_openapi_validation = false
  • disable_webhooks = false
  • force_update = false
  • id = (known after apply)
  • lint = false
  • manifest = (known after apply)
  • max_history = 0
  • metadata = (known after apply)
  • name = "cilium"
  • namespace = "kube-system"
  • recreate_pods = false
  • render_subchart_notes = true
  • replace = false
  • repository = "https://helm.cilium.io/"
  • reset_values = false
  • reuse_values = false
  • skip_crds = false
  • status = "deployed"
  • timeout = 300
  • verify = false
  • version = "1.11.0"
  • wait = true
  • wait_for_jobs = false }

  local_file.kube_config_openstack will be created

• resource "local_file" "kube_config_openstack" {

  • content = (sensitive)
  • directory_permission = "0777"
  • file_permission = "644"
  • filename = "kube_config_openstack"
  • id = (known after apply) }

  rancher2_cluster.cluster_openstack will be created

• resource "rancher2_cluster" "cluster_openstack" {

  • annotations = (known after apply)

  • ca_cert = (sensitive value)

  • cluster_registration_token = (known after apply)

  • default_pod_security_policy_template_id = (known after apply)

  • default_project_id = (known after apply)

  • description = "Terraform"

  • desired_agent_image = (known after apply)

  • desired_auth_image = (known after apply)

  • docker_root_dir = (known after apply)

  • driver = (known after apply)

  • enable_cluster_alerting = (known after apply)

  • enable_cluster_istio = (known after apply)

  • enable_cluster_monitoring = (known after apply)

  • enable_network_policy = (known after apply)

  • fleet_workspace_name = (known after apply)

  • id = (known after apply)

  • istio_enabled = (known after apply)

  • kube_config = (sensitive value)

  • labels = (known after apply)

  • name = (known after apply)

  • system_project_id = (known after apply)

  • windows_prefered_cluster = false

  • cluster_auth_endpoint {

    • ca_certs = (known after apply)
    • enabled = (known after apply)
    • fqdn = (known after apply) }

  • cluster_template_answers {

    • cluster_id = (known after apply)
    • project_id = (known after apply)
    • values = (known after apply) }

  • cluster_template_questions {

    • default = (known after apply)
    • required = (known after apply)
    • type = (known after apply)
    • variable = (known after apply) }

  • eks_config_v2 {

    • cloud_credential_id = (known after apply)

    • imported = (known after apply)

    • kms_key = (known after apply)

    • kubernetes_version = (known after apply)

    • logging_types = (known after apply)

    • name = (known after apply)

    • private_access = (known after apply)

    • public_access = (known after apply)

    • public_access_sources = (known after apply)

    • region = (known after apply)

    • secrets_encryption = (known after apply)

    • security_groups = (known after apply)

    • service_role = (known after apply)

    • subnets = (known after apply)

    • tags = (known after apply)

    • node_groups {

      • desired_size = (known after apply)

      • disk_size = (known after apply)

      • ec2_ssh_key = (known after apply)

      • gpu = (known after apply)

      • image_id = (known after apply)

      • instance_type = (known after apply)

      • labels = (known after apply)

      • max_size = (known after apply)

      • min_size = (known after apply)

      • name = (known after apply)

      • request_spot_instances = (known after apply)

      • resource_tags = (known after apply)

      • spot_instance_types = (known after apply)

      • subnets = (known after apply)

      • tags = (known after apply)

      • user_data = (known after apply)

      • version = (known after apply)

      • launch_template {

        • id = (known after apply)
        • name = (known after apply)
        • version = (known after apply) } } }
          • rke_config {

    • addon_job_timeout = (known after apply)

    • ignore_docker_version = false

    • kubernetes_version = "v1.21.6-rancher1-1"

    • prefix_path = (known after apply)

    • ssh_agent_auth = false

    • ssh_cert_path = (known after apply)

    • ssh_key_path = (known after apply)

    • win_prefix_path = (known after apply)

    • authentication {

      • sans = (known after apply)
      • strategy = (known after apply) }

    • authorization {

      • mode = (known after apply)
      • options = (known after apply) }

    • bastion_host {

      • address = (known after apply)
      • port = (known after apply)
      • ssh_agent_auth = (known after apply)
      • ssh_key = (sensitive value)
      • ssh_key_path = (known after apply)
      • user = (known after apply) }

    • cloud_provider {

      • custom_cloud_provider = (known after apply)

      • name = "openstack"

      • openstack_cloud_provider {

        • block_storage {

          • bs_version = (known after apply)
          • ignore_volume_az = (known after apply)
          • trust_device_path = false }

        • global {

          • auth_url = "openstack url"
          • ca_file = (known after apply)
          • domain_id = (sensitive value)
          • domain_name = "Default"
          • password = (sensitive value)
          • region = (known after apply)
          • tenant_id = (sensitive value)
          • tenant_name = (known after apply)
          • trust_id = (sensitive value)
          • username = (sensitive value) }

        • load_balancer {

          • create_monitor = true
          • floating_network_id = "d8b7f3d4-a48c-4f85-b713-208a2d53fbb3"
          • lb_method = (known after apply)
          • lb_provider = (known after apply)
          • lb_version = (known after apply)
          • manage_security_groups = (known after apply)
          • monitor_delay = "60s"
          • monitor_max_retries = 5
          • monitor_timeout = "30s"
          • subnet_id = "d8b7f3d4-a48c-4f85-b713-208a2d53fbb3"
          • use_octavia = true }

        • metadata {

          • request_timeout = 0
          • search_order = (known after apply) }

        • route {

          • router_id = (known after apply) } } }

    • dns {

      • node_selector = (known after apply)

      • options = (known after apply)

      • provider = (known after apply)

      • reverse_cidrs = (known after apply)

      • upstream_nameservers = (known after apply)

      • linear_autoscaler_params {

        • cores_per_replica = (known after apply)
        • max = (known after apply)
        • min = (known after apply)
        • nodes_per_replica = (known after apply)
        • prevent_single_point_failure = (known after apply) }

      • nodelocal {

        • ip_address = (known after apply)
        • node_selector = (known after apply) }

      • tolerations {

        • effect = (known after apply)
        • key = (known after apply)
        • operator = (known after apply)
        • seconds = (known after apply)
        • value = (known after apply) }

      • update_strategy {

        • strategy = (known after apply)

        • rolling_update {

          • max_surge = (known after apply)
          • max_unavailable = (known after apply) } } }

    • ingress {

      • default_backend = (known after apply)

      • dns_policy = (known after apply)

      • extra_args = (known after apply)

      • http_port = (known after apply)

      • https_port = (known after apply)

      • network_mode = (known after apply)

      • node_selector = (known after apply)

      • options = (known after apply)

      • provider = (known after apply)

      • tolerations {

        • effect = (known after apply)
        • key = (known after apply)
        • operator = (known after apply)
        • seconds = (known after apply)
        • value = (known after apply) }

      • update_strategy {

        • strategy = (known after apply)

        • rolling_update {

          • max_unavailable = (known after apply) } } }

    • monitoring {

      • node_selector = (known after apply)

      • options = (known after apply)

      • provider = (known after apply)

      • replicas = (known after apply)

      • tolerations {

        • effect = (known after apply)
        • key = (known after apply)
        • operator = (known after apply)
        • seconds = (known after apply)
        • value = (known after apply) }

      • update_strategy {

        • strategy = (known after apply)

        • rolling_update {

          • max_surge = (known after apply)
          • max_unavailable = (known after apply) } } }

    • network {

      • mtu = 1450
      • options = (known after apply)
      • plugin = "none" }

    • services {

      • etcd {

        • ca_cert = (known after apply)

        • cert = (sensitive value)

        • creation = (known after apply)

        • extra_args = (known after apply)

        • gid = 0

        • image = (known after apply)

        • key = (sensitive value)

        • path = (known after apply)

        • retention = (known after apply)

        • snapshot = (known after apply)

        • uid = 0

        • backup_config {

          • enabled = false
          • interval_hours = 12
          • retention = 6
          • safe_timestamp = false
          • timeout = (known after apply) } }

      • kube_api {

        • admission_configuration = (known after apply)

        • always_pull_images = (known after apply)

        • extra_args = (known after apply)

        • extra_binds = (known after apply)

        • extra_env = (known after apply)

        • image = (known after apply)

        • pod_security_policy = (known after apply)

        • service_cluster_ip_range = (known after apply)

        • service_node_port_range = (known after apply)

        • audit_log {

          • enabled = (known after apply)

          • configuration {

            • format = (known after apply)
            • max_age = (known after apply)
            • max_backup = (known after apply)
            • max_size = (known after apply)
            • path = (known after apply)
            • policy = (known after apply) } }

        • event_rate_limit {

          • configuration = (known after apply)
          • enabled = (known after apply) }

        • secrets_encryption_config {

          • custom_config = (known after apply)
          • enabled = (known after apply) } }

      • kube_controller {

        • cluster_cidr = (known after apply)
        • extra_args = (known after apply)
        • extra_binds = (known after apply)
        • extra_env = (known after apply)
        • image = (known after apply)
        • service_cluster_ip_range = (known after apply) }

      • kubelet {

        • cluster_dns_server = (known after apply)
        • cluster_domain = (known after apply)
        • extra_args = (known after apply)
        • extra_binds = (known after apply)
        • extra_env = (known after apply)
        • fail_swap_on = (known after apply)
        • generate_serving_certificate = (known after apply)
        • image = (known after apply)
        • infra_container_image = (known after apply) }

      • kubeproxy {

        • extra_args = (known after apply)
        • extra_binds = (known after apply)
        • extra_env = (known after apply)
        • image = (known after apply) }

      • scheduler {

        • extra_args = (known after apply)
        • extra_binds = (known after apply)
        • extra_env = (known after apply)
        • image = (known after apply) } }

    • upgrade_strategy {

      • drain = (known after apply)

      • max_unavailable_controlplane = (known after apply)

      • max_unavailable_worker = (known after apply)

      • drain_input {

        • delete_local_data = (known after apply)
        • force = (known after apply)
        • grace_period = (known after apply)
        • ignore_daemon_sets = (known after apply)
        • timeout = (known after apply) } } }

  • scheduled_cluster_scan {

    • enabled = (known after apply)

    • scan_config {

      • cis_scan_config {
        • debug_master = (known after apply)
        • debug_worker = (known after apply)
        • override_benchmark_version = (known after apply)
        • override_skip = (known after apply)
        • profile = (known after apply) } }

    • schedule_config {

      • cron_schedule = (known after apply)
      • retention = (known after apply) } } }

  rancher2_cluster_sync.cluster_openstack will be created

• resource "rancher2_cluster_sync" "cluster_openstack" {

  • cluster_id = (known after apply)
  • default_project_id = (known after apply)
  • id = (known after apply)
  • kube_config = (sensitive value)
  • node_pool_ids = (known after apply)
  • nodes = (known after apply)
  • state_confirm = 1
  • synced = true
  • system_project_id = (known after apply)
  • wait_alerting = false
  • wait_catalogs = false
  • wait_monitoring = false }

  rancher2_node_pool.ctrl_pool will be created

• resource "rancher2_node_pool" "ctrl_pool" {

  • annotations = (known after apply)
  • cluster_id = (known after apply)
  • control_plane = true
  • delete_not_ready_after_secs = 0
  • hostname_prefix = (known after apply)
  • id = (known after apply)
  • labels = (known after apply)
  • name = "ctrl"
  • node_template_id = (known after apply)
  • quantity = 3 }

  rancher2_node_pool.etcd_pool will be created

• resource "rancher2_node_pool" "etcd_pool" {

  • annotations = (known after apply)
  • cluster_id = (known after apply)
  • delete_not_ready_after_secs = 0
  • etcd = true
  • hostname_prefix = (known after apply)
  • id = (known after apply)
  • labels = (known after apply)
  • name = "etcd"
  • node_template_id = (known after apply)
  • quantity = 3 }

  rancher2_node_pool.worker_pool will be created

• resource "rancher2_node_pool" "worker_pool" {

  • annotations = (known after apply)
  • cluster_id = (known after apply)
  • delete_not_ready_after_secs = 0
  • hostname_prefix = (known after apply)
  • id = (known after apply)
  • labels = (known after apply)
  • name = "wrk"
  • node_template_id = (known after apply)
  • quantity = 2
  • worker = true }

  rancher2_node_template.openstack-etcdtemplate will be created

• resource "rancher2_node_template" "openstack-etcdtemplate" {

  • annotations = (known after apply)

  • description = "terraform testtemplate"

  • driver = (known after apply)

  • driver_id = (known after apply)

  • engine_install_url = "https://releases.rancher.com/install-docker/20.10.sh"

  • id = (known after apply)

  • labels = (known after apply)

  • name = (known after apply)

  • use_internal_ip_address = true

  • openstack_config {

    • active_timeout = "200"
    • auth_url = "openstack url"
    • availability_zone = ""
    • boot_from_volume = false
    • config_drive = false
    • domain_name = "Default"
    • flavor_name = "k8s.c4.m8.d40.sandybridge"
    • image_name = "Ubuntu 20.04 ETG K8S 2021-02-22"
    • insecure = false
    • ip_version = "4"
    • keypair_name = "key-pair"
    • net_name = "OTA-TEST-K8-staging-01"
    • nova_network = false
    • password = (sensitive value)
    • private_key_file = (sensitive value)
    • region = "RegionOne"
    • sec_groups = "Rancher-k8s-all"
    • ssh_port = "22"
    • ssh_user = "username"
    • tenant_name = "tenant-name"

    • user_data_file = <<-EOT

      cloud-config

      packages:
       - sdparm
      
      write_files:
       - path: /etc/udev/rules.d/99-write-cache.rules
         permissions: 0644
         owner: root
         content: |
           ACTION=="add|change", KERNEL=="sda", ATTR{queue/write_cache}="write through"
      
      runcmd:
        - 'udevadm control --reload-rules'
        - 'udevadm trigger'
        - 'sudo sdparm -l  --set WCE=0 /dev/sda' 
      
      final_message: "The system is finally up, after $UPTIME seconds"  

      EOT

    • username = (sensitive) } }

  rancher2_node_template.openstack-testtemplate will be created

• resource "rancher2_node_template" "openstack-testtemplate" {

  • annotations = (known after apply)

  • description = "terraform testtemplate"

  • driver = (known after apply)

  • driver_id = (known after apply)

  • engine_install_url = "https://releases.rancher.com/install-docker/20.10.sh"

  • id = (known after apply)

  • labels = (known after apply)

  • name = (known after apply)

  • use_internal_ip_address = true

  • openstack_config {

    • active_timeout = "200"
    • auth_url = "openstack url"
    • availability_zone = ""
    • boot_from_volume = false
    • config_drive = false
    • domain_name = "Default"
    • flavor_name = "k8s.c4.m8.d40.sandybridge"
    • image_name = "Ubuntu 20.04 ETG K8S 2021-02-22"
    • insecure = false
    • ip_version = "4"
    • keypair_name = "keypair"
    • net_name = "OTA-TEST-K8-staging-01"
    • nova_network = false
    • password = (sensitive value)
    • private_key_file = (sensitive value)
    • region = "RegionOne"
    • sec_groups = "Rancher-k8s-all"
    • ssh_port = "22"
    • ssh_user = "username"
    • tenant_name = "tenant-name"
    • username = (sensitive) } }

  random_id.instance_id will be created

• resource "random_id" "instance_id" {
  • b64_std = (known after apply)
  • b64_url = (known after apply)
  • byte_length = 3
  • dec = (known after apply)
  • hex = (known after apply)
  • id = (known after apply) }

Plan: 10 to add, 0 to change, 0 to destroy.

[andersjohansson2021]

Solved with udevadm and "ACTION=="add|change", KERNEL=="sda", ATTR{queue/write_cache}="write through""