[Bug]: Primary IP limit exceeded

[dustinmoris]

Description

I am a complete new customer to Hetzner, I signed up with the intention to migrate my GKE cluster over to Hetzner and I've followed the steps in this repo here and the Terraform deployment stopped mid way with the following errors:

│ Error: Primary IP limit exceeded (resource_limit_exceeded, 9256605a177cfa0c)
│
│   with module.kube-hetzner.module.control_planes["0-0-control-plane-fsn1"].hcloud_server.server,
│   on .terraform/modules/kube-hetzner/modules/host/main.tf line 22, in resource "hcloud_server" "server":
│   22: resource "hcloud_server" "server" {
│
╵
╷
│ Error: Primary IP limit exceeded (resource_limit_exceeded, fc67fb19e665de73)
│
│   with module.kube-hetzner.module.agents["2-0-storage"].hcloud_server.server,
│   on .terraform/modules/kube-hetzner/modules/host/main.tf line 22, in resource "hcloud_server" "server":
│   22: resource "hcloud_server" "server" {

Looking in the Hetzner cloud console I can see that I have had 8 Primary IPs created by the script so far.

This is a complete clean Hetzner account, I have not had any resources yet.

What could be the issue? How many primary IPs should I expect to have in the end? I've used mostly the default template here, so 3 control plane pools with 1 node each, 3 agent pools with 1 node each and 1 storage pool with 1 node inside it, so a total of 7 nodes I would expect plus 1 Hetzner LB.

Kube.tf file

module "kube-hetzner" {
  providers = {
    hcloud = hcloud
  }
  hcloud_token = var.hcloud_token

  source = "kube-hetzner/kube-hetzner/hcloud"

  ssh_public_key  = file("~/.ssh/id_ed25519.pub")
  ssh_private_key = file("~/.ssh/id_ed25519")

  network_region = "eu-central"

  control_plane_nodepools = [
    {
      name        = "control-plane-fsn1",
      server_type = "cax11",
      location    = "fsn1",
      labels      = [],
      taints      = [],
      count       = 1
    },
    {
      name        = "control-plane-nbg1",
      server_type = "cax11",
      location    = "nbg1",
      labels      = [],
      taints      = [],
      count       = 1
    },
    {
      name        = "control-plane-hel1",
      server_type = "cax11",
      location    = "hel1",
      labels      = [],
      taints      = [],
      count       = 1
    }
  ]

  agent_nodepools = [
    {
      name        = "agent-small",
      server_type = "cx22",
      location    = "fsn1",
      labels      = [],
      taints      = [],
      count       = 1
    },
    {
      name        = "agent-large",
      server_type = "cx32",
      location    = "nbg1",
      labels      = [],
      taints      = [],
      count       = 1
    },
    {
      name        = "storage",
      server_type = "cx32",
      location    = "fsn1",
      labels = [
        "node.kubernetes.io/server-usage=storage",
        "node.longhorn.io/create-default-disk=true"
      ],
      taints = [],
      count  = 1,
      longhorn_volume_size = 0
    },
  ]

  load_balancer_type     = "lb11"
  load_balancer_location = "fsn1"

  # Enable delete protection on compatible resources to prevent accidental deletion from the Hetzner Cloud Console.
  # This does not protect deletion from Terraform itself.
  enable_delete_protection = {
    floating_ip   = true
    load_balancer = true
    volume        = true
  }

  # To enable Hetzner Storage Box support, you can enable csi-driver-smb, default is "false".
  enable_csi_driver_smb = false

  # To enable iscid without setting enable_longhorn = true, set enable_iscsid = true. You will need this if you install your own version of longhorn outside of this module.
  # Default is false. If enable_longhorn=true, this variable is ignored and iscsid is enabled anyway.
  enable_iscsid = true

  # To use local storage on the nodes, you can enable Longhorn, default is "false".
  # See a full recap on how to configure agent nodepools for longhorn here https://github.com/kube-hetzner/terraform-hcloud-kube-hetzner/discussions/373#discussioncomment-3983159
  # Also see Longhorn best practices here https://gist.github.com/ifeulner/d311b2868f6c00e649f33a72166c2e5b
  enable_longhorn = true

  # By default, longhorn is pulled from https://charts.longhorn.io.
  # If you need a version of longhorn which assures compatibility with rancher you can set this variable to https://charts.rancher.io.
  longhorn_repository = "https://charts.longhorn.io"

  # The namespace for longhorn deployment, default is "longhorn-system".
  longhorn_namespace = "longhorn-system"

  # The file system type for Longhorn, if enabled (ext4 is the default, otherwise you can choose xfs).
  longhorn_fstype = "ext4"

  # how many replica volumes should longhorn create (default is 3).
  longhorn_replica_count = 3

  # See: https://medium.com/@mahdad.ghasemian/setting-up-a-highly-available-kubernetus-cluster-k3s-on-hetzner-cloud-with-terraform-7a409a7a8528
  longhorn_values = <<EOT
defaultSettings:
  createDefaultDiskLabeledNodes: true
  defaultDataPath: /var/longhorn
  node-down-pod-deletion-policy: delete-both-statefulset-and-deployment-pod
persistence:
  defaultFsType: ext4
  defaultClassReplicaCount: 3
  defaultClass: true
  reclaimPolicy: Retain
  EOT

  # To disable Hetzner CSI storage, you can set the following to "true", default is "false".
  disable_hetzner_csi = true

  ingress_controller = "nginx"
  ingress_target_namespace = "ingress-nginx"

  enable_cert_manager = true

  # If you want to disable the metric server set this to "false". Default is "true".
  enable_metrics_server = true

  # If you want to enable the k3s built-in local-storage controller set this to "true". Default is "false".
  enable_local_storage = true

  # IP Addresses to use for the DNS Servers, the defaults are the ones provided by Hetzner https://docs.hetzner.com/dns-console/dns/general/recursive-name-servers/.
  # The number of different DNS servers is limited to 3 by Kubernetes itself.
  # It's always a good idea to have at least 1 IPv4 and 1 IPv6 DNS server for robustness.
  dns_servers = [
    "1.1.1.1",
    "8.8.8.8",
    "2606:4700:4700::1111",
  ]

  # It is best practice to turn this off, but for backwards compatibility it is set to "true" by default.
  # See https://github.com/kube-hetzner/terraform-hcloud-kube-hetzner/issues/349
  # When "false". The kubeconfig file can instead be created by executing: "terraform output --raw kubeconfig > cluster_kubeconfig.yaml"
  # Always be careful to not commit this file!
  create_kubeconfig = true

  # Export the values.yaml files used for the deployment of traefik, longhorn, cert-manager, etc.
  # This can be helpful to use them for later deployments like with ArgoCD.
  # The default is false.
  export_values = true
}

provider "hcloud" {
  token = var.hcloud_token
}

terraform {
  required_version = ">= 1.5.0"
  required_providers {
    hcloud = {
      source  = "hetznercloud/hcloud"
      version = ">= 1.43.0"
    }
  }
}

output "kubeconfig" {
  value     = module.kube-hetzner.kubeconfig
  sensitive = true
}

variable "hcloud_token" {
  sensitive = true
  default   = ""
}

Screenshots

Platform

Mac

[dustinmoris]

I wonder is there an option to disable the primary IPs on all my nodes, including the control plane nodes? I don't see why they even need a primary IP attached to them? Presumably I just want 1 LB for my agent nodes and 1LB for the control plane and both LB having a floating IP attached so that my IP which I assign to some A record remain static even if I tear down and recreate the cluster, right? What am I missing that all these nodes get a primary IPv4 attached?

[patope]

@dustinmoris You can request limit increase here https://console.hetzner.cloud/limits

[dustinmoris]

Unfortunately I cannot, my account is "too new". I don't understand what I am supposed to do now, like how can I use Hetzner if they don't allow me to create the resources for it to become a viable replacement for my GKE setup?

This would help though:

I wonder is there an option to disable the primary IPs on all my nodes, including the control plane nodes? I don't see why they even need a primary IP attached to them? Presumably I just want 1 LB for my agent nodes and 1LB for the control plane and both LB having a floating IP attached so that my IP which I assign to some A record remain static even if I tear down and recreate the cluster, right? What am I missing that all these nodes get a primary IPv4 attached?

Is this possible using kube-hetzner right now? If not, is there a reason why my nodes need primary IPs? Looks like this is an additional addon now and charged separately:

https://docs.hetzner.com/robot/dedicated-server/ip/faq-primary-ipv4

[patope]

It's possible to create severs without public ip addresses. Afaik this terraform setup does not support that.

Hetzner servers without public ip addresses requires some additional effort because Heztner does not provide Internet Gateway as a service. You need to setup one server to do just that. Without one servers cannot access internet to download container images and make automatic updates..

[dustinmoris]

Thanks for the info. I'm closing this issue because it's not a bug, but a Hetzner limit issue right now.

If someone is interested, I found a discussion which already discusses the disabling of primary IPv4s.

Only thing I'm not clear about yet is how to configure the Hetzner LB with an existing floating IP, but that is unrelated to this issue.