Closed Dani4kor closed 2 months ago
same here
Interesting, I'll have a look later today.
@Dani4kor Can you terraform destroy, terraform init -upgrade and try again please?
terraform destory
Destroy complete! Resources: X destroyed.
terraform init --upgrade
Initializing the backend...
Upgrading modules...
Downloading git::https://github.com/kube-hetzner/terraform-hcloud-kube-hetzner.git for kube-hetzner...
- kube-hetzner in .terraform/modules/kube-hetzner
- kube-hetzner.agents in .terraform/modules/kube-hetzner/modules/host
- kube-hetzner.control_planes in .terraform/modules/kube-hetzner/modules/host
Initializing provider plugins...
- Finding hashicorp/local versions matching ">= 2.4.0"...
- Finding tenstad/remote versions matching ">= 0.1.2"...
- Finding latest version of hashicorp/null...
- Finding latest version of hashicorp/random...
- Finding latest version of hashicorp/cloudinit...
- Finding hetznercloud/hcloud versions matching ">= 1.43.0"...
- Finding integrations/github versions matching ">= 5.44.0"...
- Using previously-installed hashicorp/local v2.5.1
- Using previously-installed tenstad/remote v0.1.3
- Using previously-installed hashicorp/null v3.2.2
- Using previously-installed hashicorp/random v3.6.2
- Using previously-installed hashicorp/cloudinit v2.3.4
- Using previously-installed hetznercloud/hcloud v1.47.0
- Installing integrations/github v6.2.2...
- Installed integrations/github v6.2.2 (signed by a HashiCorp partner, key ID 38027F80D7FD5FB2)
....
module.kube-hetzner.module.control_planes["0-1-control-plane-fsn1"].null_resource.registries (remote-exec): Backing up /etc/rancher/k3s/registries.yaml to /tmp/registries_2024-06-21_17-35-53.yaml
module.kube-hetzner.module.control_planes["0-1-control-plane-fsn1"].null_resource.registries (remote-exec): cp: cannot stat '/etc/rancher/k3s/registries.yaml': No such file or directory
module.kube-hetzner.module.control_planes["0-1-control-plane-fsn1"].null_resource.registries (remote-exec): Updated registries.yaml detected, restart of k3s service required
module.kube-hetzner.module.control_planes["0-1-control-plane-fsn1"].null_resource.registries (remote-exec): cp: cannot create regular file '/etc/rancher/k3s/registries.yaml': No such file or directory
module.kube-hetzner.module.control_planes["0-1-control-plane-fsn1"].null_resource.registries (remote-exec): No active k3s or k3s-agent service found
module.kube-hetzner.module.control_planes["0-1-control-plane-fsn1"].null_resource.registries (remote-exec): k3s service or k3s-agent service restarted successfully
module.kube-hetzner.module.control_planes["0-0-control-plane-fsn1"].null_resource.registries (remote-exec): Backing up /etc/rancher/k3s/registries.yaml to /tmp/registries_2024-06-21_17-35-53.yaml
module.kube-hetzner.module.control_planes["0-0-control-plane-fsn1"].null_resource.registries (remote-exec): cp: cannot stat '/etc/rancher/k3s/registries.yaml': No such file or directory
module.kube-hetzner.module.control_planes["0-0-control-plane-fsn1"].null_resource.registries (remote-exec): Updated registries.yaml detected, restart of k3s service required
module.kube-hetzner.module.control_planes["0-0-control-plane-fsn1"].null_resource.registries (remote-exec): cp: cannot create regular file '/etc/rancher/k3s/registries.yaml': No such file or directory
module.kube-hetzner.module.control_planes["0-1-control-plane-fsn1"].null_resource.registries: Creation complete after 3s [id=3551722489441731771]
module.kube-hetzner.module.control_planes["0-0-control-plane-fsn1"].null_resource.registries (remote-exec): No active k3s or k3s-agent service found
module.kube-hetzner.module.control_planes["0-0-control-plane-fsn1"].null_resource.registries (remote-exec): k3s service or k3s-agent service restarted successfully
module.kube-hetzner.module.control_planes["0-2-control-plane-fsn1"].null_resource.registries (remote-exec): Backing up /etc/rancher/k3s/registries.yaml to /tmp/registries_2024-06-21_17-35-53.yaml
module.kube-hetzner.module.control_planes["0-2-control-plane-fsn1"].null_resource.registries (remote-exec): cp: cannot stat '/etc/rancher/k3s/registries.yaml': No such file or directory
module.kube-hetzner.module.control_planes["0-2-control-plane-fsn1"].null_resource.registries (remote-exec): Updated registries.yaml detected, restart of k3s service required
module.kube-hetzner.module.control_planes["0-2-control-plane-fsn1"].null_resource.registries (remote-exec): cp: cannot create regular file '/etc/rancher/k3s/registries.yaml': No such file or directory
module.kube-hetzner.module.control_planes["0-2-control-plane-fsn1"].null_resource.registries (remote-exec): No active k3s or k3s-agent service found
module.kube-hetzner.module.control_planes["0-2-control-plane-fsn1"].null_resource.registries (remote-exec): k3s service or k3s-agent service restarted successfully
module.kube-hetzner.module.control_planes["0-2-control-plane-fsn1"].null_resource.registries: Creation complete after 3s [id=6349535723117724841]
module.kube-hetzner.module.control_planes["0-1-control-plane-fsn1"].null_resource.zram: Provisioning with 'remote-exec'...
module.kube-hetzner.module.control_planes["0-1-control-plane-fsn1"].null_resource.zram (remote-exec): Connecting to remote host via SSH...
module.kube-hetzner.module.control_planes["0-1-control-plane-fsn1"].null_resource.zram (remote-exec): Host: x.x.x.x
module.kube-hetzner.module.control_planes["0-1-control-plane-fsn1"].null_resource.zram (remote-exec): User: root
module.kube-hetzner.module.control_planes["0-1-control-plane-fsn1"].null_resource.zram (remote-exec): Password: false
module.kube-hetzner.module.control_planes["0-1-control-plane-fsn1"].null_resource.zram (remote-exec): Private key: true
module.kube-hetzner.module.control_planes["0-1-control-plane-fsn1"].null_resource.zram (remote-exec): Certificate: false
module.kube-hetzner.module.control_planes["0-1-control-plane-fsn1"].null_resource.zram (remote-exec): SSH Agent: true
module.kube-hetzner.module.control_planes["0-1-control-plane-fsn1"].null_resource.zram (remote-exec): Checking Host Key: false
module.kube-hetzner.module.control_planes["0-1-control-plane-fsn1"].null_resource.zram (remote-exec): Target Platform: unix
module.kube-hetzner.module.control_planes["0-0-control-plane-fsn1"].null_resource.zram: Provisioning with 'remote-exec'...
module.kube-hetzner.module.control_planes["0-0-control-plane-fsn1"].null_resource.zram (remote-exec): Connecting to remote host via SSH...
module.kube-hetzner.module.control_planes["0-0-control-plane-fsn1"].null_resource.zram (remote-exec): Host: x.x.x.x
module.kube-hetzner.module.control_planes["0-0-control-plane-fsn1"].null_resource.zram (remote-exec): User: root
module.kube-hetzner.module.control_planes["0-0-control-plane-fsn1"].null_resource.zram (remote-exec): Password: false
module.kube-hetzner.module.control_planes["0-0-control-plane-fsn1"].null_resource.zram (remote-exec): Private key: true
module.kube-hetzner.module.control_planes["0-0-control-plane-fsn1"].null_resource.zram (remote-exec): Certificate: false
module.kube-hetzner.module.control_planes["0-0-control-plane-fsn1"].null_resource.zram (remote-exec): SSH Agent: true
module.kube-hetzner.module.control_planes["0-0-control-plane-fsn1"].null_resource.zram (remote-exec): Checking Host Key: false
module.kube-hetzner.module.control_planes["0-0-control-plane-fsn1"].null_resource.zram (remote-exec): Target Platform: unix
module.kube-hetzner.module.control_planes["0-0-control-plane-fsn1"].null_resource.registries: Creation complete after 3s [id=1060475260337499317]
module.kube-hetzner.module.control_planes["0-2-control-plane-fsn1"].null_resource.zram: Provisioning with 'remote-exec'...
module.kube-hetzner.module.control_planes["0-2-control-plane-fsn1"].null_resource.zram (remote-exec): Connecting to remote host via SSH...
module.kube-hetzner.module.control_planes["0-2-control-plane-fsn1"].null_resource.zram (remote-exec): Host: x.x.x.x
module.kube-hetzner.module.control_planes["0-2-control-plane-fsn1"].null_resource.zram (remote-exec): User: root
module.kube-hetzner.module.control_planes["0-2-control-plane-fsn1"].null_resource.zram (remote-exec): Password: false
module.kube-hetzner.module.control_planes["0-2-control-plane-fsn1"].null_resource.zram (remote-exec): Private key: true
module.kube-hetzner.module.control_planes["0-2-control-plane-fsn1"].null_resource.zram (remote-exec): Certificate: false
module.kube-hetzner.module.control_planes["0-2-control-plane-fsn1"].null_resource.zram (remote-exec): SSH Agent: true
module.kube-hetzner.module.control_planes["0-2-control-plane-fsn1"].null_resource.zram (remote-exec): Checking Host Key: false
module.kube-hetzner.module.control_planes["0-2-control-plane-fsn1"].null_resource.zram (remote-exec): Target Platform: unix
module.kube-hetzner.module.control_planes["0-1-control-plane-fsn1"].null_resource.zram (remote-exec): Connected!
module.kube-hetzner.module.control_planes["0-0-control-plane-fsn1"].null_resource.zram (remote-exec): Connected!
module.kube-hetzner.module.control_planes["0-0-control-plane-fsn1"].hcloud_server_network.server: Creation complete after 4s [id=49163361-4361117]
module.kube-hetzner.module.control_planes["0-2-control-plane-fsn1"].null_resource.zram (remote-exec): Connected!
module.kube-hetzner.module.control_planes["0-2-control-plane-fsn1"].hcloud_server_network.server: Creation complete after 5s [id=49163360-4361117]
module.kube-hetzner.module.control_planes["0-1-control-plane-fsn1"].null_resource.zram: Creation complete after 5s [id=6697467964240100583]
module.kube-hetzner.module.control_planes["0-0-control-plane-fsn1"].null_resource.zram: Creation complete after 5s [id=6815029547267037962]
module.kube-hetzner.module.control_planes["0-2-control-plane-fsn1"].null_resource.zram: Creation complete after 6s [id=689469212275291105]
module.kube-hetzner.null_resource.first_control_plane: Creating...
module.kube-hetzner.null_resource.first_control_plane: Provisioning with 'file'...
module.kube-hetzner.data.cloudinit_config.autoscaler_config[0]: Reading...
module.kube-hetzner.data.cloudinit_config.autoscaler_config[0]: Read complete after 0s [id=3577989300]
module.kube-hetzner.null_resource.first_control_plane: Provisioning with 'remote-exec'...
module.kube-hetzner.null_resource.first_control_plane (remote-exec): Connecting to remote host via SSH...
module.kube-hetzner.null_resource.first_control_plane (remote-exec): Host: x.x.x.x
module.kube-hetzner.null_resource.first_control_plane (remote-exec): User: root
module.kube-hetzner.null_resource.first_control_plane (remote-exec): Password: false
module.kube-hetzner.null_resource.first_control_plane (remote-exec): Private key: true
module.kube-hetzner.null_resource.first_control_plane (remote-exec): Certificate: false
module.kube-hetzner.null_resource.first_control_plane (remote-exec): SSH Agent: true
module.kube-hetzner.null_resource.first_control_plane (remote-exec): Checking Host Key: false
module.kube-hetzner.null_resource.first_control_plane (remote-exec): Target Platform: unix
module.kube-hetzner.null_resource.first_control_plane (remote-exec): Connected!
module.kube-hetzner.null_resource.first_control_plane (remote-exec): + /etc/cloud/rename_interface.sh
module.kube-hetzner.null_resource.first_control_plane (remote-exec): /tmp/terraform_752420511.sh: line 3: /etc/cloud/rename_interface.sh: No such file or directory
╷
│ Error: remote-exec provisioner error
│
│ with module.kube-hetzner.null_resource.first_control_plane,
│ on .terraform/modules/kube-hetzner/init.tf line 70, in resource "null_resource" "first_control_plane":
│ 70: provisioner "remote-exec" {
│
│ error executing "/tmp/terraform_752420511.sh": Process exited with status 127
kube.tf stay same
terraform --version
Terraform v1.8.5
on darwin_arm64
+ provider registry.terraform.io/hashicorp/cloudinit v2.3.4
+ provider registry.terraform.io/hashicorp/local v2.5.1
+ provider registry.terraform.io/hashicorp/null v3.2.2
+ provider registry.terraform.io/hashicorp/random v3.6.2
+ provider registry.terraform.io/hetznercloud/hcloud v1.47.0
+ provider registry.terraform.io/integrations/github v6.2.2
+ provider registry.terraform.io/tenstad/remote v0.1.3
@Dani4kor Try placing the autoscaler in another placement group, who knows, and remove the version for autoscaler, use the latest instead (default now) of v1.30.1
.
Also remove or increase ssh_max_auth_tries
by a lot.
For the nodepool labels, carefully with the format, you might want to have an equal sign in there, see kube.tf.example
. Not sure if it's really needed, but at least it's good practice, try it.
Last but not least, if all of the above do not work, proceed by first trying without all of those, just comment them out:
cluster_autoscaler_server_creation_timeout = 15
enable_delete_protection = {
floating_ip = true
load_balancer = true
volume = true
}
enable_csi_driver_smb = true
ingress_controller = "nginx"
enable_local_storage = true
enable_cert_manager = true
export_values = true
create_kubeconfig = true
enable_metrics_server = true
restrict_outbound_traffic = false
And then proceed by elimination, intuition, also looking at the k3s logs (see debug section in the readme). Also executing the fail script helps, it shows you the error.
But, /etc/cloud/rename_interface.sh: No such file or directory
indicates a probable SSH failure, so increasing the retry by a factor of 10x or 100x should help, try that first.
Following your suggetions i start removing settings one-by-one, and its not helps:(
But you pointed on "ssh" problem and i start dive
TL;DR
Problem was in ssh key, i change the key- and all goes good
ssh_public_key = file("~/.ssh/id_rsa_self.pub")
ssh_private_key = file("~/.ssh/id_rsa_self")
#to
ssh_public_key = file("~/.ssh/id_rsa.pub")
ssh_private_key = file("~/.ssh/id_rsa")
As im able to connect to servers with key deployed by module(checked path/logs whatever) - i dont pointing my attention for the key.
Trying to find difference in keys i found thats "wrong"
➜ k3s-hetzner git:(main) ✗ ssh-keygen -lf ~/.ssh/id_rsa_self.pub
3072 SHA256:xxxx no comment (RSA)
➜ k3s-hetzner git:(main) ✗ ssh-keygen -lf ~/.ssh/id_rsa.pub
3072 SHA256:xxxx name@pc (RSA)
type, lenght etc was the same except comment
In "bad key" comment was on new line ("\n")
#bad key
ssh-rsa ….
here@comment
#good key
ssh-rsa ... here@comment
Thats means:
openssh-client is able to connect even if authorized_keys contains bad format:
# openssh-client will able to connect with this authorized_keys on server
ssh-rsa …. [comment] <- checked
ssh-rsa …. <- checked
# below bad data <- ignored
somedata <- ignored
hello <- ignored
asdfsadfsadfsad <- ignored
cat /etc/passwd <- ignored
ssh-rsa ..bad_format.. [comment] <- ignored
ssh-rsa …. [comment] <- checked
Possbile iam wrong somewhere in my research, feel free to fix/point where iam wrong Anyway, after changing it to right format(remove comment) on id_rsa_self - all comes good
Thanks for help and your time @mysticaltech
Description
Following README Getting started instructions , facing this problem
no /etc/cloud/rename_interface.sh is located on server
builded image with packer as described in readme
Kube.tf file
Screenshots
No response
Platform
mac m1