Closed steverhoades closed 5 years ago
Some additional information.
$ kubectl get pods -n ingress
NAME READY STATUS RESTARTS AGE
default-http-backend-58d7cfd5bc-xnfz7 0/1 Pending 0 36m
nginx-ingress-controller-68c4654b64-mmn45 1/1 Running 1 36m
I just noticed that if I hit my domain I do get a 404 from nginx and if i hit /healthz I get a 200 http status code. However it shows that the default-http-backend is not running.
One thing I just noticed is that even though I have 3 VM's provisioned with Vultr, my kubernetes cluster is only reporting one node.
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
vultr.guest Ready master 168m v1.13.2
I would wager a guess this is why the default-http-backend and other services I have tried to spin up, like the certificate manager, are staying in the pending state?
It's hard to debug this w/o much information, but it seems that Wireguard couldn't establish a connection between hosts. You can check this using wg
.
Could you please also attach your Terraform configuration?
Here is the terraform configuration for Vultr (I think this is what your asking for?). Other than that I am using the terraform configuration that is in hobby-kube/provisioning.
variable "token" {}
variable "hosts" {
default = 0
}
variable "ssh_keys" {
type = "list"
}
variable "hostname_format" {
type = "string"
}
variable "region" {
type = "string"
}
variable "image" {
type = "string"
}
variable "apt_packages" {
type = "list"
default = []
}
// Find the ID of the Silicon Valley region.
data "vultr_region" "region" {
filter {
name = "name"
values = ["${var.region}"]
}
}
// Find the ID for CoreOS Container Linux.
data "vultr_os" "container_linux" {
filter {
name = "name"
values = ["${var.image}"]
}
}
// Find the ID for a starter plan.
data "vultr_plan" "starter" {
filter {
name = "price_per_month"
values = ["10.00"]
}
filter {
name = "ram"
values = ["2048"]
}
}
// Find the ID of an existing SSH key.
data "vultr_ssh_key" "squat" {
filter {
name = "name"
values = ["${var.ssh_keys}"]
}
}
provider "vultr" {
api_key = "${var.token}"
}
resource "vultr_instance" "host" {
name = "${format(var.hostname_format, count.index + 1)}"
region_id = "${data.vultr_region.region.id}"
plan_id = "${data.vultr_plan.starter.id}"
os_id = "${data.vultr_os.container_linux.id}"
ssh_key_ids = ["${data.vultr_ssh_key.squat.id}"]
tag = "container-linux"
count = "${var.hosts}"
provisioner "remote-exec" {
inline = [
"while fuser /var/lib/dpkg/lock >/dev/null 2>&1; do sleep 1; done",
"apt-get update",
"apt-get install -yq ufw ${join(" ", var.apt_packages)}",
]
}
}
output "hostnames" {
value = ["${vultr_instance.host.*.name}"]
}
output "public_ips" {
value = ["${vultr_instance.host.*.ipv4_address}"]
}
output "private_ips" {
value = ["${vultr_instance.host.*.ipv4_address}"]
}
output "private_network_interface" {
value = "ens3"
}
Here is the output of the wg
command.
root@vultr:~# wg
interface: wg0
public key: Zdgs+Za9oFijWbHPUORrsqABd47oC3r+Twbb2vcpD0E=
private key: (hidden)
listening port: 51820
peer: /Txeg7KeRfPMH2AZRvpJ7ERQNmuh7RSRgbPyu046zBk=
endpoint: <ip>:51820
allowed ips: 10.0.1.2/32
latest handshake: 1 minute, 48 seconds ago
transfer: 3.68 MiB received, 5.48 MiB sent
peer: f+dxzWZspU/j2U5+YE+lfFwYCjfy2qgMiTJUvI18y1c=
endpoint: <ip>:51820
allowed ips: 10.0.1.3/32
latest handshake: 1 minute, 48 seconds ago
transfer: 4.39 MiB received, 7.81 MiB sent
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
vultr.guest Ready master 114s v1.13.2
I have checked each of the nodes and it appears that kubelet and etcd are running on each of them.
I noticed that the master node is continually rebooting, I'm not sure why.
➜ kubectl get events
LAST SEEN TYPE REASON KIND MESSAGE
33m Normal Starting Node Starting kubelet.
33m Normal NodeHasSufficientMemory Node Node vultr.guest status is now: NodeHasSufficientMemory
33m Normal NodeHasNoDiskPressure Node Node vultr.guest status is now: NodeHasNoDiskPressure
33m Normal NodeHasSufficientPID Node Node vultr.guest status is now: NodeHasSufficientPID
33m Normal NodeAllocatableEnforced Node Updated Node Allocatable limit across pods
32m Normal RegisteredNode Node Node vultr.guest event: Registered Node vultr.guest in Controller
32m Normal Starting Node Starting kube-proxy.
32m Normal Starting Node Starting kubelet.
32m Normal Starting Node Starting kubelet.
32m Normal NodeAllocatableEnforced Node Updated Node Allocatable limit across pods
32m Normal NodeHasSufficientMemory Node Node vultr.guest status is now: NodeHasSufficientMemory
32m Normal NodeHasNoDiskPressure Node Node vultr.guest status is now: NodeHasNoDiskPressure
32m Normal NodeHasSufficientPID Node Node vultr.guest status is now: NodeHasSufficientPID
2m41s Warning Rebooted Node Node vultr.guest has been rebooted, boot id: dd534659-0994-422c-a118-cea5628f218a
32m Normal NodeAllocatableEnforced Node Updated Node Allocatable limit across pods
32m Normal NodeHasSufficientMemory Node Node vultr.guest status is now: NodeHasSufficientMemory
32m Normal NodeHasNoDiskPressure Node Node vultr.guest status is now: NodeHasNoDiskPressure
32m Normal NodeHasSufficientPID Node Node vultr.guest status is now: NodeHasSufficientPID
2m41s Warning Rebooted Node Node vultr.guest has been rebooted, boot id: dbc43870-97d6-4461-b492-bb52d3edadbd
3m8s Warning Rebooted Node Node vultr.guest has been rebooted, boot id: f897845b-a7ca-4e79-96e5-4ab6f3958217
32m Normal Starting Node Starting kube-proxy.
32m Normal Starting Node Starting kube-proxy.
I see the following streaming non-stop in the logs as well. I'm not quite sure what this means at this point.
Jan 14 16:35:24 guest kubelet[12655]: W0114 16:35:24.129172 12655 kubelet.go:1647] Deleting mirror pod "kube-apiserver-vultr.guest_kube-system(63410cb3-181a-11e9-8776-560001d80420)" because it is outdated
Jan 14 16:35:24 guest kubelet[12655]: W0114 16:35:24.131733 12655 kubelet.go:1647] Deleting mirror pod "kube-scheduler-vultr.guest_kube-system(6304025b-181a-11e9-8776-560001d80420)" because it is outdated
Jan 14 16:35:25 guest kubelet[12655]: W0114 16:35:25.134534 12655 kubelet.go:1647] Deleting mirror pod "kube-controller-manager-vultr.guest_kube-system(63a16dc4-181a-11e9-8776-560001d80420)" because it is outdated
Jan 14 16:35:25 guest kubelet[12655]: W0114 16:35:25.136411 12655 kubelet.go:1647] Deleting mirror pod "kube-apiserver-vultr.guest_kube-system(63410cb3-181a-11e9-8776-560001d80420)" because it is outdated
Jan 14 16:35:26 guest kubelet[12655]: W0114 16:35:26.138890 12655 kubelet.go:1647] Deleting mirror pod "kube-scheduler-vultr.guest_kube-system(643dba69-181a-11e9-8776-560001d80420)" because it is outdated
Jan 14 16:35:27 guest kubelet[12655]: W0114 16:35:27.143278 12655 kubelet.go:1647] Deleting mirror pod "kube-apiserver-vultr.guest_kube-system(64d76e21-181a-11e9-8776-560001d80420)" because it is outdated
Jan 14 16:35:27 guest kubelet[12655]: W0114 16:35:27.145020 12655 kubelet.go:1647] Deleting mirror pod "kube-controller-manager-vultr.guest_kube-system(64d4dbda-181a-11e9-8776-560001d80420)" because it is outdated
I think i might be getting further down the line here. It appears that with Vultr that all nodes are getting the same hostname vultr.guest. My guess is that this is what is messing with kubernetes. I am going to revisit the provisioning step to see how I can change this. Will report back.
So I was correct in my assumption. It appears the default hostnames created on Vultr are vultr.guest. Since each of the nodes had the same hostname this was creating my problem - I didn't expect this.
Here is the fixed provision file for Vultr. If you are interested in using it to follow along with the guide than you will need to add the following plugin https://github.com/squat/terraform-provider-vultr.
variable "token" {}
variable "hosts" {
default = 0
}
variable "ssh_keys" {
type = "list"
}
variable "hostname_format" {
type = "string"
}
variable "region" {
type = "string"
}
variable "image" {
type = "string"
}
variable "apt_packages" {
type = "list"
default = []
}
// Find the ID of the Silicon Valley region.
data "vultr_region" "region" {
filter {
name = "name"
values = ["${var.region}"]
}
}
// Find the ID for CoreOS Container Linux.
data "vultr_os" "container_linux" {
filter {
name = "name"
values = ["${var.image}"]
}
}
// Find the ID for a starter plan.
data "vultr_plan" "starter" {
filter {
name = "price_per_month"
values = ["10.00"]
}
filter {
name = "ram"
values = ["2048"]
}
}
// Find the ID of an existing SSH key.
data "vultr_ssh_key" "squat" {
filter {
name = "name"
values = ["${var.ssh_keys}"]
}
}
provider "vultr" {
api_key = "${var.token}"
}
resource "vultr_instance" "host" {
name = "${format(var.hostname_format, count.index + 1)}"
region_id = "${data.vultr_region.region.id}"
plan_id = "${data.vultr_plan.starter.id}"
os_id = "${data.vultr_os.container_linux.id}"
ssh_key_ids = ["${data.vultr_ssh_key.squat.id}"]
tag = "container-linux"
hostname = "${format(var.hostname_format, count.index + 1)}"
count = "${var.hosts}"
provisioner "remote-exec" {
inline = [
"while fuser /var/lib/dpkg/lock >/dev/null 2>&1; do sleep 1; done",
"apt-get update",
"apt-get install -yq ufw ${join(" ", var.apt_packages)}",
]
}
}
output "hostnames" {
value = ["${vultr_instance.host.*.name}"]
}
output "public_ips" {
value = ["${vultr_instance.host.*.ipv4_address}"]
}
output "private_ips" {
value = ["${vultr_instance.host.*.ipv4_address}"]
}
output "private_network_interface" {
value = "ens3"
}
Disclaimer: New to kubernetes and terraform.
I am working through your guide and have successfully deployed kubernetes to Vultr by way of the provision repository.
I am currently running into an issue however with the ingress/nginx-ingress-controller. According to the event log it is failing due to the readiness checks (http://\<ip>:10254/healthz).
Any assistance here would be greatly appreciated.