search

coreos / tectonic-installer

Install a Kubernetes cluster the CoreOS Tectonic Way: HA, self-hosted, RBAC, etcd Operator, and more
Apache License 2.0
601 stars 266 forks source link

Grafana was not created #2677

Open r-moiseev opened 6 years ago

r-moiseev commented 6 years ago

Is this a BUG REPORT or FEATURE REQUEST?

BUG REPORT

Versions

What happened?

When new cluster is started /grafana path returns 503 Service Unavailable. There is Ingress named tectonic-monitoring-ingress that has /grafaba path points to tectonic-monitoring-auth-grafana service. But this service is missing. Also there is no grafana deployment

What you expected to happen?

Grafana to be available

How to reproduce it (as minimally and precisely as possible)?

Provision new cluster with terraform (not UI).

There is my terraform.tfvars:

tectonic_license_path = "./license.txt"
tectonic_pull_secret_path = "./pull_secret.json"

tectonic_admin_email = "***"
tectonic_admin_password = "***"

tectonic_base_domain = "unused"

tectonic_cluster_cidr = "10.2.0.0/16"

tectonic_cluster_name = "dev-cluster"

tectonic_container_linux_version = "latest"

tectonic_networking = "canal"

tectonic_kube_apiserver_service_ip = "10.3.0.1"

tectonic_kube_dns_service_ip = "10.3.0.10"

tectonic_kube_etcd_service_ip = "10.3.0.15"

tectonic_metal_controller_domain = "master.cluster.***.com"

tectonic_metal_controller_domains = [
  "master.cluster.***.com"
]

tectonic_metal_controller_macs = [
  "56:00:01:53:11:c1"
]

tectonic_metal_controller_names = [
  "master"
]
tectonic_metal_ingress_domain = "tectonic.cluster.***.com"

tectonic_metal_matchbox_ca = "-----BEGIN CERTIFICATE-----***-----END CERTIFICATE-----"

tectonic_metal_matchbox_client_cert = "-----BEGIN CERTIFICATE-----***-----END CERTIFICATE-----\n"

tectonic_metal_matchbox_client_key = "-----BEGIN RSA PRIVATE KEY-----***----END RSA PRIVATE KEY-----\n"

tectonic_metal_matchbox_http_url = "http://matchbox.dev.***.com:8080"

tectonic_metal_matchbox_rpc_endpoint = "matchbox.dev.***.com:8081"

tectonic_worker_count = "2"

tectonic_metal_worker_domains = [
  "node1.cluster.***.com",
  "node2.cluster.***.com"
]

tectonic_metal_worker_macs = [
  "56:00:01:53:11:cd",
  "56:00:01:54:53:50"
]

tectonic_metal_worker_names = [
  "node1",
  "node2"
]

tectonic_service_cidr = "10.3.0.0/16"

tectonic_ssh_authorized_key =  "ssh-rsa ***"

Anything else we need to know?

Somehow I am able to see embedded panels on tectonic dashboard

ant31 commented 6 years ago

Hi,

The tectonic-monitoring-auth-grafana svc should be created or recreated automatically if deleted by the tectonic-prometheus-operator. Looks like it failed to create it in your cluster, could provide the log of the tectonic-prometheus-operator pod? 

kubectl logs  -n tectonic-system  prometheus-operator-$PODID
r-moiseev commented 6 years ago

@ant31 There is lot of text but I found one message that was repeated many times

E0110 19:04:04.987832       5 operator.go:577] Sync "tectonic-system/k8s" failed: updating statefulset failed: StatefulSet.apps "prometheus-k8s" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden.
r-moiseev commented 6 years ago

Updating to 1.8.4-tectonic.3 fixed the problem

r-moiseev commented 6 years ago

nope it was working during update, but after it was broken again

robszumski commented 6 years ago

Can you share some logs for the pods after the update?

brianredbeard commented 6 years ago

FYI @robszumski this appears to still be a problem - https://github.com/coreos/tectonic-installer/blob/track-1/config.tf#L114-L115

NOTE: The URL displayed in the hyperlink is for the purpose of referencing the branch while the link points at the current (as of 2017-01-19) commit.