search

timescale / tobs

tobs - The Observability Stack for Kubernetes. Easy install of a full observability stack into a k8s cluster with Helm charts.
Apache License 2.0
563 stars 60 forks source link

Prometheus PVC name too long #563

Closed furynick closed 2 years ago

furynick commented 2 years ago

What happened? Installing with tobs helm chart, Prometheus DB volume PVC Name is more than 63 characters which is not supported by CSI.

Did you expect to see something different? PVC Name should be less than 63 characters

How to reproduce it (as minimally and precisely as possible):

helm install \
  -n monitoring \
  --create-namespace \
  --set opentelemetry-operator.enabled=false \
  --set promscale.connection.uri=postgres://prometheus:xxxxxx@externaldatabase:5432/prometheus?sslmode=require \
  --set timescaledb-single.enabled=false \
  tobs timescale/tobs

Environment

# kubectl get pvc -n monitoring
NAME                                                                                         STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS     AGE
prometheus-tobs-kube-prometheus-prometheus-db-prometheus-tobs-kube-prometheus-prometheus-0   Bound    pvc-878e2454-58da-4dca-97ff-e97b9fc05cd2   8Gi        RWO            cstor-csi-disk   84s
prometheus-tobs-kube-prometheus-prometheus-db-prometheus-tobs-kube-prometheus-prometheus-1   Bound    pvc-297ae3ae-266d-4d0d-ab3e-d0077a4fd52b   8Gi        RWO            cstor-csi-disk   84s
tobs-grafana                                                                                 Bound    pvc-a11b82b4-ac25-4d13-9947-1294523bfd66   10Gi       RWO            cstor-csi-disk   92s
# kc describe -n monitoring pod/prometheus-tobs-kube-prometheus-prometheus-0
Name:           prometheus-tobs-kube-prometheus-prometheus-0
Namespace:      monitoring
Priority:       0
Node:           kwrctvm3/10.5.6.7
Start Time:     Wed, 24 Aug 2022 12:21:24 +0200
Labels:         app.kubernetes.io/instance=tobs-kube-prometheus-prometheus
                app.kubernetes.io/managed-by=prometheus-operator
                app.kubernetes.io/name=prometheus
                app.kubernetes.io/version=2.38.0
                controller-revision-hash=prometheus-tobs-kube-prometheus-prometheus-6f494cb5d8
                operator.prometheus.io/name=tobs-kube-prometheus-prometheus
                operator.prometheus.io/shard=0
                prometheus=tobs-kube-prometheus-prometheus
                statefulset.kubernetes.io/pod-name=prometheus-tobs-kube-prometheus-prometheus-0
Annotations:    kubectl.kubernetes.io/default-container: prometheus
Status:         Pending
IP:
IPs:            <none>
Controlled By:  StatefulSet/prometheus-tobs-kube-prometheus-prometheus
Init Containers:
  init-config-reloader:
    Container ID:
    Image:         quay.io/prometheus-operator/prometheus-config-reloader:v0.58.0
    Image ID:
    Port:          8080/TCP
    Host Port:     0/TCP
    Command:
      /bin/prometheus-config-reloader
    Args:
      --watch-interval=0
      --listen-address=:8080
      --config-file=/etc/prometheus/config/prometheus.yaml.gz
      --config-envsubst-file=/etc/prometheus/config_out/prometheus.env.yaml
      --watched-dir=/etc/prometheus/rules/prometheus-tobs-kube-prometheus-prometheus-rulefiles-0
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:     200m
      memory:  50Mi
    Requests:
      cpu:     100m
      memory:  50Mi
    Environment:
      POD_NAME:  prometheus-tobs-kube-prometheus-prometheus-0 (v1:metadata.name)
      SHARD:     0
    Mounts:
      /etc/prometheus/config from config (rw)
      /etc/prometheus/config_out from config-out (rw)
      /etc/prometheus/rules/prometheus-tobs-kube-prometheus-prometheus-rulefiles-0 from prometheus-tobs-kube-prometheus-prometheus-rulefiles-0 (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-68cnc (ro)
Containers:
  prometheus:
    Container ID:
    Image:         quay.io/prometheus/prometheus:v2.38.0
    Image ID:
    Port:          9090/TCP
    Host Port:     0/TCP
    Args:
      --web.console.templates=/etc/prometheus/consoles
      --web.console.libraries=/etc/prometheus/console_libraries
      --storage.tsdb.retention.time=1d
      --config.file=/etc/prometheus/config_out/prometheus.env.yaml
      --storage.tsdb.path=/prometheus
      --web.enable-lifecycle
      --web.external-url=http://tobs-kube-prometheus-prometheus.monitoring:9090
      --web.route-prefix=/
      --web.config.file=/etc/prometheus/web_config/web-config.yaml
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Requests:
      cpu:        40m
      memory:     400Mi
    Liveness:     http-get http://:http-web/-/healthy delay=0s timeout=3s period=5s #success=1 #failure=6
    Readiness:    http-get http://:http-web/-/ready delay=0s timeout=3s period=5s #success=1 #failure=3
    Startup:      http-get http://:http-web/-/ready delay=0s timeout=3s period=15s #success=1 #failure=60
    Environment:  <none>
    Mounts:
      /etc/prometheus/certs from tls-assets (ro)
      /etc/prometheus/config_out from config-out (ro)
      /etc/prometheus/rules/prometheus-tobs-kube-prometheus-prometheus-rulefiles-0 from prometheus-tobs-kube-prometheus-prometheus-rulefiles-0 (rw)
      /etc/prometheus/web_config/web-config.yaml from web-config (ro,path="web-config.yaml")
      /prometheus from prometheus-tobs-kube-prometheus-prometheus-db (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-68cnc (ro)
  config-reloader:
    Container ID:
    Image:         quay.io/prometheus-operator/prometheus-config-reloader:v0.58.0
    Image ID:
    Port:          8080/TCP
    Host Port:     0/TCP
    Command:
      /bin/prometheus-config-reloader
    Args:
      --listen-address=:8080
      --reload-url=http://127.0.0.1:9090/-/reload
      --config-file=/etc/prometheus/config/prometheus.yaml.gz
      --config-envsubst-file=/etc/prometheus/config_out/prometheus.env.yaml
      --watched-dir=/etc/prometheus/rules/prometheus-tobs-kube-prometheus-prometheus-rulefiles-0
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:     200m
      memory:  50Mi
    Requests:
      cpu:     100m
      memory:  50Mi
    Environment:
      POD_NAME:  prometheus-tobs-kube-prometheus-prometheus-0 (v1:metadata.name)
      SHARD:     0
    Mounts:
      /etc/prometheus/config from config (rw)
      /etc/prometheus/config_out from config-out (rw)
      /etc/prometheus/rules/prometheus-tobs-kube-prometheus-prometheus-rulefiles-0 from prometheus-tobs-kube-prometheus-prometheus-rulefiles-0 (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-68cnc (ro)
Conditions:
  Type              Status
  Initialized       False
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  prometheus-tobs-kube-prometheus-prometheus-db:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  prometheus-tobs-kube-prometheus-prometheus-db-prometheus-tobs-kube-prometheus-prometheus-0
    ReadOnly:   false
  config:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  prometheus-tobs-kube-prometheus-prometheus
    Optional:    false
  tls-assets:
    Type:                Projected (a volume that contains injected data from multiple sources)
    SecretName:          prometheus-tobs-kube-prometheus-prometheus-tls-assets-0
    SecretOptionalName:  <nil>
  config-out:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  prometheus-tobs-kube-prometheus-prometheus-rulefiles-0:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      prometheus-tobs-kube-prometheus-prometheus-rulefiles-0
    Optional:  false
  web-config:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  prometheus-tobs-kube-prometheus-prometheus-web-config
    Optional:    false
  kube-api-access-68cnc:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason            Age                   From               Message
  ----     ------            ----                  ----               -------
  Warning  FailedScheduling  7m8s                  default-scheduler  0/11 nodes are available: 11 pod has unbound immediate PersistentVolumeClaims. preemption: 0/11 nodes are available: 11 Preemption is not helpful for scheduling.
  Normal   Scheduled         7m7s                  default-scheduler  Successfully assigned monitoring/prometheus-tobs-kube-prometheus-prometheus-0 to kwrctvm3
  Warning  FailedMount       2m49s                 kubelet            Unable to attach or mount volumes: unmounted volumes=[prometheus-tobs-kube-prometheus-prometheus-db], unattached volumes=[prometheus-tobs-kube-prometheus-prometheus-rulefiles-0 kube-api-access-68cnc tls-assets prometheus-tobs-kube-prometheus-prometheus-db web-config config config-out]: timed out waiting for the condition
  Warning  FailedMount       77s (x10 over 6m57s)  kubelet            MountVolume.MountDevice failed for volume "pvc-878e2454-58da-4dca-97ff-e97b9fc05cd2" : rpc error: code = Internal desc = Waiting for pvc-878e2454-58da-4dca-97ff-e97b9fc05cd2's CVC to be bound
  Warning  FailedMount       31s (x2 over 5m4s)    kubelet            Unable to attach or mount volumes: unmounted volumes=[prometheus-tobs-kube-prometheus-prometheus-db], unattached volumes=[prometheus-tobs-kube-prometheus-prometheus-db web-config config config-out prometheus-tobs-kube-prometheus-prometheus-rulefiles-0 kube-api-access-68cnc tls-assets]: timed out waiting for the condition
# kubectl describe cvc -n openebs | grep pvc-878e2454-58da-4dca-97ff-e97b9fc05cd2
Name:         pvc-878e2454-58da-4dca-97ff-e97b9fc05cd2
              openebs.io/volumeID: pvc-878e2454-58da-4dca-97ff-e97b9fc05cd2
  Warning  Provisioning  67s (x35 over 11m)  cstorvolumeclaim-controller  CStorVolume.cstor.openebs.io "pvc-878e2454-58da-4dca-97ff-e97b9fc05cd2" is invalid: metadata.labels: Invalid value: "prometheus-tobs-kube-prometheus-prometheus-db-prometheus-tobs-kube-prometheus-prometheus-0": must be no more than 63 characters
nhudson commented 2 years ago

@furynick so after some testing I would suggest that if you're using openebs to add the following to the configuration to make this work. 

kube-prometheus-stack:
  storageSpec:
    volumeClaimTemplate: 
        metadata:
          name: foo

The main issue is with kube-prometheus-stack the way it's creating the PVC name is defined there and is not being set by tobs. I did create a PR to to kube-prometheus-stack Helm chart to allow for templating in both the Prometheus and Alertmanager storage specification https://github.com/prometheus-community/helm-charts/pull/2415 which will help with this.

After talking about this issue with the team I think the best path forward is to make the configuration of the volume claim optional in tobs and document how to make it work depending on the option you are wanting to set.

furynick commented 2 years ago

Sorry for late answer but it doesn't seem to make any difference unless I made a mistake in syntax or misunderstood your comment.

#> helm install --debug --wait -n monitoring --create-namespace --set opentelemetry-operator.enabled=false --set kube-prometheus-stack.storageSpec.volumeClaimTemplate.metadata.name=prometheus-data tobs timescale/tobs
install.go:178: [debug] Original chart version: ""
install.go:195: [debug] CHART PATH: /root/.cache/helm/repository/tobs-14.7.0.tgz

client.go:128: [debug] creating 1 resource(s)
client.go:128: [debug] creating 1 resource(s)
client.go:128: [debug] creating 1 resource(s)
client.go:128: [debug] creating 1 resource(s)
client.go:128: [debug] creating 1 resource(s)
client.go:128: [debug] creating 1 resource(s)
client.go:128: [debug] creating 1 resource(s)
client.go:128: [debug] creating 1 resource(s)
install.go:165: [debug] Clearing discovery cache
wait.go:48: [debug] beginning wait for 8 resources with timeout of 1m0s
client.go:128: [debug] creating 1 resource(s)
client.go:310: [debug] Starting delete for "custom-secret-scripts" Secret
client.go:339: [debug] secrets "custom-secret-scripts" not found
client.go:128: [debug] creating 1 resource(s)
client.go:310: [debug] Starting delete for "tobs-certificate" Secret
client.go:339: [debug] secrets "tobs-certificate" not found
client.go:128: [debug] creating 1 resource(s)
client.go:310: [debug] Starting delete for "tobs-credentials" Secret
client.go:339: [debug] secrets "tobs-credentials" not found
client.go:128: [debug] creating 1 resource(s)
client.go:310: [debug] Starting delete for "tobs-kube-prometheus-stack-admission" ServiceAccount
client.go:339: [debug] serviceaccounts "tobs-kube-prometheus-stack-admission" not found
client.go:128: [debug] creating 1 resource(s)
client.go:310: [debug] Starting delete for "tobs-kube-prometheus-stack-admission" ClusterRole
client.go:339: [debug] clusterroles.rbac.authorization.k8s.io "tobs-kube-prometheus-stack-admission" not found
client.go:128: [debug] creating 1 resource(s)
client.go:310: [debug] Starting delete for "tobs-kube-prometheus-stack-admission" ClusterRoleBinding
client.go:339: [debug] clusterrolebindings.rbac.authorization.k8s.io "tobs-kube-prometheus-stack-admission" not found
client.go:128: [debug] creating 1 resource(s)
client.go:310: [debug] Starting delete for "tobs-kube-prometheus-stack-admission" Role
client.go:339: [debug] roles.rbac.authorization.k8s.io "tobs-kube-prometheus-stack-admission" not found
client.go:128: [debug] creating 1 resource(s)
client.go:310: [debug] Starting delete for "tobs-kube-prometheus-stack-admission" RoleBinding
client.go:339: [debug] rolebindings.rbac.authorization.k8s.io "tobs-kube-prometheus-stack-admission" not found
client.go:128: [debug] creating 1 resource(s)
client.go:310: [debug] Starting delete for "tobs-kube-prometheus-stack-admission-create" Job
client.go:339: [debug] jobs.batch "tobs-kube-prometheus-stack-admission-create" not found
client.go:128: [debug] creating 1 resource(s)
client.go:540: [debug] Watching for changes to Job tobs-kube-prometheus-stack-admission-create with timeout of 5m0s
client.go:568: [debug] Add/Modify event for tobs-kube-prometheus-stack-admission-create: ADDED
client.go:607: [debug] tobs-kube-prometheus-stack-admission-create: Jobs active: 0, jobs failed: 0, jobs succeeded: 0
client.go:568: [debug] Add/Modify event for tobs-kube-prometheus-stack-admission-create: MODIFIED
client.go:607: [debug] tobs-kube-prometheus-stack-admission-create: Jobs active: 1, jobs failed: 0, jobs succeeded: 0
client.go:568: [debug] Add/Modify event for tobs-kube-prometheus-stack-admission-create: MODIFIED
client.go:310: [debug] Starting delete for "tobs-pgbackrest" Secret
client.go:339: [debug] secrets "tobs-pgbackrest" not found
client.go:128: [debug] creating 1 resource(s)
client.go:310: [debug] Starting delete for "tobs-promscale" ConfigMap
client.go:339: [debug] configmaps "tobs-promscale" not found
client.go:128: [debug] creating 1 resource(s)
client.go:310: [debug] Starting delete for "tobs-kube-prometheus-stack-admission" ServiceAccount
client.go:310: [debug] Starting delete for "tobs-kube-prometheus-stack-admission" ClusterRole
client.go:310: [debug] Starting delete for "tobs-kube-prometheus-stack-admission" ClusterRoleBinding
client.go:310: [debug] Starting delete for "tobs-kube-prometheus-stack-admission" Role
client.go:310: [debug] Starting delete for "tobs-kube-prometheus-stack-admission" RoleBinding
client.go:310: [debug] Starting delete for "tobs-kube-prometheus-stack-admission-create" Job
client.go:229: [debug] checking 138 resources for changes
client.go:250: [debug] Created a new ServiceAccount called "tobs-grafana" in monitoring

client.go:250: [debug] Created a new ServiceAccount called "tobs-grafana-test" in monitoring

client.go:250: [debug] Created a new ServiceAccount called "tobs-kube-state-metrics" in monitoring

client.go:250: [debug] Created a new ServiceAccount called "tobs-prometheus-node-exporter" in monitoring

client.go:250: [debug] Created a new ServiceAccount called "tobs-kube-prometheus-stack-alertmanager" in monitoring

client.go:250: [debug] Created a new ServiceAccount called "tobs-kube-prometheus-stack-operator" in monitoring

client.go:250: [debug] Created a new ServiceAccount called "tobs-kube-prometheus-stack-prometheus" in monitoring

client.go:250: [debug] Created a new ServiceAccount called "tobs-promscale" in monitoring

client.go:250: [debug] Created a new ServiceAccount called "tobs-timescaledb" in monitoring

client.go:250: [debug] Created a new ServiceAccount called "tobs-promscale-initializer-sa" in monitoring

client.go:250: [debug] Created a new Secret called "tobs-grafana" in monitoring

client.go:250: [debug] Created a new Secret called "alertmanager-tobs-kube-prometheus-stack-alertmanager" in monitoring

client.go:250: [debug] Created a new Secret called "tobs-promscale-connection" in monitoring

client.go:250: [debug] Created a new Secret called "tobs-grafana-datasources" in monitoring

client.go:250: [debug] Created a new Secret called "tobs-grafana-db" in monitoring

client.go:250: [debug] Created a new Secret called "tobs-scrape-config" in monitoring

client.go:250: [debug] Created a new ConfigMap called "tobs-grafana-config-dashboards" in monitoring

client.go:250: [debug] Created a new ConfigMap called "tobs-grafana" in monitoring

client.go:250: [debug] Created a new ConfigMap called "tobs-grafana-test" in monitoring

client.go:250: [debug] Created a new ConfigMap called "tobs-kube-prometheus-stack-grafana-datasource" in monitoring

client.go:250: [debug] Created a new ConfigMap called "tobs-kube-prometheus-stack-alertmanager-overview" in monitoring

client.go:250: [debug] Created a new ConfigMap called "tobs-kube-prometheus-stack-apiserver" in monitoring

client.go:250: [debug] Created a new ConfigMap called "tobs-kube-prometheus-stack-cluster-total" in monitoring

client.go:250: [debug] Created a new ConfigMap called "tobs-kube-prometheus-stack-controller-manager" in monitoring

client.go:250: [debug] Created a new ConfigMap called "tobs-kube-prometheus-stack-etcd" in monitoring

client.go:250: [debug] Created a new ConfigMap called "tobs-kube-prometheus-stack-grafana-overview" in monitoring

client.go:250: [debug] Created a new ConfigMap called "tobs-kube-prometheus-stack-k8s-coredns" in monitoring

client.go:250: [debug] Created a new ConfigMap called "tobs-kube-prometheus-stack-k8s-resources-cluster" in monitoring

client.go:250: [debug] Created a new ConfigMap called "tobs-kube-prometheus-stack-k8s-resources-namespace" in monitoring

client.go:250: [debug] Created a new ConfigMap called "tobs-kube-prometheus-stack-k8s-resources-node" in monitoring

client.go:250: [debug] Created a new ConfigMap called "tobs-kube-prometheus-stack-k8s-resources-pod" in monitoring

client.go:250: [debug] Created a new ConfigMap called "tobs-kube-prometheus-stack-k8s-resources-workload" in monitoring

client.go:250: [debug] Created a new ConfigMap called "tobs-kube-prometheus-stack-k8s-resources-workloads-namespace" in monitoring

client.go:250: [debug] Created a new ConfigMap called "tobs-kube-prometheus-stack-kubelet" in monitoring

client.go:250: [debug] Created a new ConfigMap called "tobs-kube-prometheus-stack-namespace-by-pod" in monitoring

client.go:250: [debug] Created a new ConfigMap called "tobs-kube-prometheus-stack-namespace-by-workload" in monitoring

client.go:250: [debug] Created a new ConfigMap called "tobs-kube-prometheus-stack-node-cluster-rsrc-use" in monitoring

client.go:250: [debug] Created a new ConfigMap called "tobs-kube-prometheus-stack-node-rsrc-use" in monitoring

client.go:250: [debug] Created a new ConfigMap called "tobs-kube-prometheus-stack-nodes-darwin" in monitoring

client.go:250: [debug] Created a new ConfigMap called "tobs-kube-prometheus-stack-nodes" in monitoring

client.go:250: [debug] Created a new ConfigMap called "tobs-kube-prometheus-stack-persistentvolumesusage" in monitoring

client.go:250: [debug] Created a new ConfigMap called "tobs-kube-prometheus-stack-pod-total" in monitoring

client.go:250: [debug] Created a new ConfigMap called "tobs-kube-prometheus-stack-prometheus" in monitoring

client.go:250: [debug] Created a new ConfigMap called "tobs-kube-prometheus-stack-proxy" in monitoring

client.go:250: [debug] Created a new ConfigMap called "tobs-kube-prometheus-stack-scheduler" in monitoring

client.go:250: [debug] Created a new ConfigMap called "tobs-kube-prometheus-stack-workload-total" in monitoring

client.go:250: [debug] Created a new ConfigMap called "tobs-timescaledb-patroni" in monitoring

client.go:250: [debug] Created a new ConfigMap called "tobs-timescaledb-pgbackrest" in monitoring

client.go:250: [debug] Created a new ConfigMap called "tobs-timescaledb-scripts" in monitoring

client.go:250: [debug] Created a new ConfigMap called "tobs-connection-initializer" in monitoring

client.go:250: [debug] Created a new ConfigMap called "tobs-grafana-dashboards" in monitoring

client.go:250: [debug] Created a new ConfigMap called "custom-init-scripts" in monitoring

client.go:250: [debug] Created a new PersistentVolumeClaim called "tobs-grafana" in monitoring

client.go:512: [debug] Looks like there are no changes for ClusterRole "tobs-grafana-clusterrole"
client.go:512: [debug] Looks like there are no changes for ClusterRole "tobs-kube-state-metrics"
client.go:512: [debug] Looks like there are no changes for ClusterRole "tobs-kube-prometheus-stack-operator"
client.go:512: [debug] Looks like there are no changes for ClusterRole "tobs-kube-prometheus-stack-prometheus"
client.go:512: [debug] Looks like there are no changes for ClusterRoleBinding "tobs-grafana-clusterrolebinding"
client.go:512: [debug] Looks like there are no changes for ClusterRoleBinding "tobs-kube-state-metrics"
client.go:512: [debug] Looks like there are no changes for ClusterRoleBinding "tobs-kube-prometheus-stack-operator"
client.go:512: [debug] Looks like there are no changes for ClusterRoleBinding "tobs-kube-prometheus-stack-prometheus"
client.go:250: [debug] Created a new Role called "tobs-grafana" in monitoring

client.go:250: [debug] Created a new Role called "tobs-timescaledb" in monitoring

client.go:250: [debug] Created a new Role called "tobs-promscale-initializer-role" in monitoring

client.go:250: [debug] Created a new RoleBinding called "tobs-grafana" in monitoring

client.go:250: [debug] Created a new RoleBinding called "tobs-timescaledb" in monitoring

client.go:250: [debug] Created a new RoleBinding called "tobs-promscale-initializer-rolebinding" in monitoring

client.go:250: [debug] Created a new Service called "tobs-grafana" in monitoring

client.go:250: [debug] Created a new Service called "tobs-kube-state-metrics" in monitoring

client.go:250: [debug] Created a new Service called "tobs-prometheus-node-exporter" in monitoring

client.go:250: [debug] Created a new Service called "tobs-kube-prometheus-stack-alertmanager" in monitoring

client.go:512: [debug] Looks like there are no changes for Service "tobs-kube-prometheus-stack-coredns"
client.go:512: [debug] Looks like there are no changes for Service "tobs-kube-prometheus-stack-kube-controller-manager"
client.go:512: [debug] Looks like there are no changes for Service "tobs-kube-prometheus-stack-kube-etcd"
client.go:512: [debug] Looks like there are no changes for Service "tobs-kube-prometheus-stack-kube-proxy"
client.go:512: [debug] Looks like there are no changes for Service "tobs-kube-prometheus-stack-kube-scheduler"
client.go:250: [debug] Created a new Service called "tobs-kube-prometheus-stack-operator" in monitoring

client.go:250: [debug] Created a new Service called "tobs-kube-prometheus-stack-prometheus" in monitoring

client.go:250: [debug] Created a new Service called "tobs-promscale" in monitoring

client.go:250: [debug] Created a new Service called "tobs-config" in monitoring

client.go:250: [debug] Created a new Service called "tobs-replica" in monitoring

client.go:250: [debug] Created a new Service called "tobs" in monitoring

client.go:250: [debug] Created a new DaemonSet called "tobs-prometheus-node-exporter" in monitoring

client.go:250: [debug] Created a new Deployment called "tobs-grafana" in monitoring

client.go:250: [debug] Created a new Deployment called "tobs-kube-state-metrics" in monitoring

client.go:250: [debug] Created a new Deployment called "tobs-kube-prometheus-stack-operator" in monitoring

W0916 09:03:46.769218 2340124 warnings.go:70] spec.template.spec.containers[0].env[2].name: duplicate name "TOBS_TELEMETRY_INSTALLED_BY"
W0916 09:03:46.769237 2340124 warnings.go:70] spec.template.spec.containers[0].env[3].name: duplicate name "TOBS_TELEMETRY_VERSION"
client.go:250: [debug] Created a new Deployment called "tobs-promscale" in monitoring

client.go:250: [debug] Created a new StatefulSet called "tobs-timescaledb" in monitoring

client.go:250: [debug] Created a new Job called "tobs-connection-secret" in monitoring

client.go:250: [debug] Created a new Alertmanager called "tobs-kube-prometheus-stack-alertmanager" in monitoring

client.go:521: [debug] Patch MutatingWebhookConfiguration "tobs-kube-prometheus-stack-admission" in namespace
client.go:250: [debug] Created a new PodMonitor called "tobs-timescaledb" in monitoring

client.go:250: [debug] Created a new Prometheus called "tobs-kube-prometheus-stack-prometheus" in monitoring

Error: INSTALLATION FAILED: failed to create resource: Internal error occurred: failed calling webhook "prometheusrulemutate.monitoring.coreos.com": failed to call webhook: Post "https://tobs-kube-prometheus-stack-operator.monitoring.svc:443/admission-prometheusrules/validate?timeout=10s": dial tcp 172.16.29.173:443: connect: connection refused
helm.go:84: [debug] Internal error occurred: failed calling webhook "prometheusrulemutate.monitoring.coreos.com": failed to call webhook: Post "https://tobs-kube-prometheus-stack-operator.monitoring.svc:443/admission-prometheusrules/validate?timeout=10s": dial tcp 172.16.29.173:443: connect: connection refused
failed to create resource
helm.sh/helm/v3/pkg/kube.(*Client).Update.func1
        helm.sh/helm/v3/pkg/kube/client.go:246
helm.sh/helm/v3/pkg/kube.ResourceList.Visit
        helm.sh/helm/v3/pkg/kube/resource.go:32
helm.sh/helm/v3/pkg/kube.(*Client).Update
        helm.sh/helm/v3/pkg/kube/client.go:230
helm.sh/helm/v3/pkg/action.(*Install).performInstall
        helm.sh/helm/v3/pkg/action/install.go:375
runtime.goexit
        runtime/asm_amd64.s:1581
INSTALLATION FAILED
main.newInstallCmd.func2
        helm.sh/helm/v3/cmd/helm/install.go:127
github.com/spf13/cobra.(*Command).execute
        github.com/spf13/cobra@v1.4.0/command.go:856
github.com/spf13/cobra.(*Command).ExecuteC
        github.com/spf13/cobra@v1.4.0/command.go:974
github.com/spf13/cobra.(*Command).Execute
        github.com/spf13/cobra@v1.4.0/command.go:902
main.main
        helm.sh/helm/v3/cmd/helm/helm.go:83
runtime.main
        runtime/proc.go:255
runtime.goexit
        runtime/asm_amd64.s:1581
#> kc get all
NAME                                                         READY   STATUS      RESTARTS        AGE
pod/alertmanager-tobs-kube-prometheus-stack-alertmanager-0   2/2     Running     0               5m34s
pod/alertmanager-tobs-kube-prometheus-stack-alertmanager-1   2/2     Running     0               5m34s
pod/alertmanager-tobs-kube-prometheus-stack-alertmanager-2   2/2     Running     0               5m34s
pod/prometheus-tobs-kube-prometheus-stack-prometheus-0       0/2     Init:0/1    0               5m34s
pod/prometheus-tobs-kube-prometheus-stack-prometheus-1       0/2     Init:0/1    0               5m34s
pod/tobs-connection-secret-rpdwj                             0/1     Completed   0               5m38s
pod/tobs-grafana-c99c96767-9t6s7                             3/3     Running     0               5m39s
pod/tobs-kube-prometheus-stack-operator-7485c776c-qn4k9      1/1     Running     0               5m39s
pod/tobs-kube-state-metrics-79569746c9-vslr2                 1/1     Running     0               5m39s
pod/tobs-prometheus-node-exporter-7wt2c                      1/1     Running     0               5m39s
pod/tobs-prometheus-node-exporter-cnqp2                      1/1     Running     0               5m39s
pod/tobs-prometheus-node-exporter-dggg6                      1/1     Running     0               5m39s
pod/tobs-prometheus-node-exporter-fnkbh                      1/1     Running     0               5m39s
pod/tobs-prometheus-node-exporter-k262s                      1/1     Running     0               5m39s
pod/tobs-prometheus-node-exporter-kmsq2                      1/1     Running     0               5m39s
pod/tobs-prometheus-node-exporter-kn7vv                      1/1     Running     0               5m39s
pod/tobs-prometheus-node-exporter-pnn8p                      1/1     Running     0               5m39s
pod/tobs-prometheus-node-exporter-qtwg4                      1/1     Running     0               5m39s
pod/tobs-prometheus-node-exporter-vjxw4                      1/1     Running     0               5m39s
pod/tobs-prometheus-node-exporter-vnmr4                      1/1     Running     0               5m39s
pod/tobs-promscale-7c5b5d56d5-szfnn                          1/1     Running     3 (2m21s ago)   5m39s
pod/tobs-timescaledb-0                                       2/2     Running     0               5m38s

NAME                                              TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
service/alertmanager-operated                     ClusterIP   None            <none>        9093/TCP,9094/TCP,9094/UDP   5m34s
service/prometheus-operated                       ClusterIP   None            <none>        9090/TCP                     5m34s
service/tobs                                      ClusterIP   172.16.31.57    <none>        5432/TCP                     5m39s
service/tobs-config                               ClusterIP   None            <none>        8008/TCP                     5m39s
service/tobs-grafana                              ClusterIP   172.16.29.205   <none>        80/TCP                       5m39s
service/tobs-kube-prometheus-stack-alertmanager   ClusterIP   172.16.29.70    <none>        9093/TCP                     5m39s
service/tobs-kube-prometheus-stack-operator       ClusterIP   172.16.29.173   <none>        443/TCP                      5m39s
service/tobs-kube-prometheus-stack-prometheus     ClusterIP   172.16.31.62    <none>        9090/TCP                     5m39s
service/tobs-kube-state-metrics                   ClusterIP   172.16.22.0     <none>        8080/TCP                     5m39s
service/tobs-prometheus-node-exporter             ClusterIP   172.16.25.202   <none>        9100/TCP                     5m39s
service/tobs-promscale                            ClusterIP   172.16.17.201   <none>        9201/TCP,9202/TCP            5m39s
service/tobs-replica                              ClusterIP   172.16.28.144   <none>        5432/TCP                     5m39s

NAME                                           DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
daemonset.apps/tobs-prometheus-node-exporter   11        11        11      11           11          <none>          5m39s

NAME                                                  READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/tobs-grafana                          1/1     1            1           5m39s
deployment.apps/tobs-kube-prometheus-stack-operator   1/1     1            1           5m39s
deployment.apps/tobs-kube-state-metrics               1/1     1            1           5m39s
deployment.apps/tobs-promscale                        1/1     1            1           5m39s

NAME                                                            DESIRED   CURRENT   READY   AGE
replicaset.apps/tobs-grafana-c99c96767                          1         1         1       5m39s
replicaset.apps/tobs-kube-prometheus-stack-operator-7485c776c   1         1         1       5m39s
replicaset.apps/tobs-kube-state-metrics-79569746c9              1         1         1       5m39s
replicaset.apps/tobs-promscale-7c5b5d56d5                       1         1         1       5m39s

NAME                                                                    READY   AGE
statefulset.apps/alertmanager-tobs-kube-prometheus-stack-alertmanager   3/3     5m34s
statefulset.apps/prometheus-tobs-kube-prometheus-stack-prometheus       0/2     5m34s
statefulset.apps/tobs-timescaledb                                       1/1     5m39s

NAME                               COMPLETIONS   DURATION   AGE
job.batch/tobs-connection-secret   1/1           6s         5m39s
#> kc describe pod/prometheus-tobs-kube-prometheus-stack-prometheus-0
Name:           prometheus-tobs-kube-prometheus-stack-prometheus-0
Namespace:      monitoring
Priority:       0
Node:           kwvm1/10.x.y.z
Start Time:     Fri, 16 Sep 2022 09:03:52 +0200
Labels:         app.kubernetes.io/instance=tobs-kube-prometheus-stack-prometheus
                app.kubernetes.io/managed-by=prometheus-operator
                app.kubernetes.io/name=prometheus
                app.kubernetes.io/version=2.38.0
                controller-revision-hash=prometheus-tobs-kube-prometheus-stack-prometheus-bc8d6769d
                operator.prometheus.io/name=tobs-kube-prometheus-stack-prometheus
                operator.prometheus.io/shard=0
                prometheus=tobs-kube-prometheus-stack-prometheus
                statefulset.kubernetes.io/pod-name=prometheus-tobs-kube-prometheus-stack-prometheus-0
Annotations:    kubectl.kubernetes.io/default-container: prometheus
Status:         Pending
IP:
IPs:            <none>
Controlled By:  StatefulSet/prometheus-tobs-kube-prometheus-stack-prometheus
Init Containers:
  init-config-reloader:
    Container ID:
    Image:         quay.io/prometheus-operator/prometheus-config-reloader:v0.59.1
    Image ID:
    Port:          8080/TCP
    Host Port:     0/TCP
    Command:
      /bin/prometheus-config-reloader
    Args:
      --watch-interval=0
      --listen-address=:8080
      --config-file=/etc/prometheus/config/prometheus.yaml.gz
      --config-envsubst-file=/etc/prometheus/config_out/prometheus.env.yaml
      --watched-dir=/etc/prometheus/rules/prometheus-tobs-kube-prometheus-stack-prometheus-rulefiles-0
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:     200m
      memory:  50Mi
    Requests:
      cpu:     100m
      memory:  50Mi
    Environment:
      POD_NAME:  prometheus-tobs-kube-prometheus-stack-prometheus-0 (v1:metadata.name)
      SHARD:     0
    Mounts:
      /etc/prometheus/config from config (rw)
      /etc/prometheus/config_out from config-out (rw)
      /etc/prometheus/rules/prometheus-tobs-kube-prometheus-stack-prometheus-rulefiles-0 from prometheus-tobs-kube-prometheus-stack-prometheus-rulefiles-0 (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-2mdgw (ro)
Containers:
  prometheus:
    Container ID:
    Image:         quay.io/prometheus/prometheus:v2.38.0
    Image ID:
    Port:          9090/TCP
    Host Port:     0/TCP
    Args:
      --web.console.templates=/etc/prometheus/consoles
      --web.console.libraries=/etc/prometheus/console_libraries
      --storage.tsdb.retention.time=1d
      --config.file=/etc/prometheus/config_out/prometheus.env.yaml
      --storage.tsdb.path=/prometheus
      --web.enable-lifecycle
      --web.external-url=http://tobs-kube-prometheus-stack-prometheus.monitoring:9090
      --web.route-prefix=/
      --web.config.file=/etc/prometheus/web_config/web-config.yaml
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Requests:
      cpu:        40m
      memory:     400Mi
    Liveness:     http-get http://:http-web/-/healthy delay=0s timeout=3s period=5s #success=1 #failure=6
    Readiness:    http-get http://:http-web/-/ready delay=0s timeout=3s period=5s #success=1 #failure=3
    Startup:      http-get http://:http-web/-/ready delay=0s timeout=3s period=15s #success=1 #failure=60
    Environment:  <none>
    Mounts:
      /etc/prometheus/certs from tls-assets (ro)
      /etc/prometheus/config_out from config-out (ro)
      /etc/prometheus/rules/prometheus-tobs-kube-prometheus-stack-prometheus-rulefiles-0 from prometheus-tobs-kube-prometheus-stack-prometheus-rulefiles-0 (rw)
      /etc/prometheus/web_config/web-config.yaml from web-config (ro,path="web-config.yaml")
      /prometheus from prometheus-tobs-kube-prometheus-stack-prometheus-db (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-2mdgw (ro)
  config-reloader:
    Container ID:
    Image:         quay.io/prometheus-operator/prometheus-config-reloader:v0.59.1
    Image ID:
    Port:          8080/TCP
    Host Port:     0/TCP
    Command:
      /bin/prometheus-config-reloader
    Args:
      --listen-address=:8080
      --reload-url=http://127.0.0.1:9090/-/reload
      --config-file=/etc/prometheus/config/prometheus.yaml.gz
      --config-envsubst-file=/etc/prometheus/config_out/prometheus.env.yaml
      --watched-dir=/etc/prometheus/rules/prometheus-tobs-kube-prometheus-stack-prometheus-rulefiles-0
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:     200m
      memory:  50Mi
    Requests:
      cpu:     100m
      memory:  50Mi
    Environment:
      POD_NAME:  prometheus-tobs-kube-prometheus-stack-prometheus-0 (v1:metadata.name)
      SHARD:     0
    Mounts:
      /etc/prometheus/config from config (rw)
      /etc/prometheus/config_out from config-out (rw)
      /etc/prometheus/rules/prometheus-tobs-kube-prometheus-stack-prometheus-rulefiles-0 from prometheus-tobs-kube-prometheus-stack-prometheus-rulefiles-0 (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-2mdgw (ro)
Conditions:
  Type              Status
  Initialized       False
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  prometheus-tobs-kube-prometheus-stack-prometheus-db:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  prometheus-tobs-kube-prometheus-stack-prometheus-db-prometheus-tobs-kube-prometheus-stack-prometheus-0
    ReadOnly:   false
  config:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  prometheus-tobs-kube-prometheus-stack-prometheus
    Optional:    false
  tls-assets:
    Type:                Projected (a volume that contains injected data from multiple sources)
    SecretName:          prometheus-tobs-kube-prometheus-stack-prometheus-tls-assets-0
    SecretOptionalName:  <nil>
  config-out:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  prometheus-tobs-kube-prometheus-stack-prometheus-rulefiles-0:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      prometheus-tobs-kube-prometheus-stack-prometheus-rulefiles-0
    Optional:  false
  web-config:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  prometheus-tobs-kube-prometheus-stack-prometheus-web-config
    Optional:    false
  kube-api-access-2mdgw:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason            Age                  From               Message
  ----     ------            ----                 ----               -------
  Warning  FailedScheduling  5m56s                default-scheduler  0/11 nodes are available: 11 pod has unbound immediate PersistentVolumeClaims. preemption: 0/11 nodes are available: 11 Preemption is not helpful for scheduling.
  Normal   Scheduled         5m55s                default-scheduler  Successfully assigned monitoring/prometheus-tobs-kube-prometheus-stack-prometheus-0 to kwrctvm1
  Warning  FailedMount       3m52s                kubelet            Unable to attach or mount volumes: unmounted volumes=[prometheus-tobs-kube-prometheus-stack-prometheus-db], unattached volumes=[config-out prometheus-tobs-kube-prometheus-stack-prometheus-rulefiles-0 kube-api-access-2mdgw tls-assets prometheus-tobs-kube-prometheus-stack-prometheus-db web-config config]: timed out waiting for the condition
  Warning  FailedMount       96s                  kubelet            Unable to attach or mount volumes: unmounted volumes=[prometheus-tobs-kube-prometheus-stack-prometheus-db], unattached volumes=[prometheus-tobs-kube-prometheus-stack-prometheus-db web-config config config-out prometheus-tobs-kube-prometheus-stack-prometheus-rulefiles-0 kube-api-access-2mdgw tls-assets]: timed out waiting for the condition
  Warning  FailedMount       3s (x10 over 5m43s)  kubelet            MountVolume.MountDevice failed for volume "pvc-ababcc46-3f79-4410-b3d9-a2c6d3bf6184" : rpc error: code = Internal desc = Waiting for pvc-ababcc46-3f79-4410-b3d9-a2c6d3bf6184's CVC to be bound
#> kc get -n openebs cvc | grep "pvc-ababcc46-3f79-4410-b3d9-a2c6d3bf6184"
pvc-ababcc46-3f79-4410-b3d9-a2c6d3bf6184              Pending   6m28s
#> kc describe -n openebs cvc | grep "pvc-ababcc46-3f79-4410-b3d9-a2c6d3bf6184"
Name:         pvc-ababcc46-3f79-4410-b3d9-a2c6d3bf6184
              openebs.io/volumeID: pvc-ababcc46-3f79-4410-b3d9-a2c6d3bf6184
  Warning  Provisioning  87s (x26 over 7m3s)  cstorvolumeclaim-controller  CStorVolume.cstor.openebs.io "pvc-ababcc46-3f79-4410-b3d9-a2c6d3bf6184" is invalid: metadata.labels: Invalid value: "prometheus-tobs-kube-prometheus-stack-prometheus-db-prometheus-tobs-kube-prometheus-stack-prometheus-0": must be no more than 63 characters
#> kc get pvc
NAME                                                                                                     STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS     AGE
prometheus-tobs-kube-prometheus-stack-prometheus-db-prometheus-tobs-kube-prometheus-stack-prometheus-0   Bound    pvc-ababcc46-3f79-4410-b3d9-a2c6d3bf6184   8Gi        RWO            cstor-csi-disk   19m
prometheus-tobs-kube-prometheus-stack-prometheus-db-prometheus-tobs-kube-prometheus-stack-prometheus-1   Bound    pvc-73283d5a-e136-45b0-bfa8-ed31e38153aa   8Gi        RWO            cstor-csi-disk   19m
storage-volume-tobs-timescaledb-0                                                                        Bound    pvc-2d47da48-f18e-41d9-838a-19d4f7e0ddf3   150Gi      RWO            cstor-csi-disk   19m
tobs-grafana                                                                                             Bound    pvc-71d7144a-c6a2-4681-b1cf-996d8b8e7438   10Gi       RWO            cstor-csi-disk   19m
wal-volume-tobs-timescaledb-0                                                                            Bound    pvc-803a46be-d60a-4b28-9062-4a25a90df8d2   20Gi       RWO            cstor-csi-disk   19m
furynick commented 2 years ago

I fixed command line : kube-prometheus-stack.prometheus.prometheusSpec.storageSpec.volumeClaimTemplate.metadata.name instead of kube-prometheus-stack.storageSpec.volumeClaimTemplate.metadata.name I removed failing webhooks to fix the installation failure but still the same issue.

Any other idea to workaround or any clue of a mistake on my command line ?

nhudson commented 2 years ago

The PR I created (https://github.com/prometheus-community/helm-charts/pull/2415) to allow you to set the PVC name through kube-prometheus-stack was just merged. So once tobs is updated to use that version you should be able to template the PVC names of the volumes that are created with kube-prometheus-stack.

furynick commented 2 years ago

Thanks, I'll keep an eye on new tobs releases and give it a try.

furynick commented 2 years ago

Fixed !

PVC names are slightly shorter with following syntax and OpenEBS volumes are successfully provisionned.

helm install \
  -n monitoring \
  --create-namespace \
  --set timescaledbExternal.enabled=true \
  --set opentelemetry-operator.enabled=false \
  --set kube-prometheus-stack.prometheus.prometheusSpec.storageSpec.volumeClaimTemplate.metadata.name=data \
  --set promscale.connection.uri="postgres://$DB_USER:$DB_PASS@$DB_HOST:$DB_PORT/$DB_NAME?sslmode=require" \
  --set timescaledb-single.enabled=false \
  tobs timescale/tobs