search

Kimi450 / ubuntu_server

Setup an ubuntu server with ease.
85 stars 10 forks source link

Grafana doesnt come up after a machine restart #10

Closed Kimi450 closed 1 year ago

Kimi450 commented 1 year ago

The pod doesnt seem to come up after the machine restart

k describe pod kube-prometheus-stack-grafana-664bfb4777-brcfs

Events:
  Type     Reason          Age                     From               Message
  ----     ------          ----                    ----               -------
  Normal   Scheduled       9m58s                   default-scheduler  Successfully assigned monitoring/kube-prometheus-stack-grafana-664bfb4777-brcfs to minikube
  Normal   Pulled          9m57s                   kubelet            Container image "busybox:1.31.1" already present on machine
  Normal   Created         9m57s                   kubelet            Created container init-chown-data
  Normal   Started         9m57s                   kubelet            Started container init-chown-data
  Normal   Pulled          9m56s                   kubelet            Container image "quay.io/kiwigrid/k8s-sidecar:1.22.0" already present on machine
  Normal   Created         9m56s                   kubelet            Created container grafana-sc-dashboard
  Normal   Started         9m56s                   kubelet            Started container grafana-sc-dashboard
  Normal   Pulled          9m56s                   kubelet            Container image "quay.io/kiwigrid/k8s-sidecar:1.22.0" already present on machine
  Normal   Created         9m56s                   kubelet            Created container grafana-sc-datasources
  Normal   Started         9m55s                   kubelet            Started container grafana-sc-datasources
  Normal   Pulled          9m55s                   kubelet            Container image "grafana/grafana:9.3.8" already present on machine
  Normal   Created         9m55s                   kubelet            Created container grafana
  Normal   Started         9m55s                   kubelet            Started container grafana
  Warning  Unhealthy       9m55s                   kubelet            Readiness probe failed: Get "http://172.17.0.8:3000/api/health": dial tcp 172.17.0.8:3000: connect: connection refused
  Normal   SandboxChanged  7m29s                   kubelet            Pod sandbox changed, it will be killed and re-created.
  Normal   Pulled          7m29s                   kubelet            Container image "busybox:1.31.1" already present on machine
  Normal   Created         7m29s                   kubelet            Created container init-chown-data
  Normal   Started         7m28s                   kubelet            Started container init-chown-data
  Normal   Pulled          7m27s                   kubelet            Container image "quay.io/kiwigrid/k8s-sidecar:1.22.0" already present on machine
  Normal   Created         7m27s                   kubelet            Created container grafana-sc-dashboard
  Normal   Started         7m27s                   kubelet            Started container grafana-sc-dashboard
  Normal   Pulled          7m27s                   kubelet            Container image "quay.io/kiwigrid/k8s-sidecar:1.22.0" already present on machine
  Normal   Created         7m27s                   kubelet            Created container grafana-sc-datasources
  Normal   Started         7m27s                   kubelet            Started container grafana-sc-datasources
  Normal   Pulled          6m56s (x2 over 7m27s)   kubelet            Container image "grafana/grafana:9.3.8" already present on machine
  Normal   Created         6m56s (x2 over 7m27s)   kubelet            Created container grafana
  Normal   Started         6m55s (x2 over 7m26s)   kubelet            Started container grafana
  Warning  Unhealthy       6m55s (x4 over 7m23s)   kubelet            Readiness probe failed: Get "http://172.17.0.8:3000/api/health": dial tcp 172.17.0.8:3000: connect: connection refused
  Warning  BackOff         2m27s (x31 over 7m17s)  kubelet            Back-off restarting failed container

kubectl logs kube-prometheus-stack-grafana-664bfb4777-brcfs -c grafana

logger=settings t=2023-03-28T19:54:52.419438412Z level=info msg="Starting Grafana" version=9.3.8 commit=7ab70dbce8 branch=HEAD compiled=2023-02-27T12:36:16Z
logger=settings t=2023-03-28T19:54:52.419674649Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini
logger=settings t=2023-03-28T19:54:52.419690366Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini
logger=settings t=2023-03-28T19:54:52.419696218Z level=info msg="Config overridden from command line" arg="default.paths.data=/var/lib/grafana/"
logger=settings t=2023-03-28T19:54:52.419701052Z level=info msg="Config overridden from command line" arg="default.paths.logs=/var/log/grafana"
logger=settings t=2023-03-28T19:54:52.419706683Z level=info msg="Config overridden from command line" arg="default.paths.plugins=/var/lib/grafana/plugins"
logger=settings t=2023-03-28T19:54:52.419716105Z level=info msg="Config overridden from command line" arg="default.paths.provisioning=/etc/grafana/provisioning"
logger=settings t=2023-03-28T19:54:52.419723727Z level=info msg="Config overridden from command line" arg="default.log.mode=console"
logger=settings t=2023-03-28T19:54:52.419731733Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana/"
logger=settings t=2023-03-28T19:54:52.419741201Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana"
logger=settings t=2023-03-28T19:54:52.41974942Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins"
logger=settings t=2023-03-28T19:54:52.419762574Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning"
logger=settings t=2023-03-28T19:54:52.419771586Z level=info msg="Config overridden from Environment variable" var="GF_SECURITY_ADMIN_USER=admin"
logger=settings t=2023-03-28T19:54:52.419780473Z level=info msg="Config overridden from Environment variable" var="GF_SECURITY_ADMIN_PASSWORD=*********"
logger=settings t=2023-03-28T19:54:52.419788827Z level=info msg="Path Home" path=/usr/share/grafana
logger=settings t=2023-03-28T19:54:52.419797311Z level=info msg="Path Data" path=/var/lib/grafana/
logger=settings t=2023-03-28T19:54:52.419805624Z level=info msg="Path Logs" path=/var/log/grafana
logger=settings t=2023-03-28T19:54:52.419815426Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins
logger=settings t=2023-03-28T19:54:52.419823325Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning
logger=settings t=2023-03-28T19:54:52.419832662Z level=info msg="App mode production"
logger=sqlstore t=2023-03-28T19:54:52.419911804Z level=info msg="Connecting to DB" dbtype=sqlite3
logger=migrator t=2023-03-28T19:54:52.442653864Z level=info msg="Starting DB migrations"
logger=migrator t=2023-03-28T19:54:52.450275943Z level=info msg="migrations completed" performed=0 skipped=464 duration=856.664µs
logger=plugin.loader t=2023-03-28T19:54:52.527535662Z level=info msg="Plugin registered" pluginID=input
logger=secrets t=2023-03-28T19:54:52.528693724Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1
logger=query_data t=2023-03-28T19:54:52.536724003Z level=info msg="Query Service initialization"
logger=live.push_http t=2023-03-28T19:54:52.544600968Z level=info msg="Live Push Gateway initialization"
logger=infra.usagestats.collector t=2023-03-28T19:54:52.869050753Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2
Failed to start grafana. error: Datasource provisioning error: datasource.yaml config is invalid. Only one datasource per organization can be marked as default
Datasource provisioning error: datasource.yaml config is invalid. Only one datasource per organization can be marked as default
logger=provisioning t=2023-03-28T19:54:52.878440243Z level=error msg="Failed to provision data sources" error="Datasource provisioning error: datasource.yaml config is invalid. Only one datasource per organization can be marked as default"
Kimi450 commented 1 year ago

Issue was caused because of loki.isDefault config which defaults to true, needs to be set to false. Related issue: https://github.com/prometheus-community/helm-charts/issues/2251