search

prometheus-operator / kube-prometheus

Use Prometheus to monitor Kubernetes and applications running on Kubernetes
https://prometheus-operator.dev/
Apache License 2.0
6.73k stars 1.93k forks source link

error="open /etc/grafana/provisioning/notifiers: no such file or directory" #1934

Open EricCaoHZ opened 1 year ago

EricCaoHZ commented 1 year ago

k8s v1.24.7 kube-prometheus main

grafana page cannot be opened

kubectl get pods -n monitoring 
NAME                                  READY   STATUS    RESTARTS       AGE
alertmanager-main-0                   2/2     Running   1 (128m ago)   133m
alertmanager-main-1                   2/2     Running   1 (132m ago)   133m
alertmanager-main-2                   2/2     Running   0              133m
blackbox-exporter-c5547b8bb-dgjzp     3/3     Running   0              134m
grafana-7c4645d965-d4hq5              1/1     Running   0              134m
kube-state-metrics-6dfb6db86d-zj86m   3/3     Running   0              63m
node-exporter-57nl6                   2/2     Running   0              134m
node-exporter-6k5n9                   2/2     Running   0              134m
node-exporter-7zvsq                   2/2     Running   0              134m
node-exporter-l9k2b                   2/2     Running   0              134m
node-exporter-lmfzt                   2/2     Running   0              134m
node-exporter-m5gqh                   2/2     Running   0              134m
node-exporter-t6bh5                   2/2     Running   0              134m
node-exporter-vwclw                   2/2     Running   0              134m
node-exporter-z5nvl                   2/2     Running   0              134m
prometheus-adapter-5fdd5ff55c-28lts   1/1     Running   0              64m
prometheus-adapter-5fdd5ff55c-p2q5s   1/1     Running   0              64m
prometheus-k8s-0                      2/2     Running   0              133m
prometheus-k8s-1                      2/2     Running   0              133m
prometheus-operator-5ddfd64d6-trh8p   2/2     Running   0              134m
kubectl logs -n monitoring grafana-7c4645d965-d4hq5
logger=plugin.manager t=2022-11-06T08:34:29.561132156Z level=info msg="Plugin registered" pluginId=input
logger=secrets t=2022-11-06T08:34:29.56150779Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1
logger=query_data t=2022-11-06T08:34:29.563888364Z level=info msg="Query Service initialization"
logger=live.push_http t=2022-11-06T08:34:29.569859811Z level=info msg="Live Push Gateway initialization"
logger=ticker t=2022-11-06T08:34:29.831886397Z level=info msg=starting first_tick=2022-11-06T08:34:30Z
logger=infra.usagestats.collector t=2022-11-06T08:34:30.262100569Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2
logger=provisioning.datasources t=2022-11-06T08:34:30.287640134Z level=info msg="inserting datasource from configuration " name=prometheus uid=P1809F7CD0C75ACF3
logger=provisioning.plugins t=2022-11-06T08:34:30.372898164Z level=error msg="Failed to read plugin provisioning files from directory" path=/etc/grafana/provisioning/plugins error="open /etc/grafana/provisioning/plugins: no such file or directory"
logger=provisioning.notifiers t=2022-11-06T08:34:30.372924996Z level=error msg="Can't read alert notification provisioning files from directory" path=/etc/grafana/provisioning/notifiers error="open /etc/grafana/provisioning/notifiers: no such file or directory"
logger=provisioning.alerting t=2022-11-06T08:34:30.372939446Z level=error msg="can't read alerting provisioning files from directory" path=/etc/grafana/provisioning/alerting error="open /etc/grafana/provisioning/alerting: no such file or directory"
logger=provisioning.alerting t=2022-11-06T08:34:30.372943764Z level=info msg="starting to provision alerting"
logger=provisioning.alerting t=2022-11-06T08:34:30.372949223Z level=info msg="finished to provision alerting"
logger=grafanaStorageLogger t=2022-11-06T08:34:30.374361916Z level=info msg="storage starting"
logger=http.server t=2022-11-06T08:34:30.375650267Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=http subUrl= socket=
logger=ngalert t=2022-11-06T08:34:30.454885019Z level=info msg="warming cache for startup"
logger=ngalert.multiorg.alertmanager t=2022-11-06T08:34:30.456183186Z level=info msg="starting MultiOrg Alertmanager"
logger=sqlstore t=2022-11-06T08:49:29.769781229Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0
EricCaoHZ commented 1 year ago

It turned out to use networkPolicy, no wonder it has been unable to open