carlosedp / cluster-monitoring

Cluster monitoring stack for clusters based on Prometheus Operator
MIT License
740 stars 200 forks source link

Grafana Error: The same uid is used more than once #75

Closed mkmojo closed 4 years ago

mkmojo commented 4 years ago

Dear Team, Thanks for putting this together. This helped a lot for beginners like me.

I deployed the monitoring to namespace monitoring as described in this Youtebe tutorial by @geerlingguy.

However, the Grafana is not very stable, I can sometimes log into the dashboard but will be kicked out very with indeterministic interval. So overall the dashboard is only available sometimes.

When I checked the log by looking at the grafana Pod, here is the log.

t=2020-07-09T04:56:24+0000 lvl=eror msg="Could not load alerts" logger=alerting.ruleReader error="database is locked"
t=2020-07-09T04:56:31+0000 lvl=eror msg="Alert Rule Result Error" logger=alerting.evalContext ruleId=4 name="CPU Temperature alert" error="Could not find datasource database is locked" changing state to=alerting
t=2020-07-09T04:56:31+0000 lvl=info msg="New state change" logger=alerting.resultHandler ruleId=4 newState=alerting prev state=ok
t=2020-07-09T04:56:31+0000 lvl=eror msg="Alert Rule Result Error" logger=alerting.evalContext ruleId=6 name="CPU Usage alert" error="Could not find datasource database is locked" changing state to=alerting
t=2020-07-09T04:56:31+0000 lvl=info msg="New state change" logger=alerting.resultHandler ruleId=6 newState=alerting prev state=ok
t=2020-07-09T04:56:31+0000 lvl=eror msg="Alert Rule Result Error" logger=alerting.evalContext ruleId=10 name="CPU Usage alert" error="Could not find datasource database is locked" changing state to=alerting
t=2020-07-09T04:56:31+0000 lvl=info msg="New state change" logger=alerting.resultHandler ruleId=10 newState=alerting prev state=ok
t=2020-07-09T04:56:34+0000 lvl=eror msg="Could not load alerts" logger=alerting.ruleReader error="database is locked"
t=2020-07-09T04:56:36+0000 lvl=eror msg="Failed to save state" logger=alerting.resultHandler error="database is locked"
t=2020-07-09T04:56:36+0000 lvl=eror msg="Failed to save state" logger=alerting.resultHandler error="database is locked"
t=2020-07-09T04:56:36+0000 lvl=info msg="Database locked, sleeping then retrying" logger=sqlstore error="database is locked" retry=0
t=2020-07-09T04:56:41+0000 lvl=eror msg="Failed to save annotation for new alert state" logger=alerting.resultHandler error="database is locked"
t=2020-07-09T04:56:41+0000 lvl=eror msg="Failed to save annotation for new alert state" logger=alerting.resultHandler error="database is locked"
t=2020-07-09T04:56:41+0000 lvl=info msg="Database locked, sleeping then retrying" logger=sqlstore error="database is locked" retry=1
t=2020-07-09T04:56:42+0000 lvl=eror msg="the same 'uid' is used more than once" logger=provisioning.dashboard type=file name=0 uid=82pBZCmRkasd provider=0
t=2020-07-09T04:56:46+0000 lvl=eror msg="Failed to get alert notifiers" logger=alerting.notifier error="database is locked"
t=2020-07-09T04:56:46+0000 lvl=eror msg="handler.notifier.SendIfNeeded failed" logger=alerting.resultHandler err="database is locked"
t=2020-07-09T04:56:47+0000 lvl=eror msg="failed to search for dashboards" logger=provisioning.dashboard type=file name=0 error="database is locked"
t=2020-07-09T04:56:53+0000 lvl=eror msg="failed to search for dashboards" logger=provisioning.dashboard type=file name=0 error="database is locked"
t=2020-07-09T04:57:03+0000 lvl=eror msg="failed to search for dashboards" logger=provisioning.dashboard type=file name=0 error="database is locked"
t=2020-07-09T04:57:16+0000 lvl=eror msg="Failed to get system stats" logger=metrics error="database is locked"
t=2020-07-09T04:57:18+0000 lvl=eror msg="the same 'uid' is used more than once" logger=provisioning.dashboard type=file name=0 uid=82pBZCmRkasd provider=0
t=2020-07-09T04:57:33+0000 lvl=eror msg="the same 'uid' is used more than once" logger=provisioning.dashboard type=file name=0 uid=82pBZCmRkasd provider=0
t=2020-07-09T04:57:41+0000 lvl=eror msg="the same 'uid' is used more than once" logger=provisioning.dashboard type=file name=0 uid=82pBZCmRkasd provider=0
t=2020-07-09T04:57:49+0000 lvl=eror msg="the same 'uid' is used more than once" logger=provisioning.dashboard type=file name=0 uid=82pBZCmRkasd provider=0
t=2020-07-09T04:58:07+0000 lvl=eror msg="the same 'uid' is used more than once" logger=provisioning.dashboard type=file name=0 uid=82pBZCmRkasd provider=0
t=2020-07-09T04:58:13+0000 lvl=eror msg="failed to run garbage collect" logger=remotecache.database error="database is locked"
t=2020-07-09T04:58:13+0000 lvl=eror msg="Failed to delete expired snapshots" logger=cleanup error="database is locked"
t=2020-07-09T04:58:14+0000 lvl=eror msg="Could not load alerts" logger=alerting.ruleReader error="database is locked"
t=2020-07-09T04:58:17+0000 lvl=eror msg="the same 'uid' is used more than once" logger=provisioning.dashboard type=file name=0 uid=82pBZCmRkasd provider=0
t=2020-07-09T04:58:25+0000 lvl=eror msg="the same 'uid' is used more than once" logger=provisioning.dashboard type=file name=0 uid=82pBZCmRkasd provider=0
t=2020-07-09T04:58:46+0000 lvl=eror msg="the same 'uid' is used more than once" logger=provisioning.dashboard type=file name=0 uid=82pBZCmRkasd provider=0
t=2020-07-09T04:58:54+0000 lvl=eror msg="the same 'uid' is used more than once" logger=provisioning.dashboard type=file name=0 uid=82pBZCmRkasd provider=0
t=2020-07-09T04:59:13+0000 lvl=eror msg="the same 'uid' is used more than once" logger=provisioning.dashboard type=file name=0 uid=82pBZCmRkasd provider=0
t=2020-07-09T04:59:16+0000 lvl=eror msg="Failed to get system stats" logger=metrics error="database is locked"
t=2020-07-09T04:59:22+0000 lvl=eror msg="the same 'uid' is used more than once" logger=provisioning.dashboard type=file name=0 uid=82pBZCmRkasd provider=0
t=2020-07-09T04:59:32+0000 lvl=eror msg="the same 'uid' is used more than once" logger=provisioning.dashboard type=file name=0 uid=82pBZCmRkasd provider=0

It complains a lot about uid is used more than once. Has anyone run into similar issues?

mkmojo commented 4 years ago

Is there anyway I can clear grafana's user database so I can reset the admin account?

carlosedp commented 4 years ago

Better to delete any PV and recreate. Other than that you need to check the Grafana docs.