Grafana not accessible after update to the latest image

[borj]

Grafana container seems to be running but the interface is not accessible anymore. Container seems to be constantly restarting. Image currently in use is grafana/grafana@sha256:7dcfd5520385ad79cd7a4af24323752e68fd0d660a627a335f168d47ff643cae Container sample log follows:

created by github.com/grafana/grafana/pkg/services/provisioning/dashboards.(*Provisioner).PollChanges in goroutine 251 github.com/grafana/grafana/pkg/services/provisioning/dashboards/dashboard.go:105 +0x144 goroutine 628 [select]: runtime.gopark(0x112170c, 0x0, 0x9, 0x18, 0x1) runtime/proc.go:398 +0x104 fp=0xb73bed8 sp=0xb73bec4 pc=0x43e3958 runtime.selectgo(0xb73bfb4, 0xb73bf98, 0x0, 0x0, 0x3, 0x1) runtime/select.go:327 +0xaf0 fp=0xb73bf74 sp=0xb73bed8 pc=0x43f76d4 database/sql.(*Rows).awaitDone(0xc1b0af0, {0x191fcac, 0xc33fe90}, {0x0, 0x0}, {0x191fccc, 0xc34ab40}) database/sql/sql.go:2962 +0xcc fp=0xb73bfcc sp=0xb73bf74 pc=0x4c8ad40 database/sql.(*Rows).initContextClose.func1() database/sql/sql.go:2949 +0x58 fp=0xb73bfec sp=0xb73bfcc pc=0x4c8ac48 runtime.goexit() runtime/asm_arm.s:859 +0x4 fp=0xb73bfec sp=0xb73bfec pc=0x441d4c0 created by database/sql.(*Rows).initContextClose in goroutine 484 database/sql/sql.go:2949 +0x1ac trap 0x6 error 0x0 oldmask 0x0 r0 0x0 r1 0x0 r2 0x1 r3 0x1 r4 0xa6bd0ad4 r5 0xc320e94 r6 0xb349648  r7 0x10 r8 0x94a0000 r9 0x0 r10 0x94a2100 fp 0xad0c4aa8 ip 0x0 sp 0xad0c01dc lr 0x85ebf30 pc 0x8617b1c cpsr 0x60000010 fault 0x0 ✔ Downloaded and extracted flant-statusmap-panel v0.5.1 zip successfully to /var/lib/grafana/plugins/flant-statusmap-panel Please restart Grafana after installing or removing plugins. Refer to Grafana documentation for instructions if necessary. ✔ Downloaded and extracted ae3e-plotly-panel v0.5.0 zip successfully to /var/lib/grafana/plugins/ae3e-plotly-panel Please restart Grafana after installing or removing plugins. Refer to Grafana documentation for instructions if necessary. logger=settings t=2023-08-24T20:26:01.55536336Z level=info msg="Starting Grafana" version=10.1.0 commit=ff85ec33c5 branch=HEAD compiled=2023-08-23T20:31:27Z logger=settings t=2023-08-24T20:26:01.556517943Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini logger=settings t=2023-08-24T20:26:01.556575868Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini logger=settings t=2023-08-24T20:26:01.556845032Z level=info msg="Config overridden from command line" arg="default.paths.data=/var/lib/grafana" logger=settings t=2023-08-24T20:26:01.556884773Z level=info msg="Config overridden from command line" arg="default.paths.logs=/var/log/grafana" logger=settings t=2023-08-24T20:26:01.556939346Z level=info msg="Config overridden from command line" arg="default.paths.plugins=/var/lib/grafana/plugins" logger=settings t=2023-08-24T20:26:01.557036586Z level=info msg="Config overridden from command line" arg="default.paths.provisioning=/etc/grafana/provisioning" logger=settings t=2023-08-24T20:26:01.557236825Z level=info msg="Config overridden from command line" arg="default.log.mode=console" logger=settings t=2023-08-24T20:26:01.557291843Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" logger=settings t=2023-08-24T20:26:01.557380565Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" logger=settings t=2023-08-24T20:26:01.557544452Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" logger=settings t=2023-08-24T20:26:01.557597507Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" logger=settings t=2023-08-24T20:26:01.55769134Z level=info msg="Config overridden from Environment variable" var="GF_SECURITY_ADMIN_PASSWORD=*********" logger=settings t=2023-08-24T20:26:01.557853708Z level=info msg="Config overridden from Environment variable" var="GF_USERS_ALLOW_SIGN_UP=false" logger=settings t=2023-08-24T20:26:01.557908115Z level=info msg=Target target=[all] logger=settings t=2023-08-24T20:26:01.558123669Z level=info msg="Path Home" path=/usr/share/grafana logger=settings t=2023-08-24T20:26:01.558178761Z level=info msg="Path Data" path=/var/lib/grafana logger=settings t=2023-08-24T20:26:01.558337019Z level=info msg="Path Logs" path=/var/log/grafana logger=settings t=2023-08-24T20:26:01.558390407Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins logger=settings t=2023-08-24T20:26:01.558555036Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning logger=settings t=2023-08-24T20:26:01.558607442Z level=info msg="App mode production" logger=sqlstore t=2023-08-24T20:26:01.559911523Z level=info msg="Connecting to DB" dbtype=sqlite3 logger=migrator t=2023-08-24T20:26:01.714724139Z level=info msg="Starting DB migrations" logger=migrator t=2023-08-24T20:26:01.759611774Z level=info msg="migrations completed" performed=0 skipped=493 duration=4.300091ms logger=secrets t=2023-08-24T20:26:01.763494962Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 logger=local.finder t=2023-08-24T20:26:02.034609555Z level=warn msg="Skipping finding plugins as directory does not exist" path=/usr/share/grafana/plugins-bundled logger=plugin.loader t=2023-08-24T20:26:04.099925509Z level=info msg="Plugin registered" pluginID=ae3e-plotly-panel logger=plugin.loader t=2023-08-24T20:26:04.10004123Z level=info msg="Plugin registered" pluginID=flant-statusmap-panel logger=query_data t=2023-08-24T20:26:04.111815551Z level=info msg="Query Service initialization" logger=live.push_http t=2023-08-24T20:26:04.14227876Z level=info msg="Live Push Gateway initialization" logger=ngalert t=2023-08-24T20:26:04.165540774Z level=warn msg="Unexpected number of rows updating alert configuration history" rows=0 org=1 hash=8c409350c88d78d2ee938448449e628d logger=infra.usagestats.collector t=2023-08-24T20:26:15.619557531Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 logger=modules t=2023-08-24T20:26:15.620225951Z level=info msg=initialising module=secret-migrator logger=modules t=2023-08-24T20:26:15.620271969Z level=info msg=initialising module=http-server logger=http.server t=2023-08-24T20:26:15.630427545Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=http subUrl= socket= logger=modules t=2023-08-24T20:26:15.662017744Z level=info msg=initialising module=provisioning logger=provisioning.datasources t=2023-08-24T20:26:15.704206348Z level=info msg="deleted correlations based on configuration" ds_name=prometheus logger=provisioning.datasources t=2023-08-24T20:26:15.704419179Z level=info msg="deleted datasource based on configuration" name=prometheus logger=provisioning.datasources t=2023-08-24T20:26:15.706030869Z level=info msg="inserting datasource from configuration " name=prometheus uid=P1809F7CD0C75ACF3 logger=provisioning.plugins t=2023-08-24T20:26:15.951379174Z level=error msg="Failed to read plugin provisioning files from directory" path=/etc/grafana/provisioning/plugins error="open /etc/grafana/provisioning/plugins: no such file or directory" logger=provisioning.notifiers t=2023-08-24T20:26:15.95168019Z level=error msg="Can't read alert notification provisioning files from directory" path=/etc/grafana/provisioning/notifiers error="open /etc/grafana/provisioning/notifiers: no such file or directory" logger=provisioning.alerting t=2023-08-24T20:26:15.951926947Z level=error msg="can't read alerting provisioning files from directory" path=/etc/grafana/provisioning/alerting error="open /etc/grafana/provisioning/alerting: no such file or directory" logger=provisioning.alerting t=2023-08-24T20:26:15.952078575Z level=info msg="starting to provision alerting" logger=provisioning.alerting t=2023-08-24T20:26:15.952206463Z level=info msg="finished to provision alerting" logger=modules t=2023-08-24T20:26:15.952500572Z level=info msg=initialising module=background-services logger=ngalert.state.manager t=2023-08-24T20:26:15.952931957Z level=info msg="Warming state cache for startup" logger=modules t=2023-08-24T20:26:15.953776357Z level=info msg="All modules healthy" modules="[background-services http-server secret-migrator provisioning]" logger=grafanaStorageLogger t=2023-08-24T20:26:15.95422413Z level=info msg="storage starting" logger=ngalert.state.manager t=2023-08-24T20:26:15.954474739Z level=info msg="State cache has been initialized" states=0 duration=1.536894ms logger=ngalert.scheduler t=2023-08-24T20:26:15.958683646Z level=info msg="Starting scheduler" tickInterval=10s logger=ticker t=2023-08-24T20:26:15.961880691Z level=info msg=starting first_tick=2023-08-24T20:26:20Z logger=ngalert.multiorg.alertmanager t=2023-08-24T20:26:15.95973684Z level=info msg="Starting MultiOrg Alertmanager" logger=grafana.update.checker t=2023-08-24T20:26:16.089041462Z level=info msg="Update check succeeded" duration=135.67836ms logger=plugins.update.checker t=2023-08-24T20:26:16.130387166Z level=info msg="Update check succeeded" duration=175.813371ms

[codefreak558]

I also seem to be getting this problem however I get it whenever I try logging in I just get login failed then that same error code.

I am going to try just setting the version for Grafana back one and see if that lets me gain access to Grafana again.

[kalju]

Same here - Unknown error while trying to log in. And the same error in log file, container seems to be in restarting loop due to error.

[codefreak558]

What I've done in the mean time so that I can login to grafana is set grafana/grafana too grafana/grafana:10.0.3. Until we have a resolution for whatever is the problem with the latest version.

[borj]

in the mean time so that I can login to grafana is set grafana/grafana too grafana/grafana:10.0.3.

Works for me too.

[jerzerisz]

This seems related to a bug that has been reported on grafana's repo: https://github.com/grafana/grafana/issues/73781

recommended workaround is to use 10.0.3 tag

Unfortunately i can't reproduce from a pi 4b 4gb 64bit rasppberry pi os.

[Niriuqq]

Hey Guys I have the same problem. But I am quite new with Grafana, Docker etc. Where can I adjust the version so that it works again?

[karelkryda]

@Niriuqq replace grafana/grafana:latest with grafana/grafana:10.0.3. If you're not using manager like Portainer, you need to stop and remove your Grafana container and then create new one again with image tag I mentioned above.

[Justin31415]

For those who find this and are not familiar with docker. I found this line in the code here: https://github.com/geerlingguy/internet-pi/blob/79253709150d0e15e39b79beceed46c0de4059ea/templates/docker-compose.yml.j2#L59

I was able to solve this over SSH by the following commands: nano ./templates/docker-compose.yml.j2 going to line 59 (you can jump there with CTRL SHIFT - key combo and entering 59) and changing grafana/grafana:latest with grafana/grafana:10.0.3

restart the Pi and log in

[github-actions[bot]]

This issue has been marked 'stale' due to lack of recent activity. If there is no further activity, the issue will be closed in another 30 days. Thank you for your contribution!

Please read this blog post to see the reasons why I mark issues as stale.

[github-actions[bot]]

This issue has been closed due to inactivity. If you feel this is in error, please reopen the issue or file a new issue with the relevant details.