Grafana fails to start after upgrade to v10.4.12

rda0 commented 1 week ago

What happened?

Grafana fails to start after upgrade to v10.4.12.

Error with db migrations (Executing migration failed, You have an error in your SQL syntax;):

logger=settings t=2024-11-12T08:01:36.854037308+01:00 level=info msg="Starting Grafana" version=10.4.12 commit=9ed35a1d6fe1222d5e054237db735b5bac5e3712 branch=HEAD compiled=2024-11-12T08:01:36+01:00 
logger=settings t=2024-11-12T08:01:36.854709903+01:00 level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 
logger=settings t=2024-11-12T08:01:36.854823221+01:00 level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 
logger=settings t=2024-11-12T08:01:36.854923131+01:00 level=info msg="Config overridden from command line" arg="default.paths.data=/var/lib/grafana" 
logger=settings t=2024-11-12T08:01:36.855028392+01:00 level=info msg="Config overridden from command line" arg="default.paths.logs=/var/log/grafana" 
logger=settings t=2024-11-12T08:01:36.855132449+01:00 level=info msg="Config overridden from command line" arg="default.paths.plugins=/var/lib/grafana/plugins" 
logger=settings t=2024-11-12T08:01:36.855226524+01:00 level=info msg="Config overridden from command line" arg="default.paths.provisioning=/etc/grafana/provisioning" 
logger=settings t=2024-11-12T08:01:36.855321346+01:00 level=info msg=Target target=[all] 
logger=settings t=2024-11-12T08:01:36.855423504+01:00 level=info msg="Path Home" path=/usr/share/grafana 
logger=settings t=2024-11-12T08:01:36.855513462+01:00 level=info msg="Path Data" path=/var/lib/grafana 
logger=settings t=2024-11-12T08:01:36.855595858+01:00 level=info msg="Path Logs" path=/var/log/grafana 
logger=settings t=2024-11-12T08:01:36.855684739+01:00 level=info msg="Path Plugins" path=/var/lib/grafana/plugins 
logger=settings t=2024-11-12T08:01:36.85577342+01:00 level=info msg="Path Provisioning" path=/etc/grafana/provisioning 
logger=settings t=2024-11-12T08:01:36.855861094+01:00 level=info msg="App mode production" 
logger=sqlstore t=2024-11-12T08:01:36.856342345+01:00 level=info msg="Connecting to DB" dbtype=mysql 
logger=migrator t=2024-11-12T08:01:36.860221791+01:00 level=info msg="Starting DB migrations" 
logger=migrator t=2024-11-12T08:01:36.888621319+01:00 level=info msg="Executing migration" id="delete orphaned service account permissions" 
logger=migrator t=2024-11-12T08:01:36.896375266+01:00 level=error msg="Executing migration failed" id="delete orphaned service account permissions" error="failed to delete orphaned service accounts: Error 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 'AS p WHERE p.kind = 'serviceaccounts' AND p.identifier IN(?,?,?)' at line 1" duration=7.754554ms 
logger=migrator t=2024-11-12T08:01:36.89674722+01:00 level=error msg="Exec failed" error="failed to delete orphaned service accounts: Error 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 'AS p WHERE p.kind = 'serviceaccounts' AND p.identifier IN(?,?,?)' at line 1" sql="code migration"

I was able to downgrade to v10.4.10 again.

What did you expect to happen?

No error

Did this work before?

10.4.10: Yes
10.4.11: No, see #95219 and https://github.com/grafana/grafana/issues/95219#issuecomment-2436271539 why it is probably still broken.

How do we reproduce it?

Upgrade
Restart

Is the bug inside a dashboard panel?

No response

Environment (with versions)?

Grafana: 10.4.12 OS: Debian GNU/Linux 12 (bookworm) Browser: FF

Grafana platform?

A package manager (APT, YUM, BREW, etc.)

Datasource(s)?

No response

Jguer commented 1 week ago

Hey @rda0 , what database and database version are you running?

rda0 commented 1 week ago

Hey @rda0 , what database and database version are you running?

Still the same as in the previous issue https://github.com/grafana/grafana/issues/95219#issuecomment-2431348901

rda0 commented 1 week ago

@Jguer Consider reopening #95219. I do not have permission to reopen.

melroy89 commented 1 week ago

I'm personally not using v10.4 (but on v11.2.2, unable to upgrade further).

v10.4.12 seems to be just 1 week old, while the fix (PR) was merged 3 weeks ago: https://github.com/grafana/grafana/pull/95228

Looking at the output:

AS p WHERE p.kind = 'serviceaccounts' AND p.identifier IN(?,?,?)' at line 1" sql="code migration"

You are either not running v10.4.12. But I don't believe you are lying.

OR:

Grafana is trying to execute a OLDER migration from an older version (eg. 10.4.11 for example). No idea how to fix that.

OR...

v10.4.12 doesn't contain the fix (see PR above) somehow and the devs made a mistake.

Jguer commented 1 week ago

https://github.com/grafana/grafana/commits/v10.4.12/ https://github.com/grafana/grafana/commits/v10.4.x

10.4.12 doesn't include this fix, this probably means it did not make into this particular release cycle

gestgithub commented 1 week ago

Why not (as a workaround) execute the orphaned permission removal statements manually? I've looked at the code at orphaned.go:35 and at orphaned.go:89 and it seems in my case it would be:

MariaDB [grafana]> SELECT DISTINCT p.identifier FROM permission AS p WHERE p.kind  = 'serviceaccounts' AND NOT p.identifier = '*';
+------------+
| identifier |
+------------+
| 365        |
+------------+

and then just delete this with correct syntax?

DELETE FROM permission WHERE kind = 'serviceaccounts' AND identifier IN(365)

And then the migration and startup should work I believe? Or am I missing something and I shouldn't do this?

greenmotion commented 3 days ago

I upgraded to 11.3.1 this morning and this issue has been fixed. thumbup

grafana / grafana