Unified Alerting Migration: invalid character in string escape code

tonypowa commented 8 months ago

What happened?

We have migrated from Grafana 8.4.2 to 9.5.5 in multiple steps, and one of the steps is to migrate from legacy alerting on 9.5.5 to Unified Alerting.

After updating the grafana.ini config to disable alerting and enable unified alerting, we see an error during the move dashboards to unified alerting migration that describes an invalid character in the escape code.

Configuration via helm values.yaml file:


grafana:

...

 alerting:

 enabled: false

 unified_alerting:

 enabled: true

Log output:


logger=migrator t=2023-12-19T17:51:58.89066255Z level=info msg="Executing migration" id="clear migration entry \"remove unified alerting data\""

logger=migrator t=2023-12-19T17:51:58.932305811Z level=info msg="Executing migration" id="move dashboard alerts to unified alerting"

logger=migrator t=2023-12-19T17:51:59.050119836Z level=error msg="Executing migration failed" id="move dashboard alerts to unified alerting" error="invalid character ':' in string escape code"

logger=migrator t=2023-12-19T17:51:59.050151123Z level=error msg="Exec failed" error="invalid character ':' in string escape code" sql="code migration"

Error: ✗ migration failed (id = move dashboard alerts to unified alerting): invalid character ':' in string escape code

This appears to be the same behavior as #65097 with a different character

What did you expect to happen?

I expected the dashboards to be migrated to unified alerting from legacy alerting without any errors

Did this work before?

This is the first time that we have attempted the migration.

How do we reproduce it?

Run an instance of Grafana 8.4.2 using sqlitedb to store legacy alerts
Create some dashboards with ":" in the messaage
Upgrade the 8.4.2 instance to 9.5.5
Export the sqlite database file from the 8.4.2 instance
Use grafana/database-migrator to convert the sqlite database to a format that MySQL can consume
Configure a MySQL instance to store data for Grafana 9.5.5
Import the file created in step 5 to the MySQL instance
Configure Grafana 9.5.5 to use the MySQL instance with legacy alerting
Start Grafana 9.5.5 and ensure that all the migrations complete successfully
Stop Grafana 9.5.5 and update grafana.ini to disable alerting and enable unified_alerting
Start Grafana 9.5.5 and watch the logs for errors

Is the bug inside a dashboard panel?

No

Environment (with versions)?

Grafana: 8.4.2 upgraded to 9.5.5

OS: Linux

Browser: Arc

Grafana platform?

Kubernetes

Datasource(s)?

No response

tonypowa commented 8 months ago

This issue is a test copy of an issue in another repo. Original issue: https://github.com/grafana/grafana/issues/#79720

moxious commented 8 months ago

Hi @tonypowa, it sounds like this issue might be related to the migration process between Grafana's legacy alerting to Unified Alerting. Since this involves a core functionality update in Grafana, the best project to associate this issue with would be the Alerting project. The team associated with it will have more context on the migrations from legacy systems and how to address this error during the migration process.

{
 "title": "Unified Alerting Migration: invalid character in string escape code",
 "labels": ["bug", "unified-alerting"],
 "assignees": [],
 "projects": ["Alerting project"]
}

moxious commented 8 months ago

Elaboration:

Hello @tonypowa, thank you for your detailed report on the issue you've encountered during the migration to Unified Alerting in Grafana. It appears you have provided a comprehensive account of the steps you followed, which is very helpful. However, there are a few additional details that would help to further diagnose the issue at hand:

Could you please provide the specific section or format of the dashboard JSON that includes the : character where you mentioned adding it in the message? This might help us to reproduce the issue more accurately.
You mentioned using a custom tool grafana/database-migrator for database conversion. Would you be able to supply the exact command or set of options used with this tool?
When you encountered the error, were any specific dashboards or alerts failing to migrate, or was this a general failure affecting all dashboard migrations to Unified Alerting?
If possible, can you try to isolate and reproduce the error with a minimal example or on a test dashboard with a similar configuration, and share the detailed steps?
Have you tried manually escaping or altering the : characters in your alerts to see if the migration succeeds without these specific characters?
Additionally, could you verify that the MySQL database charset and collation settings are compatible with the migrated data from SQLite?

Your cooperation is invaluable in resolving this situation. If you can provide any screenshots, error logs (with extended logging, if possible), or other relevant details, that would be very beneficial.

Thank you for your contribution to the Grafana community, and we look forward to your response to address this issue effectively!

moxious commented 8 months ago

Summary: After migrating from Grafana 8.4.2 to 9.5.5 and attempting to switch from legacy alerting to Unified Alerting, a user encounters a migration error related to an invalid character in a string escape code, which seems similar to a previously reported issue #65097.

moxious / triage