wbh1 / grafana-sqlite-to-postgres

Grafana SQLite to Postgres Database Migrator
MIT License
97 stars 26 forks source link

Does not work for Grafana 8 #21

Closed GataullinRR closed 2 years ago

GataullinRR commented 2 years ago

Grafana Version: 8.1.5 Output:

INFO[2021-10-16T14:54:09Z] 📁 SQLlite file: /grafana.db                  
INFO[2021-10-16T14:54:09Z] 📁 Dump directory: /tmp                       
INFO[2021-10-16T14:54:09Z] ✅ sqlite3 command exists                     
INFO[2021-10-16T14:54:11Z] ✅ sqlite3 database dumped to /tmp/grafana.sql 
INFO[2021-10-16T14:54:18Z] ✅ CREATE statements removed from dump file   
INFO[2021-10-16T14:54:48Z] ✅ sqlite3 dump sanitized                     
INFO[2021-10-16T14:55:02Z] ✅ migration_log statements removed           
INFO[2021-10-16T14:55:03Z] ✅ char keyword transformed                   
INFO[2021-10-16T14:55:09Z] ✅ hex-encoded data decoded                   
FATAL[2021-10-16T14:55:21Z] ❌ pq: column "default" is of type boolean but expression is of type integer INSERT INTO "alert_configuration" VALUES(1,replace('\n{\n   "alertmanager_config": {\n      "route": {\n            "receiver": "grafana-default-email"\n       },\n        "receivers": [{\n           "name": "grafana-default-email",\n          "grafana_managed_receiver_configs": [{\n                "uid": "",\n                "name": "email receiver",\n             "type": "email",\n              "isDefault": true,\n                "settings": {\n                 "addresses": "<example@email.com>"\n                }\n         }]\n        }]\n    }\n}\n','\n',chr(10)),'v1',1631691122,1) - failed to import dump file to Postgres. 
GataullinRR commented 2 years ago

Do not have experience Go enough to create Pull Request, but there is handmade Python script which will fix statements from sqlite dump file so that it could be imported into postgres:

import re
import os

with open ("/tmp/ctmp/grafana.sql", "r") as broken:
        data=broken.read()

fixed = re.sub(r"(INSERT INTO .alert_configuration. VALUES.*?)([0])(\);.*)", r"\1FALSE\3", data)
fixed = re.sub(r"(INSERT INTO .alert_configuration. VALUES.*?)([1])(\);.*)", r"\1TRUE\3", fixed)

os.remove("/tmp/fixed.grafana.sql")
with open("/tmp/fixed.grafana.sql", "w") as fixed_file:
        fixed_file.write(fixed)
sandangel commented 2 years ago
import re
import os

with open ("./unfix.grafana.sql", "r") as broken:
        data=broken.read()

fixed = re.sub(r"(INSERT INTO .alert_configuration. VALUES[\S\s]*?)(,[0])(\);)", r"\1,FALSE\3", data)
fixed = re.sub(r"(INSERT INTO .alert_configuration. VALUES[\S\s]*?)(,[1])(\);)", r"\1,TRUE\3", fixed)

with open("./grafana.sql", "w") as fixed_file:
        fixed_file.write(fixed)

I also ran into this issue, this is my script in case the values span multi lines. We can use this script on the generated grafana.sql, comment out all the generating grafana.sql code in the main.go and rebuild the docker image, put fixed grafana.sql in the correct place and run

wbh1 commented 2 years ago

@sandangel @GataullinRR thank you both for bringing this to my attention. I'm on paternity leave from my job currently and haven't been spending much time online as a result. Once I'm back to work, I'll pick this up and try to implement a fix.

wbh1 commented 2 years ago

Closed by #24