Facilitate migration from Scalyr plugin (< v3) to Dataset plugin (>= v3)

[ReinAchten-TomTom]

We have quite some data source configured in our Grafana instance that currently are using the Scalyr plugin. These data sources are used in a lot of graphs in a lot of dashboards. Since the Dataset plugin is backwards incompatible with the Scalyr plugin at least in terms of naming, this implies manual effort in migrating data sources and graphs to the Dataset plugin. Depending on the level of backwards incompatibility that effort might turn out quite substantial.

This is inconvenient for us to say the least. Can you please take a look at providing a migration path from Scalyr plugin to Dataset plugin? Ideally, we'd be able to just migrate to the newer version of the plugin without having to manually change things on our side.

fyi: @MarkoTechno

[ReinAchten-TomTom]

I've just checked if I could use the Dataset plugin as a drop-in replacement by adding a datasource for a specific team account and replacing the data source in the dashboard json, but this does not work for various reasons (i.a. variables are named differently, e.g. filter vs expression, values are different, e.g. Standard Query vs Standard for the query type, etc). Based on this small experiment, it seems the effort involved on our side to migrate from Scalyr plugin to Dataset plugin is quite substantial. Can this please be looked into with priority?

[jmakar-s1]

Sure I can look into this. Is it possible for you to export some old dashboards as json and email them to me? Also are you using Grafana 8.2.x or a newer version? (8.2.x to 8.3.x had a small change in its datasource format)

Most likely will make a script available to convert from the old format to the new one.

[jmakar-s1]

Here's the first implementation of the migration script: https://github.com/scalyr/scalyr-grafana-datasource-plugin/blob/master/migrate.py

[ReinAchten-TomTom]

Hi @joseph-makar , thanks for following up on this! A migration script that converts dashboard json files would surely help, but that would still leave us with quite some manual effort, i.a.:

• recreate the data sources using the Dataset plugin
• replace the data source uids in the dashboards (there's quite some)
• run the migration script for each dashboard

It would be significantly easier if this would be dealt with by the plugin completely. I imagine this will similarly affect other Dataset customers that are using the Scalyr plugin for Grafana now and need to migrate to the Dataset plugin. It seems backwards compatibility was not really considered when rewriting the plugin. Can you take a look at revisiting this? Ideally, we'd just be able to update the plugin to a newer version.

[jmakar-s1]

There have been some updates to the migration script so that it automates everything except for recreating the datasources.

So the current migration process is:

• install the new plugin
• recreate the datasources
• make a mapping from the old datasources to the new datasources in a config.json file:

  {
  "datasource-uid-mapping": {
  // the datasource uids can be found at Configuration, Data Sources, then the string at the end of the url
  "<scalyr-plugin-datasource-uid1>": "dataset-plugin-datasource-uid1>",
  ...
  }
  }

• execute the migration script with the required arguments (default is 127.0.0.1:3000 and the default admin user/pass):
  • ./migrate.py -g <grafana-server-ip>:<grafana-server-port> -c <username>:<password> config.json
  • can also optionally specify the dashboard uids at the end of the command to only process those: ./migrate.py ... <dashboard-uid1> <dashboard-uid2> ...
  • can get the dashboard uids from the urls similarly to how it can be found for datasources
  • otherwise the default is to process all dashboards
  • each dashboard that uses the old scalyr plugin will cause a new dashboard to be created with the title prefixed by "Migrated: "
  • this script also works with Grafana cloud accounts: ./migrate.py -g grafana.com:443 -t -c api_key:<api-key> config.json [<optional-dashboard-uids>]
  • documentation for the migrate.py script can be found at the top of the file

Hopefully this will be sufficient to help support migration.

[jmakar-s1]

@ReinAchten-TomTom Was this sufficient? Or did you work out an alternative?

[lutzwillek-tomtom]

Hi @jmakar-s1 - sorry for the very late response, the migration got delayed on our end a little bit... I tried the script and it works fine. (there are other most likely unrelated issues however...)

In any case, I think this issue can be closed, thank you very much!

[jmakar-s1]

No problem will close it now, let us know what we can do to help