v1.2.7 has broken our production site

[RobHumphris]

We've been tracing an issue since our production instance stopped rendering data in reports in production that are working on staging. Everything is exactly the same between two environments (Docker DIGESTs match, environment variables are exactly the same). The one difference is that staging (working) is using 1.2.6 and live is using 1.2.7 this will have come about when the instance cluster for live created new instances to handle 9am Monday.

In short large amounts of data are not being displayed even though we can see that it is being returned by the API.

It appears that the 1.2.7 is sending more information, which is good as our times filters may start to work, but that the dashboards are not updating when the data is returned, my suspicion is that now refId is not being passed something is not triggering the panel rendering.

Digging into the issue the key difference is that the working instance (1.2.6) POST request payload looks like:

{
  "queries": [
    {
      "refId": "0.7387678513218177",
      "datasource": {
        "type": "grafana-bigquery-datasource",
        "uid": "-61os9l4z"
      },
      "rawSql": "SELECT COALESCE(XYZ) AS html FROM XYZ hi JOIN XYZ ON XYZ  WHERE XYZ",
      "format": 1,
      "connectionArgs": {
        "location": "europe-west2"
      },
      "datasourceId": 105
    }
  ]
}

And the 1.2.7:

{
  "queries": [
    {
      "datasource": {
        "type": "grafana-bigquery-datasource",
        "uid": "-61os9l4z"
      },
      "rawSql": "SELECT COALESCE(XYZ) AS html FROM XYZ hi JOIN XYZ ON XYZ  WHERE XYZ",
      "format": 1,
      "connectionArgs": {
        "location": "europe-west2"
      },
      "datasourceId": 105,
      "intervalMs": 0
    }
  ],
  "range": {
    "from": "2023-06-14T06:45:56.284Z",
    "to": "2023-06-14T12:45:56.284Z",
    "raw": {
      "from": "now-6h",
      "to": "now"
    }
  },
  "from": "1686725156284",
  "to": "1686746756284"
}

The net result is that the query is processed but the response differs from working 1.2.6

{
  "results": {
    "0.7387678513218177": {
      "status": 500,
      "frames": [
        {
          "schema": {
            "name": "0.7387678513218177",
            "refId": "0.7387678513218177",
            "meta": {
              "typeVersion": [
                0,
                0
              ],
              "preferredVisualisationType": "table",
              "executedQueryString": "SELECT COALESCE ..."
            },
            "fields": [
              {
                "name": "html",
                "type": "string",
                "typeInfo": {
                  "frame": "string",
                  "nullable": true
                }
              }
            ]
          },
          "data": {
            "values": [
              [
                "<h1><p>Project View</p></h1>\n<p>The project view ..."
              ]
            ]
          }
        }
      ]
    }
  }
}

And the 1.2.7:

{
  "results": {
    "A": {
      "status": 200,
      "frames": [
        {
          "schema": {
            "name": "A",
            "refId": "A",
            "meta": {
              "typeVersion": [
                0,
                0
              ],
              "preferredVisualisationType": "table",
              "executedQueryString": "SELECT COALESCE..."
            },
            "fields": [
              {
                "name": "html",
                "type": "string",
                "typeInfo": {
                  "frame": "string",
                  "nullable": true
                }
              }
            ]
          },
          "data": {
            "values": [
              [
                "<h1><p>Project View</p></h1>\n<p>The project view ..."
              ]
            ]
          }
        }
      ]
    }
  }
}

[HaddadJoe]

we are facing similar problems, the 1.2.7 breaks our dashboard. The issue comes from a google query variable that's loaded on dashboard load. The query is not being loaded for some reason and the dashboards subsequently fail

[zoltanbedi]

Thanks for the detailed explanation @RobHumphris, I'll take a look at this ASAP.

[zoltanbedi]

@RobHumphris what is the version of grafana you experience this?

I'm trying to reproduce this locally but I can't see the issue. The refID for me is not missing after I bump the plugin. It is certainly the root of the error though.

[RobHumphris]

@zoltanbedi Version 9.5.2 Docker container details:

REPOSITORY       TAG    DIGEST                                                                    IMAGE ID      CREATED      SIZE
grafana/grafana  9.5.2  sha256:39c849cebccccb22c0a5194f07c535669386190e029aa440ad535226974a5809   6c5313ea00cc  5 weeks ago  309MB

[zoltanbedi]

@RobHumphris thanks I was able to reproduce the issue. I'm going to investigate and fix it.

[zoltanbedi]

1.2.8 is out with the fix. thank you again for you detailed report @RobHumphris 🙏

[RobHumphris]

@zoltanbedi thank-you very much for your swift response.