v1.9.0 Grafana OnCall Plugin: Can't connect to Grafana Cloud OnCall

tuminansen commented 2 weeks ago

What went wrong?

What happened: I tried to connect the Grafana OnCall plugin (v1.9.0 as recommended in Grafana Community Slack, and also with newer versions) to a Grafana Cloud instance for OnCall, but ran into an issue. I set the GRAFANA_CLOUD_ONCALL_API_URL and GRAFANA_CLOUD_ONCALL_API_TOKEN environment variables, but they don't seem to have any effect.

What did you expect to happen:

I expected the Grafana OnCall plugin to successfully connect to the Grafana Cloud OnCall instance using the provided backend URL and API token.

How do we reproduce it?

In your self-hosted Grafana instance, Install Grafana OnCall Plugin
Restart the instance with GRAFANA_CLOUD_ONCALL_API_URL and GRAFANA_CLOUD_ONCALL_API_TOKEN according to API URL and API TOKEN in your Grafana Cloud instance OnCall settings.
In your self-hosted Grafana instance, insert the OnCall API URL (taken from Grafana Cloud instance) into the OnCall backend URL. In my case it's https://oncall-prod-us-central-0.grafana.net/oncall/
Observe error

Grafana OnCall Version

v.1.9.0

Product Area

Auth, API, Other

Grafana OnCall Platform?

I use Grafana Cloud

User's Browser?

Chrome Version 128.0.6613.85 (Official Build) (arm64)

Anything else to add?

Our Grafana self-hosted instance is running version grafana/grafana-enterprise:11.1.2
Raised this in the Grafana Community Slack also, link: https://grafana.slack.com/archives/C02LSUUSE2G/p1724706372386709

youve commented 1 week ago

I have a similar issue, reported it on the forums a few days ago, https://community.grafana.com/t/trying-to-connect-selfhosted-grafana-to-cloud-oncall/129985

The difference in my case is that I set the Oncall Backend URL in our self-hosted Grafana to point to our self-hosted Oncall-engine and it successfully connects to it, but the token for Grafana Cloud doesn't work even though it works with curl.

tuminansen commented 1 week ago

We are still running into this error. Can anyone take a look, please?

mderynck commented 1 week ago

Recently we made some changes to the way Grafana OnCall is initialized. Use 1.9.22, there were quite a few changes along the way from 1.9.0-1.9.22 to get things working.

If you are running Grafana 11 and newer you must have externalServiceAccounts feature toggle enabled. This has already been enabled in the docker compose files and helm charts in the oncall repo.
Plugin settings must be provided to the plugin using an API call if you are installing for the first time (Note: credentials and hostnames need to be adjusted for your configuration, stackId and orgId are expected to be the listed constants in a self-hosted configuration)
```
curl -X POST 'http://admin:admin@localhost:3000/api/plugins/grafana-oncall-app/settings' -H "Content-Type: application/json" -d '{"enabled":true, "jsonData":{"stackId":5, "orgId":100, "onCallApiUrl":"http://engine:8080/", "grafanaUrl":"http://grafana:3000/"}}'
```
Once settings are configured use this API call to install:
```
curl -X POST 'http://admin:admin@localhost:3000/api/plugins/grafana-oncall-app/resources/plugin/install'
```
Grafana OnCall should now be ready to use. For additional troubleshooting see here

tuminansen commented 6 days ago

Recently we made some changes to the way Grafana OnCall is initialized. Use 1.9.22, there were quite a few changes along the way from 1.9.0-1.9.22 to get things working.

If you are running Grafana 11 and newer you must have externalServiceAccounts feature toggle enabled. This has already been enabled in the docker compose files and helm charts in the oncall repo.

Plugin settings must be provided to the plugin using an API call if you are installing for the first time (Note: credentials and hostnames need to be adjusted for your configuration, stackId and orgId are expected to be the listed constants in a self-hosted configuration)
curl -X POST 'http://admin:admin@localhost:3000/api/plugins/grafana-oncall-app/settings' -H "Content-Type: application/json" -d '{"enabled":true, "jsonData":{"stackId":5, "orgId":100, "onCallApiUrl":"http://engine:8080/", "grafanaUrl":"http://grafana:3000/"}}'
Once settings are configured use this API call to install:
curl -X POST 'http://admin:admin@localhost:3000/api/plugins/grafana-oncall-app/resources/plugin/install'
Grafana OnCall should now be ready to use. For additional troubleshooting see here

@mdyerynck This resolves the issue I was previously facing, but now I’m encountering a new challenge. Here’s the error I get in the UI:

And this is what the API returns:

curl -X GET 'http://admin:admin@localhost:3000/api/plugins/grafana-oncall-app/resources/plugin/status' | jq .

{
  "pluginConnection": {
    "settings": {
      "ok": true
    },
    "service_account_token": {
      "ok": true
    },
    "grafana_url_from_plugin": {
      "ok": true
    },
    "grafana_url_from_engine": {
      "ok": false,
      "error": "Not validated"
    },
    "oncall_api_url": {
      "ok": false,
      "error": "Not validated"
    },
    "oncall_token": {
      "ok": false,
      "error": "Unauthorized/Forbidden while accessing OnCall engine: /oncall/api/internal/v1/plugin/v2/status, status code: 403, check token"
    }
  },
  "license": "",
  "version": "",
  "currently_undergoing_maintenance_message": "",
  "api_url": ""
}

I’ve tried setting the ONCALL_API_TOKEN as both an environment variable when starting my Grafana instance and passing it in via the API call, like this:

curl -X POST \
'http://admin:admin@localhost:3000/api/plugins/grafana-oncall-app/settings' \
-H "Content-Type: application/json" \
-d '{
  "enabled": true,
  "jsonData": {
    "stackId": 5,
    "orgId": 100,
    "onCallApiUrl": "<ONCALL API URL>",
    "onCallApiToken": "<ONCALL API TOKEN>",
    "grafanaUrl": "http://localhost:3000/"
  }
}'

Looking through the code in the repo (https://github.com/grafana/oncall), it seems ONCALL_API_TOKEN is the token causing the issue, as mentioned here: https://github.com/grafana/oncall/tree/dev/tools/migrators#configuration

grafana / oncall