Closed lstama closed 3 weeks ago
I've faced the same problem. Looks just like https://github.com/grafana/oncall/issues/316. I've reinstalled oncall deployment and now it's performing fine
I've faced the same problem. Looks just like #316. I've reinstalled oncall deployment and now it's performing fine
What do you mean by reinstalling oncall deployment? Is it the Engine and Celery part, or everything including MariaDB, Redis, and RabbitMQ (using a newly fresh DB)?
I meant re-deploying oncall helm chart (i have redis, mariadb, rabbitmq, celery and engine enabled for deployment). Also i think that it's better to delete PVC's (you should do it manually, as it's stated in docs), because sometimes plugin init fails
I meant re-deploying oncall helm chart (i have redis, mariadb, rabbitmq, celery and engine enabled for deployment). Also i think that it's better to delete PVC's (you should do it manually, as it's stated in docs), because sometimes plugin init fails
Thanks, reinstalling works.
But now all my integrations and settings are wiped out. As I don't know which DB table is safe to backup and restore.
Thanks, reinstalling works.
But now all my integrations and settings are wiped out. As I don't know which DB table is safe to backup and restore.
Yep, that's quite inconvenient. Hope this behavior would be fixed soon
Got the same issue with Grafana 9.2.6, Oncall 1.1.5 and Helm 1.0.12. After couple of days Oncall setup becomes useless. Removing OnCall API key from https://grafana/org/apikeys helps till the next page reload.
I have the same issue as @juris
I have the same issue. Helm deployment, only postgresql is external.
Grafana 9.3.2 ; OnCall 1.1.14 ; Ingress disabled (HAProxy)
After some time plugin just lost their API key:
In grafana logs I've found:
logger=context t=2023-01-12T13:21:53.551966373Z level=error msg="invalid API key" error="invalid API key" traceID= logger=data-proxy-log userId=2 orgId=1 uname=user path=/api/plugin-proxy/grafana-oncall-app/api/internal/v1/alertgroups/stats/ remote_addr=ip referer="https://fqdn/a/grafana-oncall-app/?page=incidents&status=0&status=1" t=2023-01-12T11:47:13.60359748Z level=error msg="Proxy request failed" err="dial tcp ip:8080: connect: connection refused"
Plugin configuration page say cannot communicate with oncall-engine but don't provide button to reset configuration.
Sometimes just opening general page of on call starts api key exchange as people wrote above, sometimes only redeploy helps to me.
I have the same issue as @juris
Have the same issue as @PCbIX:
Each time I leave Grafana Oncall page the Grafana API Token is supposed to be recreated, but it's not recreating and I'm loosing the connection to Grafana Oncall plugin with a message: 'There was an issue while synchronizing data required for the plugin. Verify your OnCall backend setup (ie. that Celery workers are launched and properly configured)'
First time the workaround to reopen a general page of Grafana Oncall to start api key exchange went through: the notification of the API token creation popped up and I could access Grafana Oncall. The second time it didn't work and I'm stuck on 'Initializing plugin' step with a message: 'There was an issue while synchronizing data required for the plugin. Verify your OnCall backend setup (ie. that Celery workers are launched and properly configured)'.
Recently we made some changes to the way Grafana OnCall is initialized. Use 1.9.22, there were quite a few changes along the way from 1.9.0-1.9.22 to get things working.
externalServiceAccounts
feature toggle enabled.
This has already been enabled in the docker compose files and helm charts in the oncall repo.curl -X POST 'http://admin:admin@localhost:3000/api/plugins/grafana-oncall-app/settings' -H "Content-Type: application/json" -d '{"enabled":true, "jsonData":{"stackId":5, "orgId":100, "onCallApiUrl":"http://engine:8080/", "grafanaUrl":"http://grafana:3000/"}}'
curl -X POST 'http://admin:admin@localhost:3000/api/plugins/grafana-oncall-app/resources/plugin/install'
Grafana OnCall should now be ready to use. For additional troubleshooting see here
Hello, I have a problem which make user without admin privilege can't access Grafana OnCall.
When I (as an organization admin) open OnCall page for the first time (or when doing a reload), I always greeted by this error page.
And when I click the retry button, this happened.
Then I'm redirected to the normal OnCall page
For now it looks fine, I still can access OnCall in the end, and can create alert and integration. Then my friend who isn't an admin want to view the page as an editor. I instruct him to do the same (click retry if the error page shows up). He did what I said, but instead of seeing the same view as picture two, he got this:
Turns out the Grafana API Token is always being recreated each time someone reload the page (already checked the DB value), Plugin page also has this error:
What I already did:
Also, these are some relevant logs from Grafana:
Oncall Engine:
Celery
We're using existing Grafana as OnCall frontend and deploy Grafana OnCall using helm in this repository. The alerting and OnCall system itself work normally.