grafana / oncall

Developer-friendly incident response with brilliant Slack integration
GNU Affero General Public License v3.0
3.62k stars 299 forks source link

Oncall can't connect to external grafana #4829

Open lamboksilitonga opened 5 months ago

lamboksilitonga commented 5 months ago

What went wrong?

What happened:

Log in pod oncall error is 2024-08-15 06:26:46 source=engine:app google_trace_id=none logger=root inbound latency=0.021681 status=403 method=POST path=/api/internal/v1/plugin/status user_agent=Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:129.0) Gecko/20100101 Firefox/129.0 content-length=0 slow=0 2024-08-15 06:26:46 source=engine:app google_trace_id=none logger=django.request Forbidden: /api/internal/v1/plugin/status 2024-08-15 03:35:35 source=engine:app google_trace_id=none logger=apps.grafana_plugin.helpers.client Error connecting to api instance 404 Client Error: Not Found for url: https://grafana.com/api/plugins/grafana-incident-app/settings 2024-08-15 03:35:35 source=engine:app google_trace_id=none logger=root outbound latency=0.021797078021336347 status=404 method=GET url=https://grafana.com.kepo.ninja/api/plugins/grafana-incident-app/settings slow=0 2024-08-15 03:35:35 source=engine:app google_trace_id=none logger=apps.grafana_plugin.helpers.client Error connecting to api instance 404 Client Error: Not Found for url: https://grafana.com/api/plugins/grafana-labels-app/settings 2024-08-15 03:35:35 source=engine:app google_trace_id=none logger=root outbound latency=0.02221202797954902 status=404 method=GET url=https://grafana.com/api/plugins/grafana-labels-app/settings slow=0 2024-08-15 03:35:35 source=engine:app google_trace_id=none logger=root inbound latency=1.782533 status=201 method=POST path=/api/internal/v1/plugin/self-hosted/install user_agent=Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:129.0) Gecko/20100101 Firefox/129.0 content-length=0 slow=0 2024-08-15 03:35:35 source=engine:uwsgi status=201 method=POST path=/api/internal/v1/plugin/self-hosted/install latency=1.783181 google_trace_id=- protocol=HTTP/1.1 resp_size=446 req_body_size=0 lamboksilitonga@Lambok-Silitonga ~ % 2024-08-15 03:35:35 source=engine:app google_trace_id=none logger=apps.grafana_plugin.helpers.client Error connecting to api instance 404 Client Error: Not Found for url: https://grafana.com/api/plugins/grafana-incident-app/settings 2024-08-15 03:35:35 source=engine:app google_trace_id=none logger=root outbound latency=0.021797078021336347 status=404 method=GET url=https://grafana.com/api/plugins/grafana-incident-app/settings slow=0 2024-08-15 03:35:35 source=engine:app google_trace_id=none logger=apps.grafana_plugin.helpers.client Error connecting to api instance 404 Client Error: Not Found for url: https://grafana.com/api/plugins/grafana-labels-app/settings 2024-08-15 03:35:35 source=engine:app google_trace_id=none logger=root outbound latency=0.02221202797954902 status=404 method=GET url=https://grafana.com/api/plugins/grafana-labels-app/settings slow=0 2024-08-15 03:35:35 source=engine:app google_trace_id=none logger=root inbound latency=1.782533 status=201 method=POST path=/api/internal/v1/plugin/self-hosted/install user_agent=Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:129.0) Gecko/20100101 Firefox/129.0 content-length=0 slow=0 2024-08-15 03:35:35 source=engine:uwsgi status=201 method=POST path=/api/internal/v1/plugin/self-hosted/install latency=1.783181 google_trace_id=- protocol=HTTP/1.1 resp_size=446 req_body_size=0x zsh: parse error near `)'

Here is when access the grafana domain from oncall pod grafana-oncall-engine-6ff6987857-rq4t2:/etc/app$ wget --header="Authorization: Bearer $token-grafana" https://grafana.com/api/health Connecting to grafana.com (34.87.111.98:443) saving to 'health' health 100% |***| 101 0:00:00 ETA 'health' saved grafana-oncall-engine-6ff6987857-rq4t2:/etc/app$ ls Dockerfile grpcio-1.64.1-cp312-cp312-linux_aarch64.whl settings apps health static celery_with_exporter.sh manage.py tox.ini collected_static pyproject.toml type_stubs common requirements-dev.in uwsgi.ini config_integrations requirements-dev.txt wait_for_test_mysql_start.sh conftest.py requirements.in engine requirements.txt grafana-oncall-engine-6ff6987857-rq4t2:/etc/app$ cat health { "commit": "b037f37bfe5858b91282557c126b27f7bac9246", "database": "ok", "version": "10.4.5"

grafana-584c96b4f8-d2k7g:/usr/share/grafana$ curl -v http://observability-endpoint.nonprod.tiket.tools:20005/

What did you expect to happen:

How do we reproduce it?

Get log from pod grafana oncall engine

Grafana OnCall Version

v1.8.9

Product Area

Auth, API, Helm/Kubernetes/Docker

Grafana OnCall Platform?

Kubernetes

User's Browser?

No response

Anything else to add?

No response

NEW7BEE commented 5 months ago

v1.9.4 image

lamboksilitonga commented 5 months ago

v1.9.4 image

Should i update plugin grafana oncall to V1.94 or what @NEW7BEE ?

NEW7BEE commented 5 months ago

What I mean is that even after upgrading to v1.9.4, I'm still running into issues, like the configuration options not showing up as seen in the screenshot.

slax81 commented 5 months ago

I tested with several versions of the plugin and v1.8.8 had config tab :( I haven't tested 1.8.9, but 1.8.10 doesn't work This is with bundled grafana from helm chart

Image

mderynck commented 5 months ago

Recently we made some changes to the way Grafana OnCall is initialized. Use 1.9.22, there were quite a few changes along the way from 1.9.0-1.9.22 to get things working.

douglasg14b commented 1 month ago

Is the above a workaround, or is that the final behavior? The new user experiance is very broken given the only information presented is:

sonData.stackId is not set

Image

bck01215 commented 1 month ago

stackid should be set to 5 and orgid 100 for all self hosted instances. @douglasg14b

curl -X POST 'http://admin:admin@localhost:3000/api/plugins/grafana-oncall-app/settings' -H "Content-Type: application/json" -d '{"enabled":true, "jsonData":{"stackId":5, "orgId":100, "onCallApiUrl":"http://release-oncall-engine:8080", "grafanaUrl":"https://grafana.obs.liberty.edu/"}}'
douglasg14b commented 3 weeks ago

@bck01215 After doing this the error goes away, but oncall continues to indicate it's not connected:

However, there are no errors after clicking Connect

Image

Image

bck01215 commented 3 weeks ago

@bck01215 After doing this the error goes away, but oncall continues to indicate it's not connected:

However, there are no errors after clicking Connect

Image

Image

Can you provide error logs or network error requests. Try to uninstall prior and if you used the wrong id's in a previous request it may cause errors

douglasg14b commented 3 weeks ago

Looks like an endpoint doesn't exist?

msg="Request Completed" method=GET path=/api/access-control/users/2/permissions status=404

Grafana 11.3.0

Potential fix here: https://github.com/grafana/oncall/issues/5100 ?

bck01215 commented 3 weeks ago

Yes, you need to make sure accessControlOncall is set to false is you use OSS