Alertmanager dashboard link returns JSON error response (NotFound)

ropable commented 1 year ago

Setup

Rancher version: v2.7.5
Rancher UI Extensions: None
Browser type & version: Firefox

Describe the bug After installing the built-in Monitoring (102.0.1+up40.1.2) app in a new Rancher cluster, the link to the Alertmanager UI returns an error message in the form of a JSON response. EDIT: following a bit more experimentation, I receive the same (identical) error response when trying to log into the Grafana dashboard (see additional note posted below).

To Reproduce Install Rancher in a new cluster, enable Monitoring for use without SSL via the Apps menu as per the documentation (no additional customisation), then try opening the Alertmanager UI via the link in the dashboard.

I'm not aware of any additional configuration within Rancher itself that would lead to this issue, but am happy to provide any additional context or information required.

Result The following content is the full JSON response of the error:

{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "services \"http\" not found",
  "reason": "NotFound",
  "details": {
    "name": "http",
    "kind": "services"
  },
  "code": 404
}

Expected Result The Alertmanager UI should open in a new browser tab, as per the documentation .

Additional context The alertmanager-rancher-monitoring-alertmanager StatefulSet workload has started and appears to be running OK. The rancher-monitoring-alertmanager ClusterIP has been created by the Helm chart.

richard-cox commented 1 year ago

Just to confirm, which link to alert manage is not working (I think there's a few)?

ropable commented 1 year ago

@richard-cox For clarity, I meant the link to the Alertmanager UI that is provided via the left navigation bar (Monitoring -> Alertmanager). For our cluster, the URL is https://our.custom.domain/k8s/clusters/c-z6twk/api/v1/namespaces/cattle-monitoring-system/services/http:rancher-monitoring-alertmanager/proxy/

Screenshot:

ropable commented 1 year ago

Also for clarity, the links to the other four services (Prometheus Graph, etc.) in the screenshot above do work as expected.

Further, if I start a shell session in a pod on the cluster I can make a successful HTTP request to the alertmanager workload, i.e.:

root@temp:/# curl http://rancher-monitoring-alertmanager.cattle-monitoring-system.svc.cluster.local:9093/
<!DOCTYPE html>
<html lang="en">
    <head>
        <meta charset="utf-8">
        <meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
        <link rel="icon" type="image/x-icon" href="favicon.ico" />
        <title>Alertmanager</title>
    </head>
    <body>
        <script>
            // If there is no trailing slash at the end of the path in the url,
            // add one. This ensures assets like script.js are loaded properly
            if (location.pathname.substr(-1) != '/') {
                location.pathname = location.pathname + '/';
                console.log('added slash');
            }
        </script>
        <script src="script.js"></script>
        <script>
            var app = Elm.Main.init({
                flags: {
                    production: true,
                    defaultCreator: localStorage.getItem('defaultCreator'),
                    groupExpandAll: JSON.parse(localStorage.getItem('groupExpandAll'))
                }
            });
            app.ports.persistDefaultCreator.subscribe(function(name) {
                localStorage.setItem('defaultCreator', name);
            });
            app.ports.persistGroupExpandAll.subscribe(function(expanded) {
                localStorage.setItem('groupExpandAll', JSON.stringify(expanded));
            });
        </script>
    </body>
</html>
root@temp:/#

ropable commented 1 year ago

In a further twist, I see the same error message when trying to log into the built-in Grafana UI, local URL: https://our.custom.domain/k8s/clusters/c-z6twk/api/v1/namespaces/cattle-monitoring-system/services/http:rancher-monitoring-grafana/proxy/login

This returns the same JSON error output:

{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "services \"http\" not found",
  "reason": "NotFound",
  "details": {
    "name": "http",
    "kind": "services"
  },
  "code": 404
}

nwmac commented 7 months ago

This will be because the user does not have access to the service - we should check that the services exist - we do this for longhorn - see: https://github.com/rancher/dashboard/pull/10263

To address this, the dashboard page should check access to services and only enable the links if the user can access the service - if they can not, they will get the errors reported, since the k8s proxy mechanism won't work.

rancher / dashboard

Alertmanager dashboard link returns JSON error response (NotFound) #9623