Grafana dashboard presents merged data for ingresses with same name from different namespaces

[davinkevin]

What happened:

We deployed the NGINX Ingress controller dashboard, but because many of our ingress objects are named the same over namespaces, we can't get a clear view of each in the dashbord.

What you expected to happen:

We would like to get a view where $namespace/$ingress_name is used instead of $ingress_name, making relevant data over the complete cluster.

NGINX Ingress controller version (exec into the pod and run nginx-ingress-controller --version.): Not relevant, related to the grafana dashboard, not the controller version.

Kubernetes version (use kubectl version): Not relevant, all k8s versions are concerned (1.19 to 1.26).

How to reproduce this issue:

• Create a cluster
• Install the nginx-ingress into the cluster
• Install kube-prom-stack and the external dashboard from the repository
• Create namespace-a and install any application into it with an ingress named foo
• Create namespace-b and install any application into it with an ingress named foo
• Open the dashboard, you should see only one ingress, named foo instead of two

[k8s-ci-robot]

This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

[longwuyuan]

The only deep dive, available by legacy, related to this is documented here https://kubernetes.github.io/ingress-nginx/user-guide/monitoring/#wildcard-ingresses

PRs are welcome if you want to improve the dashboard json that is available in the project.

Personally I can clearly see dropdowns for multiple options like namespace, controllerclass, controller, ingress in the dashboard that is available here https://github.com/kubernetes/ingress-nginx/blob/main/deploy/grafana/dashboards/nginx.json

So this is not really a bug to me. You can maintain dashboard code on your own and contribute here with the view you describe being a optional choice of dashboard, as a new dashboard json. Ever better would be to maintain the dashboard on the publicly available grafana dashboards collection

/remove-kind bug /feature

[davinkevin]

The dropdown is here to select the namespace where the operator is installed, and then only useful if you have multiple instance in your cluster.

My request is more about ingress objects using the nginx-ingress operator, which are deployed in several namespaces. This is, in the metric system, named as exported_namespace. Potentially and extra dropdown for this in the UI.

See here 👆, our dashboard dropdown with namespace from the operator itself, but none from the dozens of ingresses installed in the cluster and using the ingress-operator as a IngressClass.

[longwuyuan]

@davinkevin that additional info is useful. But based on that additional info, first suggestion would be for you to take a look at the new issue template, but clicking on the create a new issue button. You will see several questions in the template. Then edit your original post, about the problem description and answer those questions.

An example relevance here is you could be having multiple installations of the ingress-nginx-controller from this project in one single cluster but a single prometheus+grafana installation for observability. That is not evident from your post. Secondly you keep referring to an operator, and it will help to know where how you installed an operator and how many of them in one cluster. How many ingressClasses in all etc etc.

[davinkevin]

@longwuyuan I've skipped most of the template because, for me, most of the questions were not relevant to this dashboard problem.

I've provided an "how to reproduce", which is really straightforward… because the issue is simple too. 1 ingress-operator, 1 ingressClass… just two ingresses named the same in two different namespaces and you won't be able to dissociate them in the provided dashboard.

If you insist, I can fill the missing data in the card, but you won't get more than what I've already said in the original issue 😉.

And the tldr is mainly the dashboard doesn't filter ingresses on exported_namespace, which makes this merged result I've presented in the issue.

[longwuyuan]

I ack what you are saying. All good. But I was very very specific about some pieces of info so some kind of action item can be thought of. We don't publish operator so what operator are you talking about. What was the exact command to install the controller. What was the exact procedure to deploy Prometheus and grafana.

On Tue, 7 Mar, 2023, 7:52 pm Davin Kevin, @.***> wrote:

@longwuyuan https://github.com/longwuyuan I've skipped most of the template because, for me, most of the question wasn't relevant.

I've provided an "how to reproduce", which is really straightforward… because the issue is simple too. 1 ingress-operator, 1 ingressClass… just two ingress named the same in two different namespaces and you won't be able to dissociate them in the provided dashboard.

If you insist, I can fill the missing data in the card, but you won't get more than what I've already said in the original issue 😉.

And the tldr is mainly the dashboard doesn't filter ingresses on exported_namespace, which makes this merged result I've presented in the issue.

— Reply to this email directly, view it on GitHub https://github.com/kubernetes/ingress-nginx/issues/9699#issuecomment-1458259939, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABGZVWS4OF57B6XQYZAOI6DW25AARANCNFSM6AAAAAAVRGNW7U . You are receiving this because you were mentioned.Message ID: @.***>

[davinkevin]

I'll try to provide this, I just need to find time for it, because I'm not able to extract it from our private registry "as-is", so I need to create a replicate 😉.

To answer it, when I say operator, it's just me using the wrong word for ingress-nginx. So Sorry for the confusion, it's not an operator.

ingress-nginx can be installed using the simplest possible values.yaml and the official chart, with everything by default. It just needs the metrics to be enabled. It's done with helm install|upgrade or any other alternative way (flux for example).

About Prometheus and Grafana, the kube-prom-stack can be used, with the default setup.

We just need to configure correctly label(s) for ServiceMonitor/PodMonitor on both ends (nginx-ingress and kube-prom-stack). It's just the 101 of installation for both I would say.

And then you should be able to deploy 2 apps, in different namespace with the same ingress name to see the problem in the dashboard.

[github-actions[bot]]

This is stale, but we won't close it automatically, just bare in mind the maintainers may be busy with other tasks and will reach your issue ASAP. If you have any question or request to prioritize this, please reach #ingress-nginx-dev on Kubernetes Slack.