Not all metrics are deleted after removing ingress object from kubernetes cluster

[dkhachyan]

What happened:

After creating ingress object added host metrics with status="404" and with empty ingress and namespace labels. After removing this ingress object such metrics not being deleted

What you expected to happen:

Ingress controller ignore metric event from new host until nginx config reload complete.

NGINX Ingress controller version (exec into the pod and run nginx-ingress-controller --version.): v1.9.3

Kubernetes version (use kubectl version): v1.27.5

Environment:

• Cloud provider or hardware configuration: baremetal

• OS (e.g. from /etc/os-release): Ubuntu 20.04.5 LTS

• Kernel (e.g. uname -a): 5.15.0-43-generic

• How was the ingress-nginx-controller installed: Kubespray

How to reproduce this issue:

• Create a lot of ingress objects(2000+)
• Create one more ingress object
• Perform requests to new ingress host
• Get status 404(nginx config not ready yet)
• Wait for status 200(nginx reload complete)
• Scrape controller metrics and check that metrics with status="404" label don`t have empty ingress and namespace labels
• Remove ingress object
• Scrape controller metrics
• Check that metrics with status="404" label and empty namespace and ingress labels haven`t been removed

Anything else we need to know:

In very large environments during nginx config reload process ingress controller exposes metrics with status="404" label and empty namespace="" and ingress="" labels what causes that such metrics cat not be deleted.

[k8s-ci-robot]

This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

[longwuyuan]

the timeseries data is retained but new data is not appended. so does not seem like a bug. wait for other comments.

/remove-kind bug

[dkhachyan]

Yes the new data is not appended but this metric will remain after deleting ingress object because of empty ingress and namespace labels

https://github.com/kubernetes/ingress-nginx/blob/1bc745619d91b690c8985bbc16097e9fe804d2d2/internal/ingress/metric/collectors/socket.go#L501

I think the problem is here https://github.com/kubernetes/ingress-nginx/blob/1bc745619d91b690c8985bbc16097e9fe804d2d2/internal/ingress/controller/controller.go#L187

host list was updated before the new host was added to the nginx config

May be I should add something like this https://github.com/kubernetes/ingress-nginx/blob/1bc745619d91b690c8985bbc16097e9fe804d2d2/internal/ingress/metric/collectors/socket.go#L327

if sc.metricsPerHost {
    if !sc.hosts.Has(stats.Host) {
        klog.V(3).InfoS("Skipping metric for host not being served", "host", stats.Host)
        continue
    }
    if stats.Host != "" {
        if stats.Namespace == "" || stats.Ingress == "" {
            continue
        }
}

[github-actions[bot]]

This is stale, but we won't close it automatically, just bare in mind the maintainers may be busy with other tasks and will reach your issue ASAP. If you have any question or request to prioritize this, please reach #ingress-nginx-dev on Kubernetes Slack.