search

kubernetes-sigs / controller-runtime

Repo for the controller-runtime subproject of kubebuilder (sig-apimachinery)
Apache License 2.0
2.44k stars 1.13k forks source link

Instance of CR in the work queue are not unique #2703

Closed lmeunier-talend closed 1 month ago

lmeunier-talend commented 7 months ago

Hi,

I have a controller that reconcile resource of type A and it requeue instance of type A until the state on kubernetes match what i'm expecting

Now this controller also watch deployments which trigger my reconciling loop

My issue is that i'm watching like 5 of them and they often all change at the same time which mean i will queue 5 items of the same instance each time something happens.

Each item will be requeue until state match what is expected, then it will still proceed all remaining items in the queue even tho reconciliation is done for that instance

It's not that much of an issue but it does flood the logs for nothing and it happens that we have a http notification when reconciliation is done and i don't want to do 10 time in 1 sec

What I would like is to only queue if there are no instance of this object in work queue already so that i have never more than 1 instance of a object in the work queue

I did some search but i did not find a way to do that.

Thank you for answer

alvaroaleman commented 7 months ago

The work queue deduplicates identical objects, you will at most have a given identifier in there

lmeunier-talend commented 6 months ago

I have only 1 worker and yet according toe the log at the same second (2024-03-13T13:42:32Z) the operator does the loop 3 times for the same instance of my resources

10 second later which is my re-queue time, it does the same, 3 reconciliation loops

May be i'm missing but for me it means i have at least 3 instances of my object in the queue

alvaroaleman commented 6 months ago

Maybe you don't put a string but the full object into the workqueue, which then won't be de-duplicated because its not identical?

The mount of information you are giving here doesn't make it possible to help you, can you link to a code snippet that reproduces the behavior you are describing?

troy0820 commented 6 months ago

/kind support

lmeunier-talend commented 6 months ago

my code only put instance in the queue in two ways:

lmeunier-talend commented 6 months ago

The logs after i deleted two pods of watched deployment

2024-03-13T13:53:40Z    INFO    Reconciling {"controller": "qlikruntime", "controllerGroup": "qlik.com", "controllerKind": "QlikRuntime", "QlikRuntime": {"name":"remote-engine-dc4e5f42-0bab-46da-b43d-de24a13d25f6"}, "namespace": "", "name": "remote-engine-dc4e5f42-0bab-46da-b43d-de24a13d25f6", "reconcileID": "2393b59b-3292-4ba3-a5d0-917ef42b720d"}
2024-03-13T13:53:40Z    INFO    Install in progress, re-queueing    {"controller": "qlikruntime", "controllerGroup": "qlik.com", "controllerKind": "QlikRuntime", "QlikRuntime": {"name":"remote-engine-dc4e5f42-0bab-46da-b43d-de24a13d25f6"}, "namespace": "", "name": "remote-engine-dc4e5f42-0bab-46da-b43d-de24a13d25f6", "reconcileID": "2393b59b-3292-4ba3-a5d0-917ef42b720d"}
2024-03-13T13:53:40Z    INFO    Reconciling {"controller": "qlikruntime", "controllerGroup": "qlik.com", "controllerKind": "QlikRuntime", "QlikRuntime": {"name":"remote-engine-dc4e5f42-0bab-46da-b43d-de24a13d25f6"}, "namespace": "", "name": "remote-engine-dc4e5f42-0bab-46da-b43d-de24a13d25f6", "reconcileID": "53e2a89f-c8a4-47ce-a30e-a4f0fe3d6e55"}
2024-03-13T13:53:40Z    INFO    Install in progress, re-queueing    {"controller": "qlikruntime", "controllerGroup": "qlik.com", "controllerKind": "QlikRuntime", "QlikRuntime": {"name":"remote-engine-dc4e5f42-0bab-46da-b43d-de24a13d25f6"}, "namespace": "", "name": "remote-engine-dc4e5f42-0bab-46da-b43d-de24a13d25f6", "reconcileID": "53e2a89f-c8a4-47ce-a30e-a4f0fe3d6e55"}
2024-03-13T13:53:50Z    INFO    Reconciling {"controller": "qlikruntime", "controllerGroup": "qlik.com", "controllerKind": "QlikRuntime", "QlikRuntime": {"name":"remote-engine-dc4e5f42-0bab-46da-b43d-de24a13d25f6"}, "namespace": "", "name": "remote-engine-dc4e5f42-0bab-46da-b43d-de24a13d25f6", "reconcileID": "395670e8-ffae-49b0-9264-1abaee6541bb"}
2024-03-13T13:53:50Z    INFO    Install in progress, re-queueing    {"controller": "qlikruntime", "controllerGroup": "qlik.com", "controllerKind": "QlikRuntime", "QlikRuntime": {"name":"remote-engine-dc4e5f42-0bab-46da-b43d-de24a13d25f6"}, "namespace": "", "name": "remote-engine-dc4e5f42-0bab-46da-b43d-de24a13d25f6", "reconcileID": "395670e8-ffae-49b0-9264-1abaee6541bb"}
2024-03-13T13:53:55Z    INFO    Reconciling {"controller": "qlikruntime", "controllerGroup": "qlik.com", "controllerKind": "QlikRuntime", "QlikRuntime": {"name":"remote-engine-dc4e5f42-0bab-46da-b43d-de24a13d25f6"}, "namespace": "", "name": "remote-engine-dc4e5f42-0bab-46da-b43d-de24a13d25f6", "reconcileID": "38fa5aa7-55a9-4886-908f-6373f9e5c1b6"}
2024-03-13T13:53:55Z    INFO    Install in progress, re-queueing    {"controller": "qlikruntime", "controllerGroup": "qlik.com", "controllerKind": "QlikRuntime", "QlikRuntime": {"name":"remote-engine-dc4e5f42-0bab-46da-b43d-de24a13d25f6"}, "namespace": "", "name": "remote-engine-dc4e5f42-0bab-46da-b43d-de24a13d25f6", "reconcileID": "38fa5aa7-55a9-4886-908f-6373f9e5c1b6"}
2024-03-13T13:53:55Z    INFO    Reconciling {"controller": "qlikruntime", "controllerGroup": "qlik.com", "controllerKind": "QlikRuntime", "QlikRuntime": {"name":"remote-engine-dc4e5f42-0bab-46da-b43d-de24a13d25f6"}, "namespace": "", "name": "remote-engine-dc4e5f42-0bab-46da-b43d-de24a13d25f6", "reconcileID": "bdcec4e8-055c-4b4a-8cb9-7148a26e1c36"}
2024-03-13T13:53:55Z    INFO    Install in progress, re-queueing    {"controller": "qlikruntime", "controllerGroup": "qlik.com", "controllerKind": "QlikRuntime", "QlikRuntime": {"name":"remote-engine-dc4e5f42-0bab-46da-b43d-de24a13d25f6"}, "namespace": "", "name": "remote-engine-dc4e5f42-0bab-46da-b43d-de24a13d25f6", "reconcileID": "bdcec4e8-055c-4b4a-8cb9-7148a26e1c36"}
2024-03-13T13:54:00Z    INFO    Reconciling {"controller": "qlikruntime", "controllerGroup": "qlik.com", "controllerKind": "QlikRuntime", "QlikRuntime": {"name":"remote-engine-dc4e5f42-0bab-46da-b43d-de24a13d25f6"}, "namespace": "", "name": "remote-engine-dc4e5f42-0bab-46da-b43d-de24a13d25f6", "reconcileID": "8cbf4a3d-753b-4fdc-9f79-bd2c149681ac"}
2024-03-13T13:54:00Z    INFO    Install in progress, re-queueing    {"controller": "qlikruntime", "controllerGroup": "qlik.com", "controllerKind": "QlikRuntime", "QlikRuntime": {"name":"remote-engine-dc4e5f42-0bab-46da-b43d-de24a13d25f6"}, "namespace": "", "name": "remote-engine-dc4e5f42-0bab-46da-b43d-de24a13d25f6", "reconcileID": "8cbf4a3d-753b-4fdc-9f79-bd2c149681ac"}
2024-03-13T13:54:10Z    INFO    Reconciling {"controller": "qlikruntime", "controllerGroup": "qlik.com", "controllerKind": "QlikRuntime", "QlikRuntime": {"name":"remote-engine-dc4e5f42-0bab-46da-b43d-de24a13d25f6"}, "namespace": "", "name": "remote-engine-dc4e5f42-0bab-46da-b43d-de24a13d25f6", "reconcileID": "c427b9f9-209d-4d03-a0e2-0f756051aefe"}
2024-03-13T13:54:10Z    INFO    Install in progress, re-queueing    {"controller": "qlikruntime", "controllerGroup": "qlik.com", "controllerKind": "QlikRuntime", "QlikRuntime": {"name":"remote-engine-dc4e5f42-0bab-46da-b43d-de24a13d25f6"}, "namespace": "", "name": "remote-engine-dc4e5f42-0bab-46da-b43d-de24a13d25f6", "reconcileID": "c427b9f9-209d-4d03-a0e2-0f756051aefe"}
2024-03-13T13:54:20Z    INFO    Reconciling {"controller": "qlikruntime", "controllerGroup": "qlik.com", "controllerKind": "QlikRuntime", "QlikRuntime": {"name":"remote-engine-dc4e5f42-0bab-46da-b43d-de24a13d25f6"}, "namespace": "", "name": "remote-engine-dc4e5f42-0bab-46da-b43d-de24a13d25f6", "reconcileID": "34cd6259-7ca7-44fc-b688-4152d3f86474"} found for service 'di-job-deployer'   {"controller": "qlikruntime", "controllerGroup": "qlik.com", "controllerKind": "QlikRuntime", "QlikRuntime": {"name":"remote-engine-dc4e5f42-0bab-46da-b43d-de24a13d25f6"}, "namespace": "", "name": "remote-engine-dc4e5f42-0bab-46da-b43d-de24a13d25f6", "reconcileID": "34cd6259-7ca7-44fc-b688-4152d3f86474"}
2024-03-13T13:54:20Z    INFO    All services are running, exiting reconciliation loop   {"controller": "qlikruntime", "controllerGroup": "qlik.com", "controllerKind": "QlikRuntime", "QlikRuntime": {"name":"remote-engine-dc4e5f42-0bab-46da-b43d-de24a13d25f6"}, "namespace": "", "name": "remote-engine-dc4e5f42-0bab-46da-b43d-de24a13d25f6", "reconcileID": "34cd6259-7ca7-44fc-b688-4152d3f86474"}
2024-03-13T13:54:20Z    INFO    Reconciling {"controller": "qlikruntime", "controllerGroup": "qlik.com", "controllerKind": "QlikRuntime", "QlikRuntime": {"name":"remote-engine-dc4e5f42-0bab-46da-b43d-de24a13d25f6"}, "namespace": "", "name": "remote-engine-dc4e5f42-0bab-46da-b43d-de24a13d25f6", "reconcileID": "7147644b-d077-4f96-8cd8-f2897ee81442"}
2024-03-13T13:54:20Z    INFO    All services are running, exiting reconciliation loop   {"controller": "qlikruntime", "controllerGroup": "qlik.com", "controllerKind": "QlikRuntime", "QlikRuntime": {"name":"remote-engine-dc4e5f42-0bab-46da-b43d-de24a13d25f6"}, "namespace": "", "name": "remote-engine-dc4e5f42-0bab-46da-b43d-de24a13d25f6", "reconcileID": "7147644b-d077-4f96-8cd8-f2897ee81442"}
2024-03-13T13:54:20Z    INFO    Reconciling {"controller": "qlikruntime", "controllerGroup": "qlik.com", "controllerKind": "QlikRuntime", "QlikRuntime": {"name":"remote-engine-dc4e5f42-0bab-46da-b43d-de24a13d25f6"}, "namespace": "", "name": "remote-engine-dc4e5f42-0bab-46da-b43d-de24a13d25f6", "reconcileID": "2dd9a973-bdcc-4867-b7d6-a1c516a59e09"}
2024-03-13T13:54:21Z    INFO    All services are running, exiting reconciliation loop   {"controller": "qlikruntime", "controllerGroup": "qlik.com", "controllerKind": "QlikRuntime", "QlikRuntime": {"name":"remote-engine-dc4e5f42-0bab-46da-b43d-de24a13d25f6"}, "namespace": "", "name": "remote-engine-dc4e5f42-0bab-46da-b43d-de24a13d25f6", "reconcileID": "2dd9a973-bdcc-4867-b7d6-a1c516a59e09"}
alvaroaleman commented 6 months ago

The logs after i deleted two pods of watched deployment

You likely are just getting multiple events from the Deployment, as its status is updated to reflect the two deletions in order and then new pods created by the depoyment controller. Additionally, while the queue itself de-duplicates, the queue might hold the object that is currently being reconciled, resulting in it getting reconciled again right after.

k8s-triage-robot commented 3 months ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot commented 2 months ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

k8s-triage-robot commented 1 month ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

k8s-ci-robot commented 1 month ago

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to [this](https://github.com/kubernetes-sigs/controller-runtime/issues/2703#issuecomment-2302721936): >The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. > >This bot triages issues according to the following rules: >- After 90d of inactivity, `lifecycle/stale` is applied >- After 30d of inactivity since `lifecycle/stale` was applied, `lifecycle/rotten` is applied >- After 30d of inactivity since `lifecycle/rotten` was applied, the issue is closed > >You can: >- Reopen this issue with `/reopen` >- Mark this issue as fresh with `/remove-lifecycle rotten` >- Offer to help out with [Issue Triage][1] > >Please send feedback to sig-contributor-experience at [kubernetes/community](https://github.com/kubernetes/community). > >/close not-planned > >[1]: https://www.kubernetes.dev/docs/guide/issue-triage/ Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.