Open knkarthik opened 2 years ago
Hello! I've never seen Ruby hooks, thank you for these examples!
Q: sometimes it is combined and tasks are dropped from the queue, other times all tasks are executed A: Context combining inconsistency is not a bug. Events from Kubernetes come with unpredictable timings, so the second event can arrive during the hook execution for the first event — there is no combining in this situation. You can see this in logs:
"msg":"kubernetes[0]{OnCreateDeleteArgoApp}: Deleted argocd/Application/aut-410: send KubeEvent","time":"2021-09-29T18:56:57Z"}
"msg":"Create tasks for 'kubernetes' event 'Event 'Deleted' for argocd/Application/aut-410'","time":"2021-09-29T18:56:57Z"}
"msg":"queue task HookRun:main:kubernetes:integrations.rb:OnCreateDeleteArgoApp","operator.component":"handleEvents","queue":"main","task.id":"e56af739-fa33-4403-8503-5ba36a6e08c1","time":"2021-09-29T18:56:57Z"}
...
"msg":"Executing command '/hooks/integrations.rb' in '/hooks' dir","queue":"main","task":"HookRun","time":"2021-09-29T18:56:57Z"}
"msg":"kubernetes[0]{OnCreateDeleteArgoApp}: Deleted argocd/Application/aut-369: send KubeEvent","time":"2021-09-29T18:56:57Z"}
...
"msg":"queue task HookRun:main:kubernetes:integrations.rb:OnCreateDeleteArgoApp","operator.component":"handleEvents","queue":"main","task.id":"1546bd1e-c318-4db1-9074-59c204735aef","time":"2021-09-29T18:56:57Z"}
...
"msg":"Executing command '/hooks/integrations.rb' in '/hooks' dir","queue":"main","task":"HookRun","time":"2021-09-29T18:56:58Z"}
Sometimes both events arrive quickly and tasks are queued fast and contexts are combined.
Context combining is just a helper to withstand high loads, it can't be consistent with kubectl commands.
Q: Sometimes it runs for only one resource and sometimes two. A: I see that the context is parsed into array with two objects:
fit-qa-integrations-588c8bfb76-rbntf fit-qa-integrations {"binding":"OnCreateDeleteArgoApp","event":"kubernetes","hook":"integrations.rb","level":"info","msg":"-----2","output":"stdout","queue":"main","task":"HookRun","time":"2021-09-29T18:57:57Z"}
Can you attach the content of BINDING_CONTEXT_PATH
? You can remove "object" content.
Q: I want to run cleanup tasks based each resource that trigged the Deleted event
A:
Reacting to individual Deleted
event is not reliable for cleanup. Please consider using the snapshot mode and compare two sets of objects to detect deleted.
I observe the same issue with combining. I use forked version of this operator https://github.com/confluentinc/streaming-ops/tree/main/images/connect-operator with latest latest version of flant/shell-operator:v1.0.7. And when we create/update/delete more than one ConfigMap hook runs only for one of them. In logs there is a message:
"Binding contexts from 3 tasks are combined to 3 contexts. 2 tasks are dropped from queue 'main'"
How can we force running hook for all ConfigMaps?
@diafour any thoughts how can we force running hook for all ConfigMaps?
I have also noticed this issue since 1.0.8
(currently running 1.0.9
). In the context of resources managed by ArgoCD with self-healing enabled, if the resource is deleted and then created (via ArgoCD self-healing) seconds afterwards, the watchEvent: Added
and even the watchEvent: Modified
are sometimes combined with the Deleted
event.
I'm observing the same issue using v1.4.5 version. Any idea? it causing us to inconsistent behavior when syncing more than 1 object at once
According to https://github.com/flant/shell-operator/blob/master/HOOKS.md#shell-operator-lifecycle, If there is a sequence of hook executions in a queue, then hook is executed once with an array of binding contexts. However, I've noticed that this is not consistent - sometimes it is combined and tasks are dropped from the queue, other times all tasks are executed. I'm attaching hooks and logs below.
Expected behavior (what you expected to happen): Since I want to run cleanup tasks based each resource that trigged the
Deleted
event, when I runkubectl delete applications.argoproj.io aut-410 aut-369
I expected it to always run twice - one each for aut-410 and aut-369 resource.Actual behavior (what actually happened): Sometimes it runs for only one resource and sometimes two.
Anything else we should know?:
I inspected the json file pointed to by
BINDING_CONTEXT_PATH
and confirmed that it has both the objects.Hook script
Logs when contexts are combined
Logs when contexts are not combined
Here's a slight variation of the same hook, except that there's a failure in one of the cleanup tasks it tries to execute. Sometimes it executes only once and sometimes more than once.
Hook script 2
Logs when contexts are combined
Logs when contexts are not combined
Environment:
kubectl apply