search

argoproj / argo-cd

Declarative Continuous Deployment for Kubernetes
https://argo-cd.readthedocs.io
Apache License 2.0
17.61k stars 5.37k forks source link

Resources created in PreSync phase with BeforeHookCreation hook deletion policy get deleted after all retries have failed #14046

Open bcbrockway opened 1 year ago

bcbrockway commented 1 year ago

Checklist:

Describe the bug

If the following are true...

  1. You create some resources in the PreSync phase with the BeforeHookCreation hook deletion policy
  2. You enable retries on your Application
  3. The pipeline fails all the retries

...the resources created in the PreSync stage get deleted.

To Reproduce

See https://github.com/bcbrockway/argocd-bug.

Expected behavior

Resources created in the PreSync phase with the BeforeHookCreation hook deletion policy set should not be deleted until the start of the next sync.

Version

N/A (not using the CLI). Server version is 2.7.1 (see linked repo above)

Logs

time="2023-06-14T13:50:45Z" level=info msg="Updating operation state. phase: Running -> Running, message: 'one or more synchronization tasks completed unsuccessfully. Retrying attempt #1 at 1:50PM.' -> 'one or more tasks are running'" application=argocd/bugtest syncId=00121-pfMIJ
time="2023-06-14T13:50:45Z" level=info msg="getRepoObjs stats" application=argocd/bugtest build_options_ms=0 helm_ms=0 plugins_ms=0 repo_ms=0 time_ms=506 unmarshal_ms=506 version_ms=0
time="2023-06-14T13:50:45Z" level=info msg="Skipping auto-sync: another operation is in progress" application=argocd/bugtest
time="2023-06-14T13:50:45Z" level=info msg="Update successful" application=argocd/bugtest
time="2023-06-14T13:50:45Z" level=info msg="Reconciliation completed" application=argocd/bugtest dedup_ms=0 dest-name= dest-namespace=bugtest dest-server="https://kubernetes.default.svc" diff_ms=15 fields.level=824708752808 git_ms=506 health_ms=0 live_ms=0 settings_ms=0 sync_ms=0 time_ms=542
time="2023-06-14T13:50:45Z" level=info msg="Refreshing app status (controller refresh requested), level (1)" application=argocd/bugtest
time="2023-06-14T13:50:45Z" level=info msg="Comparing app state (cluster: https://kubernetes.default.svc, namespace: bugtest)" application=argocd/bugtest
time="2023-06-14T13:50:45Z" level=info msg="getRepoObjs stats" application=argocd/bugtest build_options_ms=0 helm_ms=0 plugins_ms=0 repo_ms=0 time_ms=5 unmarshal_ms=5 version_ms=0
time="2023-06-14T13:50:45Z" level=info msg="Skipping auto-sync: another operation is in progress" application=argocd/bugtest
time="2023-06-14T13:50:45Z" level=info msg="Update successful" application=argocd/bugtest
time="2023-06-14T13:50:45Z" level=info msg="Reconciliation completed" application=argocd/bugtest dedup_ms=0 dest-name= dest-namespace=bugtest dest-server="https://kubernetes.default.svc" diff_ms=7 fields.level=1 git_ms=5 health_ms=0 live_ms=0 settings_ms=0 sync_ms=0 time_ms=27
time="2023-06-14T13:50:47Z" level=info msg="sync/terminate complete" application=argocd/bugtest duration=2.1469295s syncId=00121-pfMIJ
time="2023-06-14T13:50:47Z" level=info msg="updated 'argocd/bugtest' operation (phase: Running)"
time="2023-06-14T13:50:47Z" level=info msg="Resuming in-progress operation. phase: Running, message: one or more tasks are running" application=argocd/bugtest
time="2023-06-14T13:50:47Z" level=info msg="Comparing app state (cluster: https://kubernetes.default.svc, namespace: bugtest)" application=argocd/bugtest
time="2023-06-14T13:50:47Z" level=info msg="getRepoObjs stats" application=argocd/bugtest build_options_ms=0 helm_ms=0 plugins_ms=0 repo_ms=0 time_ms=10 unmarshal_ms=9 version_ms=0
time="2023-06-14T13:50:47Z" level=info msg=Syncing application=argocd/bugtest skipHooks=false started=true syncId=00122-UFQXC
time="2023-06-14T13:50:47Z" level=info msg=Tasks application=argocd/bugtest syncId=00122-UFQXC tasks="[PreSync/-1 hook /ServiceAccount:bugtest/shared-sa nil->obj (,Succeeded,shared-sa created), PreSync/-1 hook /Secret:bugtest/shared-secret nil->obj (,Succeeded,shared-secret created), PreSync/0 hook batch/Job:bugtest/pre-sync-job obj->obj (,Succeeded,job.batch/pre-sync-job created), Sync/0 resource rbac.authorization.k8s.io/Role:bugtest/shared-sa obj->obj (Synced,Succeeded,role.rbac.authorization.k8s.io/shared-sa reconciled. role.rbac.authorization.k8s.io/shared-sa unchanged), Sync/0 resource rbac.authorization.k8s.io/RoleBinding:bugtest/shared-sa obj->obj (Synced,Succeeded,rolebinding.rbac.authorization.k8s.io/shared-sa reconciled. rolebinding.rbac.authorization.k8s.io/shared-sa unchanged), Sync/0 resource apps/Deployment:bugtest/deployment obj->obj (Synced,Failed,Deployment \"deployment\" exceeded its progress deadline), PostSync/0 hook batch/Job:bugtest/post-sync-job nil->obj (,,)]"
time="2023-06-14T13:50:47Z" level=info msg="Updating operation state. phase: Running -> Failed, message: 'one or more tasks are running' -> 'one or more synchronization tasks completed unsuccessfully'" application=argocd/bugtest syncId=00122-UFQXC
time="2023-06-14T13:50:47Z" level=info msg="sync/terminate complete" application=argocd/bugtest duration=12.642386ms syncId=00122-UFQXC
time="2023-06-14T13:50:47Z" level=info msg="updated 'argocd/bugtest' operation (phase: Failed)"
time="2023-06-14T13:50:47Z" level=info msg="Sync operation to 157a134a1bf129e57e87e923361b4225b2ef2f79 failed: one or more synchronization tasks completed unsuccessfully (retried 1 times)." application=bugtest dest-namespace=bugtest dest-server="https://kubernetes.default.svc" reason=OperationCompleted type=Warning
time="2023-06-14T13:50:47Z" level=info msg="Refreshing app status (controller refresh requested), level (2)" application=argocd/bugtest
time="2023-06-14T13:50:47Z" level=info msg="Comparing app state (cluster: https://kubernetes.default.svc, namespace: bugtest)" application=argocd/bugtest
time="2023-06-14T13:50:47Z" level=info msg="getRepoObjs stats" application=argocd/bugtest build_options_ms=0 helm_ms=0 plugins_ms=0 repo_ms=0 time_ms=8 unmarshal_ms=7 version_ms=0
time="2023-06-14T13:50:47Z" level=info msg="Skipping auto-sync: application status is Synced" application=argocd/bugtest
time="2023-06-14T13:50:47Z" level=info msg="Update successful" application=argocd/bugtest
time="2023-06-14T13:50:47Z" level=info msg="Reconciliation completed" application=argocd/bugtest dedup_ms=0 dest-name= dest-namespace=bugtest dest-server="https://kubernetes.default.svc" diff_ms=7 fields.level=2 git_ms=8 health_ms=0 live_ms=0 settings_ms=0 sync_ms=0 time_ms=28
bcbrockway commented 1 year ago

Think it's the same as #13429 but I've managed to pin the replication steps down!