[BUG] Restarting a rollout with workloadRef crashes the operator pod

jorgelon commented 1 month ago

Describe the bug There are 2 ways to create a rollout in argo rollouts:

The first one is with a single resource "rollout". This includes both logics: the deployment spec and the rollout spec. With the new 1.1.0 reloader release, restarting this rollout works great (deleting the pods and not creating a new replicaset)
The second one is with 2 resources. A rollout including the rollout spec and calling an existing deployment with spec.workloadRef. And the referenced deployment with the deployment spec. Restarting this rollout makes the controller crash.

To Reproduce Change the secret that reloads the rollout deployed via workloadRef

Expected behavior It works same ways as with a single rollout resource

Environment

Operator Version: v1.1.0
Kubernetes/OpenShift Version: 1.29.6

Log (via stern) reloader-7df5bc9f58-phgjj reloader E0918 08:33:15.113227 1 runtime.go:79] Observed a panic: runtime.boundsError{x:0, y:0, signed:true, code:0x0} (runtime error: index out of range [0] with length 0) reloader-7df5bc9f58-phgjj reloader goroutine 12 [running]: reloader-7df5bc9f58-phgjj reloader k8s.io/apimachinery/pkg/util/runtime.logPanic({0x19f31e0?, 0xc001912c48}) reloader-7df5bc9f58-phgjj reloader /go/pkg/mod/k8s.io/apimachinery@v0.29.3/pkg/util/runtime/runtime.go:75 +0x85 reloader-7df5bc9f58-phgjj reloader k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc000cc6380?}) reloader-7df5bc9f58-phgjj reloader /go/pkg/mod/k8s.io/apimachinery@v0.29.3/pkg/util/runtime/runtime.go:49 +0x6b reloader-7df5bc9f58-phgjj reloader panic({0x19f31e0?, 0xc001912c48?}) reloader-7df5bc9f58-phgjj reloader /usr/local/go/src/runtime/panic.go:914 +0x21f reloader-7df5bc9f58-phgjj reloader github.com/stakater/Reloader/internal/pkg/handler.getContainerUsingResource({0x1c33320, 0x1c33308, 0x1c33328, 0x1c33310, 0x1c33318, 0x1c33380, 0x1c33330, {0x1b0f268, 0x7}}, {0x1da78c8, ...}, ...) reloader-7df5bc9f58-phgjj reloader /workspace/internal/pkg/handler/upgrade.go:382 +0x238 reloader-7df5bc9f58-phgjj reloader github.com/stakater/Reloader/internal/pkg/handler.updatePodAnnotations({0x1c33320, 0x1c33308, 0x1c33328, 0x1c33310, 0x1c33318, 0x1c33380, 0x1c33330, {0x1b0f268, 0x7}}, {0x1da78c8, ...}, ...) reloader-7df5bc9f58-phgjj reloader /workspace/internal/pkg/handler/upgrade.go:399 +0xec reloader-7df5bc9f58-phgjj reloader github.com/stakater/Reloader/internal/pkg/handler.invokeReloadStrategy({0x1c33320, 0x1c33308, 0x1c33328, 0x1c33310, 0x1c33318, 0x1c33380, 0x1c33330, {0x1b0f268, 0x7}}, {0x1da78c8, ...}, ...) reloader-7df5bc9f58-phgjj reloader /workspace/internal/pkg/handler/upgrade.go:392 +0x12d reloader-7df5bc9f58-phgjj reloader github.com/stakater/Reloader/internal/pkg/handler.PerformAction({{0x1dd0348, 0xc00090d1e0}, {0x1da5f00, 0x0}, {0x1da5f28, 0xc000c26c10}}, {{0xc000c37350, 0x18}, {0xc000986eb0, 0xa}, ...}, ...) reloader-7df5bc9f58-phgjj reloader /workspace/internal/pkg/handler/upgrade.go:222 +0x16b0 reloader-7df5bc9f58-phgjj reloader github.com/stakater/Reloader/internal/pkg/handler.rollingUpgrade({{0x1dd0348, 0xc00090d1e0}, {0x1da5f00, 0x0}, {0x1da5f28, 0xc000c26c10}}, {{0xc000c37350, 0x18}, {0xc000986eb0, 0xa}, ...}, ...) reloader-7df5bc9f58-phgjj reloader /workspace/internal/pkg/handler/upgrade.go:184 +0x128 reloader-7df5bc9f58-phgjj reloader github.com/stakater/Reloader/internal/pkg/handler.doRollingUpgrade({{0xc000c37350, 0x18}, {0xc000986eb0, 0xa}, 0xc0006b4cf0, {0x1b33cca, 0x23}, {0x1b30b76, 0x21}, {0xc000b3e3f0, ...}, ...}, ...) reloader-7df5bc9f58-phgjj reloader /workspace/internal/pkg/handler/upgrade.go:173 +0xb0c reloader-7df5bc9f58-phgjj reloader github.com/stakater/Reloader/internal/pkg/handler.ResourceUpdatedHandler.Handle({{0x1add080, 0xc000688140}, {0x1add080, 0xc001892b40}, {0xc00043c208, 0xc00043c220}, {0x7f056fc315e0, 0xc000437d00}}) reloader-7df5bc9f58-phgjj reloader /workspace/internal/pkg/handler/update.go:32 +0x265 reloader-7df5bc9f58-phgjj reloader github.com/stakater/Reloader/internal/pkg/controller.(Controller).processNextItem(0xc000432d80) reloader-7df5bc9f58-phgjj reloader /workspace/internal/pkg/controller/controller.go:246 +0xd0 reloader-7df5bc9f58-phgjj reloader github.com/stakater/Reloader/internal/pkg/controller.(Controller).runWorker(...) reloader-7df5bc9f58-phgjj reloader /workspace/internal/pkg/controller/controller.go:230 reloader-7df5bc9f58-phgjj reloader k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?) reloader-7df5bc9f58-phgjj reloader /go/pkg/mod/k8s.io/apimachinery@v0.29.3/pkg/util/wait/backoff.go:226 +0x33 reloader-7df5bc9f58-phgjj reloader k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000114000?, {0x1da1620, 0xc00019baa0}, 0x1, 0xc000534000) reloader-7df5bc9f58-phgjj reloader /go/pkg/mod/k8s.io/apimachinery@v0.29.3/pkg/util/wait/backoff.go:227 +0xaf reloader-7df5bc9f58-phgjj reloader k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x0?, 0x3b9aca00, 0x0, 0x0?, 0x9bfa40?) reloader-7df5bc9f58-phgjj reloader /go/pkg/mod/k8s.io/apimachinery@v0.29.3/pkg/util/wait/backoff.go:204 +0x7f reloader-7df5bc9f58-phgjj reloader k8s.io/apimachinery/pkg/util/wait.Until(0xc00035b7d0?, 0x9b8525?, 0xc000254600?) reloader-7df5bc9f58-phgjj reloader /go/pkg/mod/k8s.io/apimachinery@v0.29.3/pkg/util/wait/backoff.go:161 +0x1e reloader-7df5bc9f58-phgjj reloader created by github.com/stakater/Reloader/internal/pkg/controller.(Controller).Run in goroutine 66 reloader-7df5bc9f58-phgjj reloader /workspace/internal/pkg/controller/controller.go:215 +0x259 reloader-7df5bc9f58-phgjj reloader panic: runtime error: index out of range [0] with length 0 [recovered] reloader-7df5bc9f58-phgjj reloader panic: runtime error: index out of range [0] with length 0 reloader-7df5bc9f58-phgjj reloader reloader-7df5bc9f58-phgjj reloader goroutine 12 [running]: reloader-7df5bc9f58-phgjj reloader k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc000cc6380?}) reloader-7df5bc9f58-phgjj reloader /go/pkg/mod/k8s.io/apimachinery@v0.29.3/pkg/util/runtime/runtime.go:56 +0xcd reloader-7df5bc9f58-phgjj reloader panic({0x19f31e0?, 0xc001912c48?}) reloader-7df5bc9f58-phgjj reloader /usr/local/go/src/runtime/panic.go:914 +0x21f reloader-7df5bc9f58-phgjj reloader github.com/stakater/Reloader/internal/pkg/handler.getContainerUsingResource({0x1c33320, 0x1c33308, 0x1c33328, 0x1c33310, 0x1c33318, 0x1c33380, 0x1c33330, {0x1b0f268, 0x7}}, {0x1da78c8, ...}, ...) reloader-7df5bc9f58-phgjj reloader /workspace/internal/pkg/handler/upgrade.go:382 +0x238 reloader-7df5bc9f58-phgjj reloader github.com/stakater/Reloader/internal/pkg/handler.updatePodAnnotations({0x1c33320, 0x1c33308, 0x1c33328, 0x1c33310, 0x1c33318, 0x1c33380, 0x1c33330, {0x1b0f268, 0x7}}, {0x1da78c8, ...}, ...) reloader-7df5bc9f58-phgjj reloader /workspace/internal/pkg/handler/upgrade.go:399 +0xec reloader-7df5bc9f58-phgjj reloader github.com/stakater/Reloader/internal/pkg/handler.invokeReloadStrategy({0x1c33320, 0x1c33308, 0x1c33328, 0x1c33310, 0x1c33318, 0x1c33380, 0x1c33330, {0x1b0f268, 0x7}}, {0x1da78c8, ...}, ...) reloader-7df5bc9f58-phgjj reloader /workspace/internal/pkg/handler/upgrade.go:392 +0x12d reloader-7df5bc9f58-phgjj reloader github.com/stakater/Reloader/internal/pkg/handler.PerformAction({{0x1dd0348, 0xc00090d1e0}, {0x1da5f00, 0x0}, {0x1da5f28, 0xc000c26c10}}, {{0xc000c37350, 0x18}, {0xc000986eb0, 0xa}, ...}, ...) reloader-7df5bc9f58-phgjj reloader /workspace/internal/pkg/handler/upgrade.go:222 +0x16b0 reloader-7df5bc9f58-phgjj reloader github.com/stakater/Reloader/internal/pkg/handler.rollingUpgrade({{0x1dd0348, 0xc00090d1e0}, {0x1da5f00, 0x0}, {0x1da5f28, 0xc000c26c10}}, {{0xc000c37350, 0x18}, {0xc000986eb0, 0xa}, ...}, ...) reloader-7df5bc9f58-phgjj reloader /workspace/internal/pkg/handler/upgrade.go:184 +0x128 reloader-7df5bc9f58-phgjj reloader github.com/stakater/Reloader/internal/pkg/handler.doRollingUpgrade({{0xc000c37350, 0x18}, {0xc000986eb0, 0xa}, 0xc0006b4cf0, {0x1b33cca, 0x23}, {0x1b30b76, 0x21}, {0xc000b3e3f0, ...}, ...}, ...) reloader-7df5bc9f58-phgjj reloader /workspace/internal/pkg/handler/upgrade.go:173 +0xb0c reloader-7df5bc9f58-phgjj reloader github.com/stakater/Reloader/internal/pkg/handler.ResourceUpdatedHandler.Handle({{0x1add080, 0xc000688140}, {0x1add080, 0xc001892b40}, {0xc00043c208, 0xc00043c220}, {0x7f056fc315e0, 0xc000437d00}}) reloader-7df5bc9f58-phgjj reloader /workspace/internal/pkg/handler/update.go:32 +0x265 reloader-7df5bc9f58-phgjj reloader github.com/stakater/Reloader/internal/pkg/controller.(Controller).processNextItem(0xc000432d80) reloader-7df5bc9f58-phgjj reloader /workspace/internal/pkg/controller/controller.go:246 +0xd0 reloader-7df5bc9f58-phgjj reloader github.com/stakater/Reloader/internal/pkg/controller.(Controller).runWorker(...) reloader-7df5bc9f58-phgjj reloader /workspace/internal/pkg/controller/controller.go:230 reloader-7df5bc9f58-phgjj reloader k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?) reloader-7df5bc9f58-phgjj reloader /go/pkg/mod/k8s.io/apimachinery@v0.29.3/pkg/util/wait/backoff.go:226 +0x33 reloader-7df5bc9f58-phgjj reloader k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000114000?, {0x1da1620, 0xc00019baa0}, 0x1, 0xc000534000) reloader-7df5bc9f58-phgjj reloader /go/pkg/mod/k8s.io/apimachinery@v0.29.3/pkg/util/wait/backoff.go:227 +0xaf reloader-7df5bc9f58-phgjj reloader k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x0?, 0x3b9aca00, 0x0, 0x0?, 0x9bfa40?) reloader-7df5bc9f58-phgjj reloader /go/pkg/mod/k8s.io/apimachinery@v0.29.3/pkg/util/wait/backoff.go:204 +0x7f reloader-7df5bc9f58-phgjj reloader k8s.io/apimachinery/pkg/util/wait.Until(0xc00035b7d0?, 0x9b8525?, 0xc000254600?) reloader-7df5bc9f58-phgjj reloader /go/pkg/mod/k8s.io/apimachinery@v0.29.3/pkg/util/wait/backoff.go:161 +0x1e reloader-7df5bc9f58-phgjj reloader created by github.com/stakater/Reloader/internal/pkg/controller.(Controller).Run in goroutine 66 reloader-7df5bc9f58-phgjj reloader /workspace/internal/pkg/controller/controller.go:215 +0x259

MuneebAijaz commented 1 month ago

it seems the code was written for spec.template.spec.containers and not for spec.workloadRef. This would be an enhancement i believe. PRs are welcome

praddy26 commented 1 month ago

Hi @MuneebAijaz

I would like to work on this issue. Could you please assign it to me?

mikly-te commented 1 week ago

Seeing this issue as well -- any ideas when this can be fixed?

wololowarrior commented 1 week ago

@mikly-te I can attempt to fix this. But im kinda struggling to recreate the issue. Could you share the manifest files if possible to have a minimum reproducible example?

mikly-te commented 1 week ago

@wololowarrior here's a stripped down version of the manifest files, although not sure if it's minimal or not.

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  annotations:
    secret.reloader.stakater.com/reload: webapps
  name: webapps
spec:
  replicas: 6
  selector:
    matchLabels:
      app: webapps
  strategy:
    canary:
      canaryService: webapps-canary
      scaleDownDelaySeconds: 300
      stableService: webapps
      steps:
        - setCanaryScale:
            weight: 100
        - setWeight: 0
        - pause:
            duration: 5m
        - setWeight: 100
        - pause:
            duration: 1m
      trafficRouting:
        istio:
          virtualService:
            name: webapps
            routes:
              - primary
  workloadRef:
    apiVersion: apps/v1
    kind: Deployment
    name: webapps

apiVersion: apps/v1
kind: Deployment
metadata:
  name: webapps
spec:
  replicas: 0
  selector:
    matchLabels:
      name: webapps
  template:
    metadata:
    spec:
      containers:
        - env:
            - name: CONFIG_APP_NAME
              value: webapps
          envFrom:
            - secretRef:
                name: webapps
          image: webapps
          name: application

wololowarrior commented 1 week ago

Thanks @mikly-te ! Appreciate it I'm also using a similar deployment approach.
For me reloader isnt crashing, but it isnt restarting the pods as well. I guess that's what the issue is. Will try to fix this.

Is reloader crashing for you?

jorgelon commented 6 days ago

This makes the controller crash

apiVersion: apps/v1
kind: Deployment
metadata:
  name: deployment
  labels:
    app: deployment
spec:
  replicas: 0
  selector:
    matchLabels:
      app: deployment
  template:
    metadata:
      labels:
        app: deployment
    spec:
      containers:
      - image: docker.io/nginx:mainline-alpine3.20-slim
        name: nginx

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: deployment
  labels:
    app: deployment
  annotations:
    secret.reloader.stakater.com/reload: "deployment"
spec:
  selector:
    matchLabels:
      app: deployment
  workloadRef: 
    apiVersion: apps/v1
    kind: Deployment
    name: deployment
  strategy:
    blueGreen:
      activeService: deployment
      autoPromotionEnabled: false
      activeMetadata:
        labels:
          role: active

apiVersion: v1
kind: Service
metadata:
  name: deployment
spec:
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: http

apiVersion: v1
kind: Secret
metadata:
  name: deployment
stringData:
  VARIABLE: "whatever"
type: Opaque

stakater / Reloader

[BUG] Restarting a rollout with workloadRef crashes the operator pod #751