changes in initContainer's command doesn't trigger statefulset sync

Please, answer some short questions which should help us to understand your problem / question better?

Which image of the operator are you using? registry.opensource.zalan.do/acid/postgres-operator:v1.8.1
Where do you run it - cloud or metal? Kubernetes or OpenShift? aws k8s v1.23
Are you running Postgres Operator in production? yes
Type of issue? bug report

Hi,

We are using initContainer to tune some kernel settings for nodes running postgres. postgresql manifest looks like this:

apiVersion: acid.zalan.do/v1
kind: postgresql
...
spec:
  ...
  additionalVolumes:
  - mountPath: /host-sys
    name: host-sys
    volumeSource:
      hostPath:
        path: /sys
  initContainers:
  - command:
    - /bin/sh
    - -c
    - |-
      sysctl -w vm.dirty_background_bytes=1600000
      sysctl -w vm.dirty_bytes=3200000
      sysctl -w vm.swappiness=1
      sysctl -w kernel.sched_autogroup_enabled=0
      sysctl -w kernel.sched_migration_cost_ns=5000000
    image: busybox
    name: sysctl-container-revision-1
    securityContext:
      privileged: true
      runAsUser: 0
    volumeMounts:
    - mountPath: /host-sys
      name: host-sys

We noticed that changes in "command" section does not trigger update of statefulset. For example, change of first line to sysctl -w vm.dirty_background_bytes=1600001 produces following log in postgres-operator (and doesn't trigger "syncing statefulsets"):

time="2023-03-10T13:58:33Z" level=info msg="UPDATE event has been queued" cluster-name=island-published/island-postgres pkg=controller worker=0
time="2023-03-10T13:58:33Z" level=info msg="update of the cluster started" cluster-name=island-published/island-postgres pkg=controller worker=0
time="2023-03-10T13:58:33Z" level=debug msg="-    resourceVersion: 16999969," cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T13:58:33Z" level=debug msg="-    generation: 24," cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T13:58:33Z" level=debug msg="+    resourceVersion: 17001340," cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T13:58:33Z" level=debug msg="+    generation: 25," cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T13:58:33Z" level=debug msg="-          sysctl -w vm.dirty_background_bytes=1600000\\nsysctl -w vm.dirty_bytes=3200000\\nsysctl -w vm.swappiness=1\\nsysctl -w kernel.sched_autogroup_enabled=0\\nsysctl -w kernel.sched_migration_cost_ns=5000000" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T13:58:33Z" level=debug msg="+          sysctl -w vm.dirty_background_bytes=1600001\\nsysctl -w vm.dirty_bytes=3200000\\nsysctl -w vm.swappiness=1\\nsysctl -w kernel.sched_autogroup_enabled=0\\nsysctl -w kernel.sched_migration_cost_ns=5000000" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T13:58:33Z" level=info msg="postgresql major version unchanged or smaller, no changes needed" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T13:58:33Z" level=debug msg="initialize users" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T13:58:33Z" level=debug msg="team API is disabled" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T13:58:33Z" level=debug msg="team API is disabled" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T13:58:33Z" level=debug msg="syncing secrets" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T13:58:33Z" level=info msg="syncing secrets" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T13:58:33Z" level=debug msg="syncing volumes using \"pvc\" storage resize mode" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T13:58:33Z" level=info msg="volume claims do not require changes" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T13:58:33Z" level=info msg="Mount additional volumes: [{Name:host-sys MountPath:/host-sys SubPath: TargetContainers:[postgres] VolumeSource:{HostPath:&HostPathVolumeSource{Path:/sys,Type:nil,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}]" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T13:58:34Z" level=info msg="Mount additional volumes: [{Name:host-sys MountPath:/host-sys SubPath: TargetContainers:[postgres] VolumeSource:{HostPath:&HostPathVolumeSource{Path:/sys,Type:nil,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}]" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T13:58:34Z" level=debug msg="syncing statefulsets" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T13:58:34Z" level=info msg="Mount additional volumes: [{Name:host-sys MountPath:/host-sys SubPath: TargetContainers:[postgres] VolumeSource:{HostPath:&HostPathVolumeSource{Path:/sys,Type:nil,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}]" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T13:58:34Z" level=debug msg="making GET http request: http://10.104.14.184:8008/config" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T13:58:46Z" level=debug msg="making GET http request: http://10.104.15.187:8008/patroni" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T13:58:46Z" level=debug msg="making GET http request: http://10.104.14.184:8008/patroni" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T13:58:46Z" level=debug msg="syncing roles" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T13:58:46Z" level=debug msg="closing database connection" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T13:58:46Z" level=debug msg="syncing connection pooler (master, replica) from (true, nil) to (true, nil)" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T13:58:46Z" level=info msg="cluster has been updated" cluster-name=island-published/island-postgres pkg=controller worker=0

To get your changes applied to sts you have to patch any other parameter of manifest, for example if I change initContainer's name to sysctl-container-revision-2 then i can see in operator's log that sts is being updated:

time="2023-03-10T14:01:27Z" level=info msg="UPDATE event has been queued" cluster-name=island-published/island-postgres pkg=controller worker=0
time="2023-03-10T14:01:27Z" level=info msg="update of the cluster started" cluster-name=island-published/island-postgres pkg=controller worker=0
time="2023-03-10T14:01:27Z" level=debug msg="-    resourceVersion: 17001458," cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:27Z" level=debug msg="-    generation: 25," cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:27Z" level=debug msg="+    resourceVersion: 17002746," cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:27Z" level=debug msg="+    generation: 26," cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:27Z" level=debug msg="-        name: sysctl-container-revision-1," cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:27Z" level=debug msg="+        name: sysctl-container-revision-2," cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:27Z" level=info msg="postgresql major version unchanged or smaller, no changes needed" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:27Z" level=debug msg="initialize users" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:27Z" level=debug msg="team API is disabled" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:27Z" level=debug msg="team API is disabled" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:27Z" level=debug msg="syncing secrets" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:27Z" level=info msg="syncing secrets" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:28Z" level=debug msg="syncing volumes using \"pvc\" storage resize mode" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:28Z" level=info msg="volume claims do not require changes" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:28Z" level=info msg="Mount additional volumes: [{Name:host-sys MountPath:/host-sys SubPath: TargetContainers:[postgres] VolumeSource:{HostPath:&HostPathVolumeSource{Path:/sys,Type:nil,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}]" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:28Z" level=info msg="Mount additional volumes: [{Name:host-sys MountPath:/host-sys SubPath: TargetContainers:[postgres] VolumeSource:{HostPath:&HostPathVolumeSource{Path:/sys,Type:nil,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}]" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:28Z" level=debug msg="syncing statefulsets" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:29Z" level=info msg="Mount additional volumes: [{Name:host-sys MountPath:/host-sys SubPath: TargetContainers:[postgres] VolumeSource:{HostPath:&HostPathVolumeSource{Path:/sys,Type:nil,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}]" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:29Z" level=debug msg="mark rolling update annotation for island-postgres-0: reason pod changes" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:29Z" level=debug msg="mark rolling update annotation for island-postgres-1: reason pod changes" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:29Z" level=info msg="statefulset island-published/island-postgres is not in the desired state and needs to be updated" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:29Z" level=debug msg="-            path: /sys," cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:29Z" level=debug msg="-            type: " cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:29Z" level=debug msg="+            path: /sys" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:29Z" level=debug msg="-          name: sysctl-container-revision-1," cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:29Z" level=debug msg="+          name: sysctl-container-revision-2," cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:29Z" level=debug msg="-            sysctl -w vm.dirty_background_bytes=1600000\\nsysctl -w vm.dirty_bytes=3200000\\nsysctl -w vm.swappiness=1\\nsysctl -w kernel.sched_autogroup_enabled=0\\nsysctl -w kernel.sched_migration_cost_ns=5000000" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:29Z" level=debug msg="+            sysctl -w vm.dirty_background_bytes=1600001\\nsysctl -w vm.dirty_bytes=3200000\\nsysctl -w vm.swappiness=1\\nsysctl -w kernel.sched_autogroup_enabled=0\\nsysctl -w kernel.sched_migration_cost_ns=5000000" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:29Z" level=debug msg="-          terminationMessagePath: /dev/termination-log," cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:29Z" level=debug msg="-          terminationMessagePolicy: File," cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:29Z" level=debug msg="-          imagePullPolicy: IfNotPresent," cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:29Z" level=debug msg="-              port: 8008," cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:29Z" level=debug msg="-              scheme: HTTP" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:29Z" level=debug msg="+              port: 8008" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:29Z" level=debug msg="-          terminationMessagePath: /dev/termination-log," cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:29Z" level=debug msg="-          terminationMessagePolicy: File," cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:29Z" level=debug msg="-      restartPolicy: Always," cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:29Z" level=debug msg="-      dnsPolicy: ClusterFirst," cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:29Z" level=debug msg="-      serviceAccount: default," cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:29Z" level=debug msg="-      schedulerName: default-scheduler," cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:29Z" level=debug msg="-      kind: PersistentVolumeClaim," cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:29Z" level=debug msg="-      apiVersion: v1," cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:29Z" level=debug msg="-      status: {" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:29Z" level=debug msg="-        phase: Pending" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:29Z" level=debug msg="-      }" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:29Z" level=debug msg="+      status: {}" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:29Z" level=debug msg="-  }," cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:29Z" level=debug msg="-  revisionHistoryLimit: 10" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:29Z" level=debug msg="+  }" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:29Z" level=info msg="reason: new statefulset initContainers's sysctl-container-revision-1 (index 0) name does not match the current one" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:29Z" level=debug msg="updating statefulset" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:29Z" level=debug msg="making GET http request: http://10.104.14.184:8008/config" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:41Z" level=debug msg="making GET http request: http://10.104.15.187:8008/patroni" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:41Z" level=debug msg="making GET http request: http://10.104.14.184:8008/patroni" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:41Z" level=debug msg="performing rolling update" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:41Z" level=info msg="there are 2 pods in the cluster to recreate" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:41Z" level=debug msg="subscribing to pod \"island-published/island-postgres-1\"" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:02:06Z" level=info msg="pod \"island-published/island-postgres-1\" has been recreated" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:02:06Z" level=debug msg="unsubscribing from pod \"island-published/island-postgres-1\" events" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:02:06Z" level=debug msg="making GET http request: http://10.104.14.184:8008/cluster" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:02:06Z" level=error msg="could not sync statefulsets: could not recreate pods: skipping switchover: no switchover candidate found" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:02:06Z" level=debug msg="syncing roles" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:02:06Z" level=debug msg="closing database connection" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:02:06Z" level=debug msg="syncing connection pooler (master, replica) from (true, nil) to (true, nil)" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:02:06Z" level=info msg="cluster has been updated" cluster-name=island-published/island-postgres pkg=controller worker=0

zalando / postgres-operator

changes in initContainer's command doesn't trigger statefulset sync #2259