backube / volsync

Asynchronous data replication for Kubernetes volumes
https://volsync.readthedocs.io
GNU Affero General Public License v3.0
587 stars 67 forks source link

Environmental Variable Passthrough to Mover with Restic #722

Closed Rojikku closed 1 year ago

Rojikku commented 1 year ago

Describe the feature you'd like to have. I would like to be able to pass environmental variables to the mover pod.

What is the value to the end user? (why is it a priority?) Restic allows you to use rclone as a backup method, but there's not really an rclone config option unless you're using rclone. Rclone allows you to do a full config via variables

How will we know we have a good solution? (acceptance criteria) Ability to pass arbitrarily named ENV variables to mover pod so that restic/rclone will recognize them.

Additional context Example I believe I configured that correctly, but I think the issue is that I took "Variables will correspond" from the docs too literally...and it doesn't just pass through everything.

Honestly, the more I look at this, the more I'm confused why it doesn't work. RESTIC_PASSWORD and RESTIC_REPOSITORY are both environment variables... Did you MANUALLY pass them through instead of passing through all portions of the secret as variables as a blanket? Pretty sure a blanket pass through would work fine for this use case. Honestly, either way. I'd be happy with anything as long as I can pass through my RCLONE config options.

tesshuflower commented 1 year ago

You're correct that only specific env vars are passed through - the RCLONE_ ones are not. See https://github.com/backube/volsync/blob/ea2d597b7a5d9bf9976635b23a2590160cf421c9/controllers/mover/restic/mover.go#L335-L381

I think that to use rclone this is a bigger change than simply passing through env vars, as restic will also need to know the location of rclone (possibly other flags are required as well?). This wouldn't have been possible previously - but now that the movers are all consolidated into one volsync image, the rclone binary will now be present on the machine. Actually I looked and rclone is put into /usr/local/bin/rclone so maybe will already be present in the path so one less hurdle.

Would passing through all env vars that start with RCLONE_ be enough?

Rojikku commented 1 year ago

Hey, thanks for looking into this for me!

Yes, as far as I am aware, passing through all env vars that start with RCLONE_ WOULD be enough. The idea crossed my mind as well.

Not sure about other flags being required, but I think I can specify everything via the RCLONE_* variables. At least, everything that can be in the config, which should be enough.

If there's an image I can deploy, I can do testing if needed! Or you could copy my variables from the linked config above, if that helps you in your testing.

On Wed, Apr 26, 2023 at 12:07 PM Tesshu Flower @.***> wrote:

You're correct that only specific env vars are passed through - the RCLONE_ ones are not. See https://github.com/backube/volsync/blob/ea2d597b7a5d9bf9976635b23a2590160cf421c9/controllers/mover/restic/mover.go#L335-L381

I think that to use rclone this is a bigger change than simply passing through env vars, as restic will also need to know the location of rclone (possibly other flags are required as well?). This wouldn't have been possible previously - but now that the movers are all consolidated into one volsync image, the rclone binary will now be present on the machine. Actually I looked and rclone is put into /usr/local/bin/rclone so maybe will already be present in the path so one less hurdle.

Would passing through all env vars that start with RCLONE_ be enough?

— Reply to this email directly, view it on GitHub https://github.com/backube/volsync/issues/722#issuecomment-1523680505, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAMZGJD2NPRYMUFVXH7PVEDXDFB3ZANCNFSM6AAAAAAXHI72MY . You are receiving this because you authored the thread.Message ID: @.***>

tesshuflower commented 1 year ago

@Rojikku - I've built an image (untested) you can try out: quay.io/tflower/volsync:restic-rclone-env

source is here: https://github.com/backube/volsync/compare/main...tesshuflower:volsync:restic_rclone_env_vars

Rojikku commented 1 year ago

@tesshuflower

TL;DR Seems to work, but there's errors on the replicationsource that cause it to re-make the job every second or so. Each backup succeeds, but I can't get logs because all pods are terminating. Kinda confused. Also I don't quite need a backup every second.

From describe replicationsource

Status:
  Conditions:
    Last Transition Time:  2023-04-28T19:46:30Z
    Message:               Operation cannot be fulfilled on jobs.batch "volsync-src-zwave-js-ui": the object has been modified; please apply your changes to the latest version and try again
    Reason:                Error
    Status:                False
    Type:                  Synchronizing
  Last Sync Start Time:    2023-04-28T19:42:33Z

Status:
  Conditions:
    Last Transition Time:  2023-04-28T19:47:05Z
    Message:               unable to update object. Deleting object so it can be recreated
    Reason:                Error
    Status:                False
    Type:                  Synchronizing
  Last Sync Start Time:    2023-04-28T19:42:33Z

From describe job:

Name:             volsync-src-zwave-js-ui
Namespace:        hass
Selector:         controller-uid=3587d5a5-80c1-4c5b-a195-a056e7f7553b
Labels:           app.kubernetes.io/created-by=volsync
                  volsync.backube/cleanup=43b9c3dc-3228-4106-a6d9-ec1ad9486c98
Annotations:      batch.kubernetes.io/job-tracking: 
Controlled By:    ReplicationSource/zwave-js-ui
Parallelism:      1
Completions:      <unset>
Completion Mode:  NonIndexed
Start Time:       Fri, 28 Apr 2023 15:48:17 -0400
Pods Statuses:    1 Active (0 Ready) / 0 Succeeded / 0 Failed
Pod Template:
  Labels:           app.kubernetes.io/created-by=volsync
                    controller-uid=3587d5a5-80c1-4c5b-a195-a056e7f7553b
                    job-name=volsync-src-zwave-js-ui
  Service Account:  volsync-src-zwave-js-ui
  Containers:
   restic:
    Image:      quay.io/backube/volsync:0.7.1
    Port:       <none>
    Host Port:  <none>
    Command:
      /mover-restic/entry.sh
    Args:
      backup
    Environment:
      FORGET_OPTIONS:                      --keep-daily 10 --keep-weekly 6 --keep-monthly 4 --keep-within 3d
      DATA_DIR:                           /data
      RESTIC_CACHE_DIR:                   /cache
      RESTORE_AS_OF:                      
      SELECT_PREVIOUS:                    0
      RESTIC_REPOSITORY:                  <set to the key 'RESTIC_REPOSITORY' in secret 'restic-conf-zwave'>                  Optional: false
      RESTIC_PASSWORD:                    <set to the key 'RESTIC_PASSWORD' in secret 'restic-conf-zwave'>                    Optional: false
      RESTIC_COMPRESSION:                 <set to the key 'RESTIC_COMPRESSION' in secret 'restic-conf-zwave'>                 Optional: true
      RESTIC_PACK_SIZE:                   <set to the key 'RESTIC_PACK_SIZE' in secret 'restic-conf-zwave'>                   Optional: true
      RESTIC_READ_CONCURRENCY:            <set to the key 'RESTIC_READ_CONCURRENCY' in secret 'restic-conf-zwave'>            Optional: true
      AWS_ACCESS_KEY_ID:                  <set to the key 'AWS_ACCESS_KEY_ID' in secret 'restic-conf-zwave'>                  Optional: true
      AWS_SECRET_ACCESS_KEY:              <set to the key 'AWS_SECRET_ACCESS_KEY' in secret 'restic-conf-zwave'>              Optional: true
      AWS_SESSION_TOKEN:                  <set to the key 'AWS_SESSION_TOKEN' in secret 'restic-conf-zwave'>                  Optional: true
      AWS_DEFAULT_REGION:                 <set to the key 'AWS_DEFAULT_REGION' in secret 'restic-conf-zwave'>                 Optional: true
      ST_AUTH:                            <set to the key 'ST_AUTH' in secret 'restic-conf-zwave'>                            Optional: true
      ST_USER:                            <set to the key 'ST_USER' in secret 'restic-conf-zwave'>                            Optional: true
      ST_KEY:                             <set to the key 'ST_KEY' in secret 'restic-conf-zwave'>                             Optional: true
      OS_AUTH_URL:                        <set to the key 'OS_AUTH_URL' in secret 'restic-conf-zwave'>                        Optional: true
      OS_REGION_NAME:                     <set to the key 'OS_REGION_NAME' in secret 'restic-conf-zwave'>                     Optional: true
      OS_USERNAME:                        <set to the key 'OS_USERNAME' in secret 'restic-conf-zwave'>                        Optional: true
      OS_USER_ID:                         <set to the key 'OS_USER_ID' in secret 'restic-conf-zwave'>                         Optional: true
      OS_PASSWORD:                        <set to the key 'OS_PASSWORD' in secret 'restic-conf-zwave'>                        Optional: true
      OS_TENANT_ID:                       <set to the key 'OS_TENANT_ID' in secret 'restic-conf-zwave'>                       Optional: true
      OS_TENANT_NAME:                     <set to the key 'OS_TENANT_NAME' in secret 'restic-conf-zwave'>                     Optional: true
      OS_USER_DOMAIN_NAME:                <set to the key 'OS_USER_DOMAIN_NAME' in secret 'restic-conf-zwave'>                Optional: true
      OS_USER_DOMAIN_ID:                  <set to the key 'OS_USER_DOMAIN_ID' in secret 'restic-conf-zwave'>                  Optional: true
      OS_PROJECT_NAME:                    <set to the key 'OS_PROJECT_NAME' in secret 'restic-conf-zwave'>                    Optional: true
      OS_PROJECT_DOMAIN_NAME:             <set to the key 'OS_PROJECT_DOMAIN_NAME' in secret 'restic-conf-zwave'>             Optional: true
      OS_PROJECT_DOMAIN_ID:               <set to the key 'OS_PROJECT_DOMAIN_ID' in secret 'restic-conf-zwave'>               Optional: true
      OS_TRUST_ID:                        <set to the key 'OS_TRUST_ID' in secret 'restic-conf-zwave'>                        Optional: true
      OS_APPLICATION_CREDENTIAL_ID:       <set to the key 'OS_APPLICATION_CREDENTIAL_ID' in secret 'restic-conf-zwave'>       Optional: true
      OS_APPLICATION_CREDENTIAL_NAME:     <set to the key 'OS_APPLICATION_CREDENTIAL_NAME' in secret 'restic-conf-zwave'>     Optional: true
      OS_APPLICATION_CREDENTIAL_SECRET:   <set to the key 'OS_APPLICATION_CREDENTIAL_SECRET' in secret 'restic-conf-zwave'>   Optional: true
      OS_STORAGE_URL:                     <set to the key 'OS_STORAGE_URL' in secret 'restic-conf-zwave'>                     Optional: true
      OS_AUTH_TOKEN:                      <set to the key 'OS_AUTH_TOKEN' in secret 'restic-conf-zwave'>                      Optional: true
      B2_ACCOUNT_ID:                      <set to the key 'B2_ACCOUNT_ID' in secret 'restic-conf-zwave'>                      Optional: true
      B2_ACCOUNT_KEY:                     <set to the key 'B2_ACCOUNT_KEY' in secret 'restic-conf-zwave'>                     Optional: true
      AZURE_ACCOUNT_NAME:                 <set to the key 'AZURE_ACCOUNT_NAME' in secret 'restic-conf-zwave'>                 Optional: true
      AZURE_ACCOUNT_KEY:                  <set to the key 'AZURE_ACCOUNT_KEY' in secret 'restic-conf-zwave'>                  Optional: true
      AZURE_ACCOUNT_SAS:                  <set to the key 'AZURE_ACCOUNT_SAS' in secret 'restic-conf-zwave'>                  Optional: true
      GOOGLE_PROJECT_ID:                  <set to the key 'GOOGLE_PROJECT_ID' in secret 'restic-conf-zwave'>                  Optional: true
      RCLONE_CONFIG_NAS_MD5SUM_COMMAND:   <set to the key 'RCLONE_CONFIG_NAS_MD5SUM_COMMAND' in secret 'restic-conf-zwave'>   Optional: true
      RCLONE_CONFIG_NAS_PORT:             <set to the key 'RCLONE_CONFIG_NAS_PORT' in secret 'restic-conf-zwave'>             Optional: true
      RCLONE_CONFIG_NAS_PASS:             <set to the key 'RCLONE_CONFIG_NAS_PASS' in secret 'restic-conf-zwave'>             Optional: true
      RCLONE_CONFIG_NAS_SHA1SUM_COMMAND:  <set to the key 'RCLONE_CONFIG_NAS_SHA1SUM_COMMAND' in secret 'restic-conf-zwave'>  Optional: true
      RCLONE_CONFIG_NAS_SHELL_TYPE:       <set to the key 'RCLONE_CONFIG_NAS_SHELL_TYPE' in secret 'restic-conf-zwave'>       Optional: true
      RCLONE_CONFIG_NAS_TYPE:             <set to the key 'RCLONE_CONFIG_NAS_TYPE' in secret 'restic-conf-zwave'>             Optional: true
      RCLONE_CONFIG_NAS_USER:             <set to the key 'RCLONE_CONFIG_NAS_USER' in secret 'restic-conf-zwave'>             Optional: true
      RCLONE_CONFIG_NAS_HOST:             <set to the key 'RCLONE_CONFIG_NAS_HOST' in secret 'restic-conf-zwave'>             Optional: true
      PRIVILEGED_MOVER:                   0
    Mounts:
      /cache from cache (rw)
      /data from data (rw)
      /tmp from tempdir (rw)
  Volumes:
   data:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  volsync-zwave-js-ui-src
    ReadOnly:   false
   cache:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  volsync-zwave-js-ui-cache
    ReadOnly:   false
   tempdir:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     Memory
    SizeLimit:  <unset>
Events:
  Type    Reason            Age   From            Message
  ----    ------            ----  ----            -------
  Normal  SuccessfulCreate  0s    job-controller  Created pod: volsync-src-zwave-js-ui-bvmsl

So, from what I can tell, it's angry at the job (for some reason?) and therefore deletes and remakes it. A lot. As a result I have a wall of terminating pods in my namespace.

Looks like they have a storage mount error because so many exist they can't share the same storage. Also I have 363 snapshots in restic now. My cron is set to once a half hour. So that's interesting.

So I figured I'd prune those, but:

repository 46b9c8a0 opened (version 2, compression level auto)
Fatal: unable to create lock in backend: repository is already locked by PID 19 on volsync-src-zwave-js-ui-w8gp7 by  (UID 0, GID 0)
lock was created at 2023-04-28 19:36:16 (17m16.777376427s ago)
storage ID caea602d

Looks like this might be part of the issue, maybe?

I was able to fix this with

restic -r sftp:k8s@nas:/k8s/r/hass/zwave unlock --remove-all

Also I suspended it, deleted it, waiting for it to clear out the pods, then let it make it again.

However, creating the replicationsource again produces the same issue... Describe of pod that's terminating:

Events:
  Type     Reason              Age   From                     Message
  ----     ------              ----  ----                     -------
  Normal   Scheduled           13s   default-scheduler        Successfully assigned hass/volsync-src-zwave-js-ui-zmnnf to k8s-3
  Warning  FailedAttachVolume  14s   attachdetach-controller  Multi-Attach error for volume "pvc-07434137-a269-4927-92f9-53ab07aa76c3" Volume is already exclusively attached to one node and can't be attached to another
  Warning  FailedAttachVolume  14s   attachdetach-controller  Multi-Attach error for volume "pvc-80159c94-525d-4298-9c5f-2f320fc5bfc4" Volume is already exclusively attached to one node and can't be attached to another
  Normal   Pulled              6s    kubelet                  Container image "quay.io/backube/volsync:0.7.1" already present on machine
  Normal   Created             5s    kubelet                  Created container restic
  Normal   Started             4s    kubelet                  Started container restic
  Normal   Killing             4s    kubelet                  Stopping container restic

Snapshots list:

repository 46b9c8a0 opened (version 2, compression level auto)
ID        Time                 Host        Tags        Paths
------------------------------------------------------------
f2d2dcc4  2023-04-28 15:50:29  volsync                 /data
96171e9d  2023-04-28 15:58:31  volsync                 /data
906f1d58  2023-04-28 15:58:33  volsync                 /data
2f11445d  2023-04-28 15:58:34  volsync                 /data
cf563533  2023-04-28 15:58:34  volsync                 /data
fc9893fd  2023-04-28 15:58:39  volsync                 /data
9c71ef6e  2023-04-28 15:58:42  volsync                 /data
a0a69085  2023-04-28 15:58:45  volsync                 /data
261d65aa  2023-04-28 15:58:45  volsync                 /data
d0d97009  2023-04-28 15:58:45  volsync                 /data
7304d206  2023-04-28 15:58:47  volsync                 /data
081d9cef  2023-04-28 15:58:49  volsync                 /data
cf53d49f  2023-04-28 15:58:49  volsync                 /data
8f8845ca  2023-04-28 15:58:53  volsync                 /data
f34b0aa1  2023-04-28 15:58:54  volsync                 /data
075114da  2023-04-28 15:58:55  volsync                 /data
ac592a90  2023-04-28 15:58:56  volsync                 /data
270b90b5  2023-04-28 15:58:58  volsync                 /data
1b52061f  2023-04-28 15:59:02  volsync                 /data
f872a95c  2023-04-28 15:59:02  volsync                 /data
5b9ee331  2023-04-28 15:59:08  volsync                 /data
3d199aad  2023-04-28 15:59:08  volsync                 /data
18cd328e  2023-04-28 15:59:10  volsync                 /data
3f30f6e8  2023-04-28 15:59:11  volsync                 /data
6e06198c  2023-04-28 15:59:14  volsync                 /data
228d6fa3  2023-04-28 15:59:18  volsync                 /data
2dbed603  2023-04-28 15:59:20  volsync                 /data
57a9ddc3  2023-04-28 15:59:21  volsync                 /data
b76adb34  2023-04-28 15:59:22  volsync                 /data
fccfea00  2023-04-28 15:59:23  volsync                 /data
83866d42  2023-04-28 15:59:24  volsync                 /data
4f8ab7ae  2023-04-28 15:59:24  volsync                 /data
91aa304d  2023-04-28 15:59:25  volsync                 /data
61039727  2023-04-28 15:59:25  volsync                 /data
289cef69  2023-04-28 15:59:25  volsync                 /data
------------------------------------------------------------
35 snapshots

As far as I can tell, the backups are succeeding... ?

Does this behavior make any sense to you? What debug output would you like?

tesshuflower commented 1 year ago

First guess was that you managed to patch it when the job was still running, and the reconciles had trouble updating the job (originally started with old env vars) with the new env vars. It may have gotten into a loop getting errors because the job has changed (perhaps finished) while thinking it needed to be updated with the new vars?

but - I think you're saying you can get this every time? Even with a new replicationsource? Maybe something is wrong with the patch - can you send me the errors you see in the volsync controller logs when it's trying to create/update the job?

Rojikku commented 1 year ago

@tesshuflower Nah, I deleted it entirely and re-created it, so wasn't that first thing.

Oh. That controller log got angry really fast...

r{Name:\"OS_PASSWORD\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc026895e20)}, core.EnvVar{Name:\"OS_TENANT_ID\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc026895e40)}, core.EnvVar{Name:\"OS_TENANT_NAME\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc026895e60)}, core.EnvVar{Name:\"OS_USER_DOMAIN_NAME\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc026895e80)}, core.EnvVar{Name:\"OS_USER_DOMAIN_ID\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc026895ea0)}, core.EnvVar{Name:\"OS_PROJECT_NAME\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc026895ec0)}, core.EnvVar{Name:\"OS_PROJECT_DOMAIN_NAME\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc026895ee0)}, core.EnvVar{Name:\"OS_PROJECT_DOMAIN_ID\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc026895f00)}, core.EnvVar{Name:\"OS_TRUST_ID\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc026895f20)}, core.EnvVar{Name:\"OS_APPLICATION_CREDENTIAL_ID\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc026895f40)}, core.EnvVar{Name:\"OS_APPLICATION_CREDENTIAL_NAME\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc026895f60)}, core.EnvVar{Name:\"OS_APPLICATION_CREDENTIAL_SECRET\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc026895f80)}, core.EnvVar{Name:\"OS_STORAGE_URL\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc026895fa0)}, core.EnvVar{Name:\"OS_AUTH_TOKEN\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc026895fc0)}, core.EnvVar{Name:\"B2_ACCOUNT_ID\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc026895fe0)}, core.EnvVar{Name:\"B2_ACCOUNT_KEY\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc026e58000)}, core.EnvVar{Name:\"AZURE_ACCOUNT_NAME\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc026e58020)}, core.EnvVar{Name:\"AZURE_ACCOUNT_KEY\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc026e58040)}, core.EnvVar{Name:\"AZURE_ACCOUNT_SAS\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc026e58060)}, core.EnvVar{Name:\"GOOGLE_PROJECT_ID\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc026e58080)}, core.EnvVar{Name:\"RCLONE_CONFIG_NAS_HOST\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc026e580a0)}, core.EnvVar{Name:\"RCLONE_CONFIG_NAS_SHA1SUM_COMMAND\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc026e580c0)}, core.EnvVar{Name:\"RCLONE_CONFIG_NAS_USER\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc026e580e0)}, core.EnvVar{Name:\"RCLONE_CONFIG_NAS_MD5SUM_COMMAND\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc026e58100)}, core.EnvVar{Name:\"RCLONE_CONFIG_NAS_PORT\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc026e58120)}, core.EnvVar{Name:\"RCLONE_CONFIG_NAS_PASS\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc026e58140)}, core.EnvVar{Name:\"RCLONE_CONFIG_NAS_SHELL_TYPE\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc026e58160)}, core.EnvVar{Name:\"RCLONE_CONFIG_NAS_TYPE\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc026e58180)}, core.EnvVar{Name:\"PRIVILEGED_MOVER\", Value:\"0\", ValueFrom:(*core.EnvVarSource)(nil)}}, Resources:core.ResourceRequirements{Limits:core.ResourceList(nil), Requests:core.ResourceList(nil), Claims:[]core.ResourceClaim(nil)}, VolumeMounts:[]core.VolumeMount{core.VolumeMount{Name:\"data\", ReadOnly:false, MountPath:\"/data\", SubPath:\"\", MountPropagation:(*core.MountPropagationMode)(nil), SubPathExpr:\"\"}, core.VolumeMount{Name:\"cache\", ReadOnly:false, MountPath:\"/cache\", SubPath:\"\", MountPropagation:(*core.MountPropagationMode)(nil), SubPathExpr:\"\"}, core.VolumeMount{Name:\"tempdir\", ReadOnly:false, MountPath:\"/tmp\", SubPath:\"\", MountPropagation:(*core.MountPropagationMode)(nil), SubPathExpr:\"\"}}, VolumeDevices:[]core.VolumeDevice(nil), LivenessProbe:(*core.Probe)(nil), ReadinessProbe:(*core.Probe)(nil), StartupProbe:(*core.Probe)(nil), Lifecycle:(*core.Lifecycle)(nil), TerminationMessagePath:\"/dev/termination-log\", TerminationMessagePolicy:\"File\", ImagePullPolicy:\"IfNotPresent\", SecurityContext:(*core.SecurityContext)(0xc012d77e60), Stdin:false, StdinOnce:false, TTY:false}}, EphemeralContainers:[]core.EphemeralContainer(nil), RestartPolicy:\"Never\", TerminationGracePeriodSeconds:(*int64)(0xc018dffdb0), ActiveDeadlineSeconds:(*int64)(nil), DNSPolicy:\"ClusterFirst\", NodeSelector:map[string]string(nil), ServiceAccountName:\"volsync-src-zwave-js-ui\", AutomountServiceAccountToken:(*bool)(nil), NodeName:\"\", SecurityContext:(*core.PodSecurityContext)(0xc04dbefb90), ImagePullSecrets:[]core.LocalObjectReference(nil), Hostname:\"\", Subdomain:\"\", SetHostnameAsFQDN:(*bool)(nil), Affinity:(*core.Affinity)(nil), SchedulerName:\"default-scheduler\", Tolerations:[]core.Toleration(nil), HostAliases:[]core.HostAlias(nil), PriorityClassName:\"\", Priority:(*int32)(nil), PreemptionPolicy:(*core.PreemptionPolicy)(nil), DNSConfig:(*core.PodDNSConfig)(nil), ReadinessGates:[]core.PodReadinessGate(nil), RuntimeClassName:(*string)(nil), Overhead:core.ResourceList(nil), EnableServiceLinks:(*bool)(nil), TopologySpreadConstraints:[]core.TopologySpreadConstraint(nil), OS:(*core.PodOS)(nil), SchedulingGates:[]core.PodSchedulingGate(nil), ResourceClaims:[]core.PodResourceClaim(nil)}}: field is immutable"}
github.com/backube/volsync/controllers/utils.CreateOrUpdateDeleteOnImmutableErr
    /workspace/controllers/utils/reconcile.go:57
github.com/backube/volsync/controllers/mover/restic.(*Mover).ensureJob
    /workspace/controllers/mover/restic/mover.go:280
github.com/backube/volsync/controllers/mover/restic.(*Mover).Synchronize
    /workspace/controllers/mover/restic/mover.go:138
github.com/backube/volsync/controllers.(*rsMachine).Synchronize
    /workspace/controllers/replicationsource_controller.go:264
github.com/backube/volsync/controllers/statemachine.doSynchronizingState
    /workspace/controllers/statemachine/machine.go:102
github.com/backube/volsync/controllers/statemachine.Run
    /workspace/controllers/statemachine/machine.go:70
github.com/backube/volsync/controllers.(*ReplicationSourceReconciler).Reconcile
    /workspace/controllers/replicationsource_controller.go:138
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-28T21:00:05.424Z    ERROR   controllers.ReplicationSource   reconcile failed    {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "job": "hass/volsync-src-zwave-js-ui", "error": "unable to update object. Deleting object so it can be recreated"}
github.com/backube/volsync/controllers/mover/restic.(*Mover).ensureJob
    /workspace/controllers/mover/restic/mover.go:519
github.com/backube/volsync/controllers/mover/restic.(*Mover).Synchronize
    /workspace/controllers/mover/restic/mover.go:138
github.com/backube/volsync/controllers.(*rsMachine).Synchronize
    /workspace/controllers/replicationsource_controller.go:264
github.com/backube/volsync/controllers/statemachine.doSynchronizingState
    /workspace/controllers/statemachine/machine.go:102
github.com/backube/volsync/controllers/statemachine.Run
    /workspace/controllers/statemachine/machine.go:70
github.com/backube/volsync/controllers.(*ReplicationSourceReconciler).Reconcile
    /workspace/controllers/replicationsource_controller.go:138
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-28T21:00:05.460Z    ERROR   Reconciler error    {"controller": "replicationsource", "controllerGroup": "volsync.backube", "controllerKind": "ReplicationSource", "ReplicationSource": {"name":"zwave-js-ui","namespace":"hass"}, "namespace": "hass", "name": "zwave-js-ui", "reconcileID": "892edf23-029d-4453-9b11-3d768109785e", "error": "unable to update object. Deleting object so it can be recreated"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-28T21:00:05.460Z    INFO    controllers.ReplicationSource   Namespace does not indicate volsync privileged movers are allowed, will assume privileged movers are not allowed. Annotation missing or not equal to 'true' {"replicationsource": "hass/zwave-js-ui", "namespace": "hass", "Annotation": "volsync.backube/privileged-movers"}
2023-04-28T21:00:05.460Z    DEBUG   controllers.ReplicationSource   temporary snapshot reconciled   {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "snapshot": "hass/volsync-zwave-js-ui-src", "operation": "unchanged"}
2023-04-28T21:00:05.460Z    DEBUG   controllers.ReplicationSource   pvc from snap reconciled    {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "pvc": "hass/volsync-zwave-js-ui-src", "operation": "unchanged"}
2023-04-28T21:00:05.460Z    INFO    controllers.ReplicationSource   allocating cache volume {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "PVC": "volsync-zwave-js-ui-cache"}
2023-04-28T21:00:05.460Z    DEBUG   controllers.ReplicationSource   PVC reconciled  {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "PVC": "volsync-zwave-js-ui-cache", "operation": "unchanged"}
2023-04-28T21:00:05.461Z    DEBUG   controllers.ReplicationSource   ServiceAccount reconciled   {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "ServiceAccount": "hass/volsync-src-zwave-js-ui", "operation": "unchanged"}
2023-04-28T21:00:05.461Z    DEBUG   controllers.ReplicationSource   Role reconciled {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "Role": "hass/volsync-src-zwave-js-ui", "operation": "unchanged"}
2023-04-28T21:00:05.461Z    DEBUG   controllers.ReplicationSource   RoleBinding reconciled  {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "RoleBinding": "hass/volsync-src-zwave-js-ui", "operation": "unchanged"}
2023-04-28T21:00:05.461Z    INFO    controllers.ReplicationSource   job actions {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "job": "hass/volsync-src-zwave-js-ui", "actions": ["backup"]}
2023-04-28T21:00:05.520Z    INFO    controllers.ReplicationSource   Namespace does not indicate volsync privileged movers are allowed, will assume privileged movers are not allowed. Annotation missing or not equal to 'true' {"replicationsource": "hass/zwave-js-ui", "namespace": "hass", "Annotation": "volsync.backube/privileged-movers"}
2023-04-28T21:00:05.520Z    DEBUG   controllers.ReplicationSource   temporary snapshot reconciled   {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "snapshot": "hass/volsync-zwave-js-ui-src", "operation": "unchanged"}
2023-04-28T21:00:05.521Z    DEBUG   controllers.ReplicationSource   pvc from snap reconciled    {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "pvc": "hass/volsync-zwave-js-ui-src", "operation": "unchanged"}
2023-04-28T21:00:05.521Z    INFO    controllers.ReplicationSource   allocating cache volume {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "PVC": "volsync-zwave-js-ui-cache"}
2023-04-28T21:00:05.521Z    DEBUG   controllers.ReplicationSource   PVC reconciled  {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "PVC": "volsync-zwave-js-ui-cache", "operation": "unchanged"}
2023-04-28T21:00:05.521Z    DEBUG   controllers.ReplicationSource   ServiceAccount reconciled   {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "ServiceAccount": "hass/volsync-src-zwave-js-ui", "operation": "unchanged"}
2023-04-28T21:00:05.521Z    DEBUG   controllers.ReplicationSource   Role reconciled {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "Role": "hass/volsync-src-zwave-js-ui", "operation": "unchanged"}
2023-04-28T21:00:05.521Z    DEBUG   controllers.ReplicationSource   RoleBinding reconciled  {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "RoleBinding": "hass/volsync-src-zwave-js-ui", "operation": "unchanged"}
2023-04-28T21:00:05.521Z    INFO    controllers.ReplicationSource   job actions {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "job": "hass/volsync-src-zwave-js-ui", "actions": ["backup"]}
2023-04-28T21:00:05.532Z    ERROR   controllers.ReplicationSource   Immutable error updating the object. Will delete so it can be recreated {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "job": "hass/volsync-src-zwave-js-ui", "error": "Job.batch \"volsync-src-zwave-js-ui\" is invalid: spec.template: Invalid value: core.PodTemplateSpec{ObjectMeta:v1.ObjectMeta{Name:\"volsync-src-zwave-js-ui\", GenerateName:\"\", Namespace:\"\", SelfLink:\"\", UID:\"\", ResourceVersion:\"\", Generation:0, CreationTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeletionTimestamp:<nil>, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{\"app.kubernetes.io/created-by\":\"volsync\", \"controller-uid\":\"2a1354ae-fb90-4fee-9c68-9240b0f93942\", \"job-name\":\"volsync-src-zwave-js-ui\"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:core.PodSpec{Volumes:[]core.Volume{core.Volume{Name:\"data\", VolumeSource:core.VolumeSource{HostPath:(*core.HostPathVolumeSource)(nil), EmptyDir:(*core.EmptyDirVolumeSource)(nil), GCEPersistentDisk:(*core.GCEPersistentDiskVolumeSource)(nil), AWSElasticBlockStore:(*core.AWSElasticBlockStoreVolumeSource)(nil), GitRepo:(*core.GitRepoVolumeSource)(nil), Secret:(*core.SecretVolumeSource)(nil), NFS:(*core.NFSVolumeSource)(nil), ISCSI:(*core.ISCSIVolumeSource)(nil), Glusterfs:(*core.GlusterfsVolumeSource)(nil), PersistentVolumeClaim:(*core.PersistentVolumeClaimVolumeSource)(0xc0084c19c8), RBD:(*core.RBDVolumeSource)(nil), Quobyte:(*core.QuobyteVolumeSource)(nil), FlexVolume:(*core.FlexVolumeSource)(nil), Cinder:(*core.CinderVolumeSource)(nil), CephFS:(*core.CephFSVolumeSource)(nil), Flocker:(*core.FlockerVolumeSource)(nil), DownwardAPI:(*core.DownwardAPIVolumeSource)(nil), FC:(*core.FCVolumeSource)(nil), AzureFile:(*core.AzureFileVolumeSource)(nil), ConfigMap:(*core.ConfigMapVolumeSource)(nil), VsphereVolume:(*core.VsphereVirtualDiskVolumeSource)(nil), AzureDisk:(*core.AzureDiskVolumeSource)(nil), PhotonPersistentDisk:(*core.PhotonPersistentDiskVolumeSource)(nil), Projected:(*core.ProjectedVolumeSource)(nil), PortworxVolume:(*core.PortworxVolumeSource)(nil), ScaleIO:(*core.ScaleIOVolumeSource)(nil), StorageOS:(*core.StorageOSVolumeSource)(nil), CSI:(*core.CSIVolumeSource)(nil), Ephemeral:(*core.EphemeralVolumeSource)(nil)}}, core.Volume{Name:\"cache\", VolumeSource:core.VolumeSource{HostPath:(*core.HostPathVolumeSource)(nil), EmptyDir:(*core.EmptyDirVolumeSource)(nil), GCEPersistentDisk:(*core.GCEPersistentDiskVolumeSource)(nil), AWSElasticBlockStore:(*core.AWSElasticBlockStoreVolumeSource)(nil), GitRepo:(*core.GitRepoVolumeSource)(nil), Secret:(*core.SecretVolumeSource)(nil), NFS:(*core.NFSVolumeSource)(nil), ISCSI:(*core.ISCSIVolumeSource)(nil), Glusterfs:(*core.GlusterfsVolumeSource)(nil), PersistentVolumeClaim:(*core.PersistentVolumeClaimVolumeSource)(0xc0084c19e0), RBD:(*core.RBDVolumeSource)(nil), Quobyte:(*core.QuobyteVolumeSource)(nil), FlexVolume:(*core.FlexVolumeSource)(nil), Cinder:(*core.CinderVolumeSource)(nil), CephFS:(*core.CephFSVolumeSource)(nil), Flocker:(*core.FlockerVolumeSource)(nil), DownwardAPI:(*core.DownwardAPIVolumeSource)(nil), FC:(*core.FCVolumeSource)(nil), AzureFile:(*core.AzureFileVolumeSource)(nil), ConfigMap:(*core.ConfigMapVolumeSource)(nil), VsphereVolume:(*core.VsphereVirtualDiskVolumeSource)(nil), AzureDisk:(*core.AzureDiskVolumeSource)(nil), PhotonPersistentDisk:(*core.PhotonPersistentDiskVolumeSource)(nil), Projected:(*core.ProjectedVolumeSource)(nil), PortworxVolume:(*core.PortworxVolumeSource)(nil), ScaleIO:(*core.ScaleIOVolumeSource)(nil), StorageOS:(*core.StorageOSVolumeSource)(nil), CSI:(*core.CSIVolumeSource)(nil), Ephemeral:(*core.EphemeralVolumeSource)(nil)}}, core.Volume{Name:\"tempdir\", VolumeSource:core.VolumeSource{HostPath:(*core.HostPathVolumeSource)(nil), EmptyDir:(*core.EmptyDirVolumeSource)(0xc0084c19f8), GCEPersistentDisk:(*core.GCEPersistentDiskVolumeSource)(nil), AWSElasticBlockStore:(*core.AWSElasticBlockStoreVolumeSource)(nil), GitRepo:(*core.GitRepoVolumeSource)(nil), Secret:(*core.SecretVolumeSource)(nil), NFS:(*core.NFSVolumeSource)(nil), ISCSI:(*core.ISCSIVolumeSource)(nil), Glusterfs:(*core.GlusterfsVolumeSource)(nil), PersistentVolumeClaim:(*core.PersistentVolumeClaimVolumeSource)(nil), RBD:(*core.RBDVolumeSource)(nil), Quobyte:(*core.QuobyteVolumeSource)(nil), FlexVolume:(*core.FlexVolumeSource)(nil), Cinder:(*core.CinderVolumeSource)(nil), CephFS:(*core.CephFSVolumeSource)(nil), Flocker:(*core.FlockerVolumeSource)(nil), DownwardAPI:(*core.DownwardAPIVolumeSource)(nil), FC:(*core.FCVolumeSource)(nil), AzureFile:(*core.AzureFileVolumeSource)(nil), ConfigMap:(*core.ConfigMapVolumeSource)(nil), VsphereVolume:(*core.VsphereVirtualDiskVolumeSource)(nil), AzureDisk:(*core.AzureDiskVolumeSource)(nil), PhotonPersistentDisk:(*core.PhotonPersistentDiskVolumeSource)(nil), Projected:(*core.ProjectedVolumeSource)(nil), PortworxVolume:(*core.PortworxVolumeSource)(nil), ScaleIO:(*core.ScaleIOVolumeSource)(nil), StorageOS:(*core.StorageOSVolumeSource)(nil), CSI:(*core.CSIVolumeSource)(nil), Ephemeral:(*core.EphemeralVolumeSource)(nil)}}}, InitContainers:[]core.Container(nil), Containers:[]core.Container{core.Container{Name:\"restic\", Image:\"quay.io/backube/volsync:0.7.1\", Command:[]string{\"/mover-restic/entry.sh\"}, Args:[]string{\"backup\"}, WorkingDir:\"\", Ports:[]core.ContainerPort(nil), EnvFrom:[]core.EnvFromSource(nil), Env:[]core.EnvVar{core.EnvVar{Name:\"FORGET_OPTIONS\", Value:\" --keep-daily 10 --keep-weekly 6 --keep-monthly 4 --keep-within 3d\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"DATA_DIR\", Value:\"/data\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"RESTIC_CACHE_DIR\", Value:\"/cache\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"RESTORE_AS_OF\", Value:\"\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"SELECT_PREVIOUS\", Value:\"0\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"RESTIC_REPOSITORY\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc026cdbfa0)}, core.EnvVar{Name:\"RESTIC_PASSWORD\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc026cdbfc0)}, core.EnvVar{Name:\"RESTIC_COMPRESSION\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc026cdbfe0)}, core.EnvVar{Name:\"RESTIC_PACK_SIZE\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc027eba000)}, core.EnvVar{Name:\"RESTIC_READ_CONCURRENCY\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc027eba020)}, core.EnvVar{Name:\"AWS_ACCESS_KEY_ID\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc027eba040)}, core.EnvVar{Name:\"AWS_SECRET_ACCESS_KEY\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc027eba060)}, core.EnvVar{Name:\"AWS_SESSION_TOKEN\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc027eba080)}, core.EnvVar{Name:\"AWS_DEFAULT_REGION\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc027eba0a0)}, core.EnvVar{Name:\"ST_AUTH\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc027eba0c0)}, core.EnvVar{Name:\"ST_USER\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc027eba0e0)}, core.EnvVar{Name:\"ST_KEY\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc027eba100)}, core.EnvVar{Name:\"OS_AUTH_URL\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc027eba120)}, core.EnvVar{Name:\"OS_REGION_NAME\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc027eba140)}, core.EnvVar{Name:\"OS_USERNAME\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc027eba160)}, core.EnvVar{Name:\"OS_USER_ID\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc027eba180)}, core.EnvVar{Name:\"OS_PASSWORD\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc027eba1a0)}, core.EnvVar{Name:\"OS_TENANT_ID\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc027eba1c0)}, core.EnvVar{Name:\"OS_TENANT_NAME\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc027eba1e0)}, core.EnvVar{Name:\"OS_USER_DOMAIN_NAME\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc027eba200)}, core.EnvVar{Name:\"OS_USER_DOMAIN_ID\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc027eba220)}, core.EnvVar{Name:\"OS_PROJECT_NAME\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc027eba240)}, core.EnvVar{Name:\"OS_PROJECT_DOMAIN_NAME\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc027eba260)}, core.EnvVar{Name:\"OS_PROJECT_DOMAIN_ID\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc027eba280)}, core.EnvVar{Name:\"OS_TRUST_ID\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc027eba2a0)}, core.EnvVar{Name:\"OS_APPLICATION_CREDENTIAL_ID\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc027eba2c0)}, core.EnvVar{Name:\"OS_APPLICATION_CREDENTIAL_NAME\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc027eba2e0)}, core.EnvVar{Name:\"OS_APPLICATION_CREDENTIAL_SECRET\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc027eba300)}, core.EnvVar{Name:\"OS_STORAGE_URL\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc027eba320)}, core.EnvVar{Name:\"OS_AUTH_TOKEN\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc027eba340)}, core.EnvVar{Name:\"B2_ACCOUNT_ID\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc027eba360)}, core.EnvVar{Name:\"B2_ACCOUNT_KEY\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc027eba380)}, core.EnvVar{Name:\"AZURE_ACCOUNT_NAME\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc027eba3a0)}, core.EnvVar{Name:\"AZURE_ACCOUNT_KEY\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc027eba3c0)}, core.EnvVar{Name:\"AZURE_ACCOUNT_SAS\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc027eba3e0)}, core.EnvVar{Name:\"GOOGLE_PROJECT_ID\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc027eba400)}, core.EnvVar{Name:\"RCLONE_CONFIG_NAS_SHELL_TYPE\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc027eba420)}, core.EnvVar{Name:\"RCLONE_CONFIG_NAS_TYPE\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc027eba440)}, core.EnvVar{Name:\"RCLONE_CONFIG_NAS_USER\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc027eba460)}, core.EnvVar{Name:\"RCLONE_CONFIG_NAS_PORT\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc027eba480)}, core.EnvVar{Name:\"RCLONE_CONFIG_NAS_MD5SUM_COMMAND\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc027eba4a0)}, core.EnvVar{Name:\"RCLONE_CONFIG_NAS_PASS\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc027eba4c0)}, core.EnvVar{Name:\"RCLONE_CONFIG_NAS_SHA1SUM_COMMAND\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc027eba4e0)}, core.EnvVar{Name:\"RCLONE_CONFIG_NAS_HOST\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc027eba500)}, core.EnvVar{Name:\"PRIVILEGED_MOVER\", Value:\"0\", ValueFrom:(*core.EnvVarSource)(nil)}}, Resources:core.ResourceRequirements{Limits:core.ResourceList(nil), Requests:core.ResourceList(nil), Claims:[]core.ResourceClaim(nil)}, VolumeMounts:[]core.VolumeMount{core.VolumeMount{Name:\"data\", ReadOnly:false, MountPath:\"/data\", SubPath:\"\", MountPropagation:(*core.MountPropagationMode)(nil), SubPathExpr:\"\"}, core.VolumeMount{Name:\"cache\", ReadOnly:false, MountPath:\"/cache\", SubPath:\"\", MountPropagation:(*core.MountPropagationMode)(nil), SubPathExpr:\"\"}, core.VolumeMount{Name:\"tempdir\", ReadOnly:false, MountPath:\"/tmp\", SubPath:\"\", MountPropagation:(*core.MountPropagationMode)(nil), SubPathExpr:\"\"}}, VolumeDevices:[]core.VolumeDevice(nil), LivenessProbe:(*core.Probe)(nil), ReadinessProbe:(*core.Probe)(nil), StartupProbe:(*core.Probe)(nil), Lifecycle:(*core.Lifecycle)(nil), TerminationMessagePath:\"/dev/termination-log\", TerminationMessagePolicy:\"File\", ImagePullPolicy:\"IfNotPresent\", SecurityContext:(*core.SecurityContext)(0xc00dac0c60), Stdin:false, StdinOnce:false, TTY:false}}, EphemeralContainers:[]core.EphemeralContainer(nil), RestartPolicy:\"Never\", TerminationGracePeriodSeconds:(*int64)(0xc03badbd90), ActiveDeadlineSeconds:(*int64)(nil), DNSPolicy:\"ClusterFirst\", NodeSelector:map[string]string(nil), ServiceAccountName:\"volsync-src-zwave-js-ui\", AutomountServiceAccountToken:(*bool)(nil), NodeName:\"\", SecurityContext:(*core.PodSecurityContext)(0xc01d0398c0), ImagePullSecrets:[]core.LocalObjectReference(nil), Hostname:\"\", Subdomain:\"\", SetHostnameAsFQDN:(*bool)(nil), Affinity:(*core.Affinity)(nil), SchedulerName:\"default-scheduler\", Tolerations:[]core.Toleration(nil), HostAliases:[]core.HostAlias(nil), PriorityClassName:\"\", Priority:(*int32)(nil), PreemptionPolicy:(*core.PreemptionPolicy)(nil), DNSConfig:(*core.PodDNSConfig)(nil), ReadinessGates:[]core.PodReadinessGate(nil), RuntimeClassName:(*string)(nil), Overhead:core.ResourceList(nil), EnableServiceLinks:(*bool)(nil), TopologySpreadConstraints:[]core.TopologySpreadConstraint(nil), OS:(*core.PodOS)(nil), SchedulingGates:[]core.PodSchedulingGate(nil), ResourceClaims:[]core.PodResourceClaim(nil)}}: field is immutable"}
github.com/backube/volsync/controllers/utils.CreateOrUpdateDeleteOnImmutableErr
    /workspace/controllers/utils/reconcile.go:57
github.com/backube/volsync/controllers/mover/restic.(*Mover).ensureJob
    /workspace/controllers/mover/restic/mover.go:280
github.com/backube/volsync/controllers/mover/restic.(*Mover).Synchronize
    /workspace/controllers/mover/restic/mover.go:138
github.com/backube/volsync/controllers.(*rsMachine).Synchronize
    /workspace/controllers/replicationsource_controller.go:264
github.com/backube/volsync/controllers/statemachine.doSynchronizingState
    /workspace/controllers/statemachine/machine.go:102
github.com/backube/volsync/controllers/statemachine.Run
    /workspace/controllers/statemachine/machine.go:70
github.com/backube/volsync/controllers.(*ReplicationSourceReconciler).Reconcile
    /workspace/controllers/replicationsource_controller.go:138
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-28T21:00:05.550Z    ERROR   controllers.ReplicationSource   reconcile failed    {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "job": "hass/volsync-src-zwave-js-ui", "error": "unable to update object. Deleting object so it can be recreated"}
github.com/backube/volsync/controllers/mover/restic.(*Mover).ensureJob
    /workspace/controllers/mover/restic/mover.go:519
github.com/backube/volsync/controllers/mover/restic.(*Mover).Synchronize
    /workspace/controllers/mover/restic/mover.go:138
github.com/backube/volsync/controllers.(*rsMachine).Synchronize
    /workspace/controllers/replicationsource_controller.go:264
github.com/backube/volsync/controllers/statemachine.doSynchronizingState
    /workspace/controllers/statemachine/machine.go:102
github.com/backube/volsync/controllers/statemachine.Run
    /workspace/controllers/statemachine/machine.go:70
github.com/backube/volsync/controllers.(*ReplicationSourceReconciler).Reconcile
    /workspace/controllers/replicationsource_controller.go:138
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-28T21:00:05.579Z    ERROR   Reconciler error    {"controller": "replicationsource", "controllerGroup": "volsync.backube", "controllerKind": "ReplicationSource", "ReplicationSource": {"name":"zwave-js-ui","namespace":"hass"}, "namespace": "hass", "name": "zwave-js-ui", "reconcileID": "eb62eea0-6e60-4f20-ab56-7e0798dbe17d", "error": "unable to update object. Deleting object so it can be recreated"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-28T21:00:05.579Z    INFO    controllers.ReplicationSource   Namespace does not indicate volsync privileged movers are allowed, will assume privileged movers are not allowed. Annotation missing or not equal to 'true' {"replicationsource": "hass/zwave-js-ui", "namespace": "hass", "Annotation": "volsync.backube/privileged-movers"}
2023-04-28T21:00:05.579Z    DEBUG   controllers.ReplicationSource   temporary snapshot reconciled   {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "snapshot": "hass/volsync-zwave-js-ui-src", "operation": "unchanged"}
2023-04-28T21:00:05.579Z    DEBUG   controllers.ReplicationSource   pvc from snap reconciled    {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "pvc": "hass/volsync-zwave-js-ui-src", "operation": "unchanged"}
2023-04-28T21:00:05.579Z    INFO    controllers.ReplicationSource   allocating cache volume {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "PVC": "volsync-zwave-js-ui-cache"}
2023-04-28T21:00:05.580Z    DEBUG   controllers.ReplicationSource   PVC reconciled  {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "PVC": "volsync-zwave-js-ui-cache", "operation": "unchanged"}
2023-04-28T21:00:05.580Z    DEBUG   controllers.ReplicationSource   ServiceAccount reconciled   {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "ServiceAccount": "hass/volsync-src-zwave-js-ui", "operation": "unchanged"}
2023-04-28T21:00:05.580Z    DEBUG   controllers.ReplicationSource   Role reconciled {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "Role": "hass/volsync-src-zwave-js-ui", "operation": "unchanged"}
2023-04-28T21:00:05.580Z    DEBUG   controllers.ReplicationSource   RoleBinding reconciled  {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "RoleBinding": "hass/volsync-src-zwave-js-ui", "operation": "unchanged"}
2023-04-28T21:00:05.580Z    INFO    controllers.ReplicationSource   job actions {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "job": "hass/volsync-src-zwave-js-ui", "actions": ["backup"]}
2023-04-28T21:00:05.658Z    INFO    controllers.ReplicationSource   Namespace does not indicate volsync privileged movers are allowed, will assume privileged movers are not allowed. Annotation missing or not equal to 'true' {"replicationsource": "hass/zwave-js-ui", "namespace": "hass", "Annotation": "volsync.backube/privileged-movers"}
2023-04-28T21:00:05.658Z    DEBUG   controllers.ReplicationSource   temporary snapshot reconciled   {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "snapshot": "hass/volsync-zwave-js-ui-src", "operation": "unchanged"}
2023-04-28T21:00:05.658Z    DEBUG   controllers.ReplicationSource   pvc from snap reconciled    {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "pvc": "hass/volsync-zwave-js-ui-src", "operation": "unchanged"}
2023-04-28T21:00:05.658Z    INFO    controllers.ReplicationSource   allocating cache volume {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "PVC": "volsync-zwave-js-ui-cache"}
2023-04-28T21:00:05.659Z    DEBUG   controllers.ReplicationSource   PVC reconciled  {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "PVC": "volsync-zwave-js-ui-cache", "operation": "unchanged"}
2023-04-28T21:00:05.659Z    DEBUG   controllers.ReplicationSource   ServiceAccount reconciled   {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "ServiceAccount": "hass/volsync-src-zwave-js-ui", "operation": "unchanged"}
2023-04-28T21:00:05.659Z    DEBUG   controllers.ReplicationSource   Role reconciled {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "Role": "hass/volsync-src-zwave-js-ui", "operation": "unchanged"}
2023-04-28T21:00:05.659Z    DEBUG   controllers.ReplicationSource   RoleBinding reconciled  {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "RoleBinding": "hass/volsync-src-zwave-js-ui", "operation": "unchanged"}
2023-04-28T21:00:05.659Z    INFO    controllers.ReplicationSource   job actions {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "job": "hass/volsync-src-zwave-js-ui", "actions": ["backup"]}
2023-04-28T21:00:05.669Z    ERROR   controllers.ReplicationSource   Immutable error updating the object. Will delete so it can be recreated {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "job": "hass/volsync-src-zwave-js-ui", "error": "Job.batch \"volsync-src-zwave-js-ui\" is invalid: spec.template: Invalid value: core.PodTemplateSpec{ObjectMeta:v1.ObjectMeta{Name:\"volsync-src-zwave-js-ui\", GenerateName:\"\", Namespace:\"\", SelfLink:\"\", UID:\"\", ResourceVersion:\"\", Generation:0, CreationTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeletionTimestamp:<nil>, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{\"app.kubernetes.io/created-by\":\"volsync\", \"controller-uid\":\"fd66afa7-370b-405a-98f5-49ae8d08f4bd\", \"job-name\":\"volsync-src-zwave-js-ui\"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:core.PodSpec{Volumes:[]core.Volume{core.Volume{Name:\"data\", VolumeSource:core.VolumeSource{HostPath:(*core.HostPathVolumeSource)(nil), EmptyDir:(*core.EmptyDirVolumeSource)(nil), GCEPersistentDisk:(*core.GCEPersistentDiskVolumeSource)(nil), AWSElasticBlockStore:(*core.AWSElasticBlockStoreVolumeSource)(nil), GitRepo:(*core.GitRepoVolumeSource)(nil), Secret:(*core.SecretVolumeSource)(nil), NFS:(*core.NFSVolumeSource)(nil), ISCSI:(*core.ISCSIVolumeSource)(nil), Glusterfs:(*core.GlusterfsVolumeSource)(nil), PersistentVolumeClaim:(*core.PersistentVolumeClaimVolumeSource)(0xc00e82b4a0), RBD:(*core.RBDVolumeSource)(nil), Quobyte:(*core.QuobyteVolumeSource)(nil), FlexVolume:(*core.FlexVolumeSource)(nil), Cinder:(*core.CinderVolumeSource)(nil), CephFS:(*core.CephFSVolumeSource)(nil), Flocker:(*core.FlockerVolumeSource)(nil), DownwardAPI:(*core.DownwardAPIVolumeSource)(nil), FC:(*core.FCVolumeSource)(nil), AzureFile:(*core.AzureFileVolumeSource)(nil), ConfigMap:(*core.ConfigMapVolumeSource)(nil), VsphereVolume:(*core.VsphereVirtualDiskVolumeSource)(nil), AzureDisk:(*core.AzureDiskVolumeSource)(nil), PhotonPersistentDisk:(*core.PhotonPersistentDiskVolumeSource)(nil), Projected:(*core.ProjectedVolumeSource)(nil), PortworxVolume:(*core.PortworxVolumeSource)(nil), ScaleIO:(*core.ScaleIOVolumeSource)(nil), StorageOS:(*core.StorageOSVolumeSource)(nil), CSI:(*core.CSIVolumeSource)(nil), Ephemeral:(*core.EphemeralVolumeSource)(nil)}}, core.Volume{Name:\"cache\", VolumeSource:core.VolumeSource{HostPath:(*core.HostPathVolumeSource)(nil), EmptyDir:(*core.EmptyDirVolumeSource)(nil), GCEPersistentDisk:(*core.GCEPersistentDiskVolumeSource)(nil), AWSElasticBlockStore:(*core.AWSElasticBlockStoreVolumeSource)(nil), GitRepo:(*core.GitRepoVolumeSource)(nil), Secret:(*core.SecretVolumeSource)(nil), NFS:(*core.NFSVolumeSource)(nil), ISCSI:(*core.ISCSIVolumeSource)(nil), Glusterfs:(*core.GlusterfsVolumeSource)(nil), PersistentVolumeClaim:(*core.PersistentVolumeClaimVolumeSource)(0xc00e82b4b8), RBD:(*core.RBDVolumeSource)(nil), Quobyte:(*core.QuobyteVolumeSource)(nil), FlexVolume:(*core.FlexVolumeSource)(nil), Cinder:(*core.CinderVolumeSource)(nil), CephFS:(*core.CephFSVolumeSource)(nil), Flocker:(*core.FlockerVolumeSource)(nil), DownwardAPI:(*core.DownwardAPIVolumeSource)(nil), FC:(*core.FCVolumeSource)(nil), AzureFile:(*core.AzureFileVolumeSource)(nil), ConfigMap:(*core.ConfigMapVolumeSource)(nil), VsphereVolume:(*core.VsphereVirtualDiskVolumeSource)(nil), AzureDisk:(*core.AzureDiskVolumeSource)(nil), PhotonPersistentDisk:(*core.PhotonPersistentDiskVolumeSource)(nil), Projected:(*core.ProjectedVolumeSource)(nil), PortworxVolume:(*core.PortworxVolumeSource)(nil), ScaleIO:(*core.ScaleIOVolumeSource)(nil), StorageOS:(*core.StorageOSVolumeSource)(nil), CSI:(*core.CSIVolumeSource)(nil), Ephemeral:(*core.EphemeralVolumeSource)(nil)}}, core.Volume{Name:\"tempdir\", VolumeSource:core.VolumeSource{HostPath:(*core.HostPathVolumeSource)(nil), EmptyDir:(*core.EmptyDirVolumeSource)(0xc00e82b4d0), GCEPersistentDisk:(*core.GCEPersistentDiskVolumeSource)(nil), AWSElasticBlockStore:(*core.AWSElasticBlockStoreVolumeSource)(nil), GitRepo:(*core.GitRepoVolumeSource)(nil), Secret:(*core.SecretVolumeSource)(nil), NFS:(*core.NFSVolumeSource)(nil), ISCSI:(*core.ISCSIVolumeSource)(nil), Glusterfs:(*core.GlusterfsVolumeSource)(nil), PersistentVolumeClaim:(*core.PersistentVolumeClaimVolumeSource)(nil), RBD:(*core.RBDVolumeSource)(nil), Quobyte:(*core.QuobyteVolumeSource)(nil), FlexVolume:(*core.FlexVolumeSource)(nil), Cinder:(*core.CinderVolumeSource)(nil), CephFS:(*core.CephFSVolumeSource)(nil), Flocker:(*core.FlockerVolumeSource)(nil), DownwardAPI:(*core.DownwardAPIVolumeSource)(nil), FC:(*core.FCVolumeSource)(nil), AzureFile:(*core.AzureFileVolumeSource)(nil), ConfigMap:(*core.ConfigMapVolumeSource)(nil), VsphereVolume:(*core.VsphereVirtualDiskVolumeSource)(nil), AzureDisk:(*core.AzureDiskVolumeSource)(nil), PhotonPersistentDisk:(*core.PhotonPersistentDiskVolumeSource)(nil), Projected:(*core.ProjectedVolumeSource)(nil), PortworxVolume:(*core.PortworxVolumeSource)(nil), ScaleIO:(*core.ScaleIOVolumeSource)(nil), StorageOS:(*core.StorageOSVolumeSource)(nil), CSI:(*core.CSIVolumeSource)(nil), Ephemeral:(*core.EphemeralVolumeSource)(nil)}}}, InitContainers:[]core.Container(nil), Containers:[]core.Container{core.Container{Name:\"restic\", Image:\"quay.io/backube/volsync:0.7.1\", Command:[]string{\"/mover-restic/entry.sh\"}, Args:[]string{\"backup\"}, WorkingDir:\"\", Ports:[]core.ContainerPort(nil), EnvFrom:[]core.EnvFromSource(nil), Env:[]core.EnvVar{core.EnvVar{Name:\"FORGET_OPTIONS\", Value:\" --keep-daily 10 --keep-weekly 6 --keep-monthly 4 --keep-within 3d\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"DATA_DIR\", Value:\"/data\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"RESTIC_CACHE_DIR\", Value:\"/cache\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"RESTORE_AS_OF\", Value:\"\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"SELECT_PREVIOUS\", Value:\"0\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"RESTIC_REPOSITORY\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02934e7a0)}, core.EnvVar{Name:\"RESTIC_PASSWORD\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02934e7c0)}, core.EnvVar{Name:\"RESTIC_COMPRESSION\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02934e7e0)}, core.EnvVar{Name:\"RESTIC_PACK_SIZE\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02934e800)}, core.EnvVar{Name:\"RESTIC_READ_CONCURRENCY\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02934e820)}, core.EnvVar{Name:\"AWS_ACCESS_KEY_ID\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02934e840)}, core.EnvVar{Name:\"AWS_SECRET_ACCESS_KEY\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02934e860)}, core.EnvVar{Name:\"AWS_SESSION_TOKEN\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02934e880)}, core.EnvVar{Name:\"AWS_DEFAULT_REGION\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02934e8a0)}, core.EnvVar{Name:\"ST_AUTH\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02934e8c0)}, core.EnvVar{Name:\"ST_USER\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02934e8e0)}, core.EnvVar{Name:\"ST_KEY\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02934e900)}, core.EnvVar{Name:\"OS_AUTH_URL\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02934e920)}, core.EnvVar{Name:\"OS_REGION_NAME\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02934e940)}, core.EnvVar{Name:\"OS_USERNAME\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02934e960)}, core.EnvVar{Name:\"OS_USER_ID\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02934e980)}, core.EnvVar{Name:\"OS_PASSWORD\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02934e9a0)}, core.EnvVar{Name:\"OS_TENANT_ID\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02934e9c0)}, core.EnvVar{Name:\"OS_TENANT_NAME\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02934e9e0)}, core.EnvVar{Name:\"OS_USER_DOMAIN_NAME\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02934ea00)}, core.EnvVar{Name:\"OS_USER_DOMAIN_ID\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02934ea20)}, core.EnvVar{Name:\"OS_PROJECT_NAME\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02934ea40)}, core.EnvVar{Name:\"OS_PROJECT_DOMAIN_NAME\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02934ea60)}, core.EnvVar{Name:\"OS_PROJECT_DOMAIN_ID\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02934ea80)}, core.EnvVar{Name:\"OS_TRUST_ID\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02934eaa0)}, core.EnvVar{Name:\"OS_APPLICATION_CREDENTIAL_ID\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02934eac0)}, core.EnvVar{Name:\"OS_APPLICATION_CREDENTIAL_NAME\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02934eae0)}, core.EnvVar{Name:\"OS_APPLICATION_CREDENTIAL_SECRET\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02934eb00)}, core.EnvVar{Name:\"OS_STORAGE_URL\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02934eb20)}, core.EnvVar{Name:\"OS_AUTH_TOKEN\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02934eb40)}, core.EnvVar{Name:\"B2_ACCOUNT_ID\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02934eb60)}, core.EnvVar{Name:\"B2_ACCOUNT_KEY\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02934eb80)}, core.EnvVar{Name:\"AZURE_ACCOUNT_NAME\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02934eba0)}, core.EnvVar{Name:\"AZURE_ACCOUNT_KEY\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02934ebc0)}, core.EnvVar{Name:\"AZURE_ACCOUNT_SAS\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02934ebe0)}, core.EnvVar{Name:\"GOOGLE_PROJECT_ID\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02934ec00)}, core.EnvVar{Name:\"RCLONE_CONFIG_NAS_SHELL_TYPE\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02934ec20)}, core.EnvVar{Name:\"RCLONE_CONFIG_NAS_USER\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02934ec40)}, core.EnvVar{Name:\"RCLONE_CONFIG_NAS_HOST\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02934ec60)}, core.EnvVar{Name:\"RCLONE_CONFIG_NAS_PORT\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02934ec80)}, core.EnvVar{Name:\"RCLONE_CONFIG_NAS_MD5SUM_COMMAND\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02934eca0)}, core.EnvVar{Name:\"RCLONE_CONFIG_NAS_PASS\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02934ecc0)}, core.EnvVar{Name:\"RCLONE_CONFIG_NAS_SHA1SUM_COMMAND\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02934ece0)}, core.EnvVar{Name:\"RCLONE_CONFIG_NAS_TYPE\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02934ed00)}, core.EnvVar{Name:\"PRIVILEGED_MOVER\", Value:\"0\", ValueFrom:(*core.EnvVarSource)(nil)}}, Resources:core.ResourceRequirements{Limits:core.ResourceList(nil), Requests:core.ResourceList(nil), Claims:[]core.ResourceClaim(nil)}, VolumeMounts:[]core.VolumeMount{core.VolumeMount{Name:\"data\", ReadOnly:false, MountPath:\"/data\", SubPath:\"\", MountPropagation:(*core.MountPropagationMode)(nil), SubPathExpr:\"\"}, core.VolumeMount{Name:\"cache\", ReadOnly:false, MountPath:\"/cache\", SubPath:\"\", MountPropagation:(*core.MountPropagationMode)(nil), SubPathExpr:\"\"}, core.VolumeMount{Name:\"tempdir\", ReadOnly:false, MountPath:\"/tmp\", SubPath:\"\", MountPropagation:(*core.MountPropagationMode)(nil), SubPathExpr:\"\"}}, VolumeDevices:[]core.VolumeDevice(nil), LivenessProbe:(*core.Probe)(nil), ReadinessProbe:(*core.Probe)(nil), StartupProbe:(*core.Probe)(nil), Lifecycle:(*core.Lifecycle)(nil), TerminationMessagePath:\"/dev/termination-log\", TerminationMessagePolicy:\"File\", ImagePullPolicy:\"IfNotPresent\", SecurityContext:(*core.SecurityContext)(0xc0334973e0), Stdin:false, StdinOnce:false, TTY:false}}, EphemeralContainers:[]core.EphemeralContainer(nil), RestartPolicy:\"Never\", TerminationGracePeriodSeconds:(*int64)(0xc036845990), ActiveDeadlineSeconds:(*int64)(nil), DNSPolicy:\"ClusterFirst\", NodeSelector:map[string]string(nil), ServiceAccountName:\"volsync-src-zwave-js-ui\", AutomountServiceAccountToken:(*bool)(nil), NodeName:\"\", SecurityContext:(*core.PodSecurityContext)(0xc02b4fc990), ImagePullSecrets:[]core.LocalObjectReference(nil), Hostname:\"\", Subdomain:\"\", SetHostnameAsFQDN:(*bool)(nil), Affinity:(*core.Affinity)(nil), SchedulerName:\"default-scheduler\", Tolerations:[]core.Toleration(nil), HostAliases:[]core.HostAlias(nil), PriorityClassName:\"\", Priority:(*int32)(nil), PreemptionPolicy:(*core.PreemptionPolicy)(nil), DNSConfig:(*core.PodDNSConfig)(nil), ReadinessGates:[]core.PodReadinessGate(nil), RuntimeClassName:(*string)(nil), Overhead:core.ResourceList(nil), EnableServiceLinks:(*bool)(nil), TopologySpreadConstraints:[]core.TopologySpreadConstraint(nil), OS:(*core.PodOS)(nil), SchedulingGates:[]core.PodSchedulingGate(nil), ResourceClaims:[]core.PodResourceClaim(nil)}}: field is immutable"}
github.com/backube/volsync/controllers/utils.CreateOrUpdateDeleteOnImmutableErr
    /workspace/controllers/utils/reconcile.go:57
github.com/backube/volsync/controllers/mover/restic.(*Mover).ensureJob
    /workspace/controllers/mover/restic/mover.go:280
github.com/backube/volsync/controllers/mover/restic.(*Mover).Synchronize
    /workspace/controllers/mover/restic/mover.go:138
github.com/backube/volsync/controllers.(*rsMachine).Synchronize
    /workspace/controllers/replicationsource_controller.go:264
github.com/backube/volsync/controllers/statemachine.doSynchronizingState
    /workspace/controllers/statemachine/machine.go:102
github.com/backube/volsync/controllers/statemachine.Run
    /workspace/controllers/statemachine/machine.go:70
github.com/backube/volsync/controllers.(*ReplicationSourceReconciler).Reconcile
    /workspace/controllers/replicationsource_controller.go:138
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-28T21:00:05.758Z    ERROR   controllers.ReplicationSource   reconcile failed    {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "job": "hass/volsync-src-zwave-js-ui", "error": "unable to update object. Deleting object so it can be recreated"}
github.com/backube/volsync/controllers/mover/restic.(*Mover).ensureJob
    /workspace/controllers/mover/restic/mover.go:519
github.com/backube/volsync/controllers/mover/restic.(*Mover).Synchronize
    /workspace/controllers/mover/restic/mover.go:138
github.com/backube/volsync/controllers.(*rsMachine).Synchronize
    /workspace/controllers/replicationsource_controller.go:264
github.com/backube/volsync/controllers/statemachine.doSynchronizingState
    /workspace/controllers/statemachine/machine.go:102
github.com/backube/volsync/controllers/statemachine.Run
    /workspace/controllers/statemachine/machine.go:70
github.com/backube/volsync/controllers.(*ReplicationSourceReconciler).Reconcile
    /workspace/controllers/replicationsource_controller.go:138
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-28T21:00:05.793Z    INFO    controllers.ReplicationSource   Namespace does not indicate volsync privileged movers are allowed, will assume privileged movers are not allowed. Annotation missing or not equal to 'true' {"replicationsource": "media/jellyfin", "namespace": "media", "Annotation": "volsync.backube/privileged-movers"}
2023-04-28T21:00:05.793Z    DEBUG   controllers.ReplicationSource   Initiate Rclone Spec validation {"replicationsource": "media/jellyfin", "method": "Rclone"}
2023-04-28T21:00:05.793Z    DEBUG   controllers.ReplicationSource   Rclone Spec validation complete.    {"replicationsource": "media/jellyfin", "method": "Rclone"}
2023-04-28T21:00:05.793Z    INFO    controllers.ReplicationSource   RcloneConfig reconciled {"replicationsource": "media/jellyfin", "method": "Rclone"}
2023-04-28T21:00:05.793Z    ERROR   Reconciler error    {"controller": "replicationsource", "controllerGroup": "volsync.backube", "controllerKind": "ReplicationSource", "ReplicationSource": {"name":"zwave-js-ui","namespace":"hass"}, "namespace": "hass", "name": "zwave-js-ui", "reconcileID": "48452cdc-4641-41fe-b946-a5e8a1ee6eaa", "error": "unable to update object. Deleting object so it can be recreated"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-28T21:00:05.793Z    INFO    controllers.ReplicationSource   Namespace does not indicate volsync privileged movers are allowed, will assume privileged movers are not allowed. Annotation missing or not equal to 'true' {"replicationsource": "hass/zwave-js-ui", "namespace": "hass", "Annotation": "volsync.backube/privileged-movers"}
2023-04-28T21:00:05.793Z    DEBUG   controllers.ReplicationSource   waiting for snapshot to be ready    {"replicationsource": "media/jellyfin", "method": "Rclone", "snapshot": "media/volsync-jellyfin-src"}
2023-04-28T21:00:05.793Z    DEBUG   controllers.ReplicationSource   temporary snapshot reconciled   {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "snapshot": "hass/volsync-zwave-js-ui-src", "operation": "unchanged"}
2023-04-28T21:00:05.794Z    DEBUG   controllers.ReplicationSource   pvc from snap reconciled    {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "pvc": "hass/volsync-zwave-js-ui-src", "operation": "unchanged"}
2023-04-28T21:00:05.794Z    INFO    controllers.ReplicationSource   allocating cache volume {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "PVC": "volsync-zwave-js-ui-cache"}
2023-04-28T21:00:05.794Z    DEBUG   controllers.ReplicationSource   PVC reconciled  {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "PVC": "volsync-zwave-js-ui-cache", "operation": "unchanged"}
2023-04-28T21:00:05.794Z    DEBUG   controllers.ReplicationSource   ServiceAccount reconciled   {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "ServiceAccount": "hass/volsync-src-zwave-js-ui", "operation": "unchanged"}
2023-04-28T21:00:05.794Z    DEBUG   controllers.ReplicationSource   Role reconciled {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "Role": "hass/volsync-src-zwave-js-ui", "operation": "unchanged"}
2023-04-28T21:00:05.794Z    DEBUG   controllers.ReplicationSource   RoleBinding reconciled  {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "RoleBinding": "hass/volsync-src-zwave-js-ui", "operation": "unchanged"}
2023-04-28T21:00:05.794Z    INFO    controllers.ReplicationSource   job actions {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "job": "hass/volsync-src-zwave-js-ui", "actions": ["backup"]}
2023-04-28T21:00:05.922Z    INFO    controllers.ReplicationSource   Namespace does not indicate volsync privileged movers are allowed, will assume privileged movers are not allowed. Annotation missing or not equal to 'true' {"replicationsource": "hass/zwave-js-ui", "namespace": "hass", "Annotation": "volsync.backube/privileged-movers"}
2023-04-28T21:00:05.922Z    DEBUG   controllers.ReplicationSource   temporary snapshot reconciled   {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "snapshot": "hass/volsync-zwave-js-ui-src", "operation": "unchanged"}
2023-04-28T21:00:05.922Z    DEBUG   controllers.ReplicationSource   pvc from snap reconciled    {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "pvc": "hass/volsync-zwave-js-ui-src", "operation": "unchanged"}
2023-04-28T21:00:05.922Z    INFO    controllers.ReplicationSource   allocating cache volume {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "PVC": "volsync-zwave-js-ui-cache"}
2023-04-28T21:00:05.923Z    DEBUG   controllers.ReplicationSource   PVC reconciled  {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "PVC": "volsync-zwave-js-ui-cache", "operation": "unchanged"}
2023-04-28T21:00:05.923Z    DEBUG   controllers.ReplicationSource   ServiceAccount reconciled   {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "ServiceAccount": "hass/volsync-src-zwave-js-ui", "operation": "unchanged"}
2023-04-28T21:00:05.923Z    DEBUG   controllers.ReplicationSource   Role reconciled {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "Role": "hass/volsync-src-zwave-js-ui", "operation": "unchanged"}
2023-04-28T21:00:05.923Z    DEBUG   controllers.ReplicationSource   RoleBinding reconciled  {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "RoleBinding": "hass/volsync-src-zwave-js-ui", "operation": "unchanged"}
2023-04-28T21:00:05.923Z    INFO    controllers.ReplicationSource   job actions {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "job": "hass/volsync-src-zwave-js-ui", "actions": ["backup"]}
2023-04-28T21:00:05.964Z    ERROR   controllers.ReplicationSource   reconcile failed    {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "job": "hass/volsync-src-zwave-js-ui", "error": "Operation cannot be fulfilled on jobs.batch \"volsync-src-zwave-js-ui\": the object has been modified; please apply your changes to the latest version and try again"}
github.com/backube/volsync/controllers/mover/restic.(*Mover).ensureJob
    /workspace/controllers/mover/restic/mover.go:519
github.com/backube/volsync/controllers/mover/restic.(*Mover).Synchronize
    /workspace/controllers/mover/restic/mover.go:138
github.com/backube/volsync/controllers.(*rsMachine).Synchronize
    /workspace/controllers/replicationsource_controller.go:264
github.com/backube/volsync/controllers/statemachine.doSynchronizingState
    /workspace/controllers/statemachine/machine.go:102
github.com/backube/volsync/controllers/statemachine.Run
    /workspace/controllers/statemachine/machine.go:70
github.com/backube/volsync/controllers.(*ReplicationSourceReconciler).Reconcile
    /workspace/controllers/replicationsource_controller.go:138
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-28T21:00:05.995Z    ERROR   Reconciler error    {"controller": "replicationsource", "controllerGroup": "volsync.backube", "controllerKind": "ReplicationSource", "ReplicationSource": {"name":"zwave-js-ui","namespace":"hass"}, "namespace": "hass", "name": "zwave-js-ui", "reconcileID": "fa38c547-e541-4ec4-8143-7dc012d7c719", "error": "Operation cannot be fulfilled on jobs.batch \"volsync-src-zwave-js-ui\": the object has been modified; please apply your changes to the latest version and try again"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-28T21:00:05.995Z    INFO    controllers.ReplicationSource   Namespace does not indicate volsync privileged movers are allowed, will assume privileged movers are not allowed. Annotation missing or not equal to 'true' {"replicationsource": "hass/zwave-js-ui", "namespace": "hass", "Annotation": "volsync.backube/privileged-movers"}
2023-04-28T21:00:05.995Z    DEBUG   controllers.ReplicationSource   temporary snapshot reconciled   {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "snapshot": "hass/volsync-zwave-js-ui-src", "operation": "unchanged"}
2023-04-28T21:00:05.996Z    DEBUG   controllers.ReplicationSource   pvc from snap reconciled    {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "pvc": "hass/volsync-zwave-js-ui-src", "operation": "unchanged"}
2023-04-28T21:00:05.996Z    INFO    controllers.ReplicationSource   allocating cache volume {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "PVC": "volsync-zwave-js-ui-cache"}
2023-04-28T21:00:05.997Z    DEBUG   controllers.ReplicationSource   PVC reconciled  {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "PVC": "volsync-zwave-js-ui-cache", "operation": "unchanged"}
2023-04-28T21:00:05.997Z    DEBUG   controllers.ReplicationSource   ServiceAccount reconciled   {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "ServiceAccount": "hass/volsync-src-zwave-js-ui", "operation": "unchanged"}
2023-04-28T21:00:05.998Z    DEBUG   controllers.ReplicationSource   Role reconciled {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "Role": "hass/volsync-src-zwave-js-ui", "operation": "unchanged"}
2023-04-28T21:00:05.998Z    DEBUG   controllers.ReplicationSource   RoleBinding reconciled  {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "RoleBinding": "hass/volsync-src-zwave-js-ui", "operation": "unchanged"}
2023-04-28T21:00:05.998Z    INFO    controllers.ReplicationSource   job actions {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "job": "hass/volsync-src-zwave-js-ui", "actions": ["backup"]}
2023-04-28T21:00:06.013Z    ERROR   controllers.ReplicationSource   Immutable error updating the object. Will delete so it can be recreated {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "job": "hass/volsync-src-zwave-js-ui", "error": "Job.batch \"volsync-src-zwave-js-ui\" is invalid: spec.template: Invalid value: core.PodTemplateSpec{ObjectMeta:v1.ObjectMeta{Name:\"volsync-src-zwave-js-ui\", GenerateName:\"\", Namespace:\"\", SelfLink:\"\", UID:\"\", ResourceVersion:\"\", Generation:0, CreationTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeletionTimestamp:<nil>, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{\"app.kubernetes.io/created-by\":\"volsync\", \"controller-uid\":\"f8dd9358-58c6-4ee5-8457-b4b5883dcd84\", \"job-name\":\"volsync-src-zwave-js-ui\"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:core.PodSpec{Volumes:[]core.Volume{core.Volume{Name:\"data\", VolumeSource:core.VolumeSource{HostPath:(*core.HostPathVolumeSource)(nil), EmptyDir:(*core.EmptyDirVolumeSource)(nil), GCEPersistentDisk:(*core.GCEPersistentDiskVolumeSource)(nil), AWSElasticBlockStore:(*core.AWSElasticBlockStoreVolumeSource)(nil), GitRepo:(*core.GitRepoVolumeSource)(nil), Secret:(*core.SecretVolumeSource)(nil), NFS:(*core.NFSVolumeSource)(nil), ISCSI:(*core.ISCSIVolumeSource)(nil), Glusterfs:(*core.GlusterfsVolumeSource)(nil), PersistentVolumeClaim:(*core.PersistentVolumeClaimVolumeSource)(0xc0312492c0), RBD:(*core.RBDVolumeSource)(nil), Quobyte:(*core.QuobyteVolumeSource)(nil), FlexVolume:(*core.FlexVolumeSource)(nil), Cinder:(*core.CinderVolumeSource)(nil), CephFS:(*core.CephFSVolumeSource)(nil), Flocker:(*core.FlockerVolumeSource)(nil), DownwardAPI:(*core.DownwardAPIVolumeSource)(nil), FC:(*core.FCVolumeSource)(nil), AzureFile:(*core.AzureFileVolumeSource)(nil), ConfigMap:(*core.ConfigMapVolumeSource)(nil), VsphereVolume:(*core.VsphereVirtualDiskVolumeSource)(nil), AzureDisk:(*core.AzureDiskVolumeSource)(nil), PhotonPersistentDisk:(*core.PhotonPersistentDiskVolumeSource)(nil), Projected:(*core.ProjectedVolumeSource)(nil), PortworxVolume:(*core.PortworxVolumeSource)(nil), ScaleIO:(*core.ScaleIOVolumeSource)(nil), StorageOS:(*core.StorageOSVolumeSource)(nil), CSI:(*core.CSIVolumeSource)(nil), Ephemeral:(*core.EphemeralVolumeSource)(nil)}}, core.Volume{Name:\"cache\", VolumeSource:core.VolumeSource{HostPath:(*core.HostPathVolumeSource)(nil), EmptyDir:(*core.EmptyDirVolumeSource)(nil), GCEPersistentDisk:(*core.GCEPersistentDiskVolumeSource)(nil), AWSElasticBlockStore:(*core.AWSElasticBlockStoreVolumeSource)(nil), GitRepo:(*core.GitRepoVolumeSource)(nil), Secret:(*core.SecretVolumeSource)(nil), NFS:(*core.NFSVolumeSource)(nil), ISCSI:(*core.ISCSIVolumeSource)(nil), Glusterfs:(*core.GlusterfsVolumeSource)(nil), PersistentVolumeClaim:(*core.PersistentVolumeClaimVolumeSource)(0xc0312492d8), RBD:(*core.RBDVolumeSource)(nil), Quobyte:(*core.QuobyteVolumeSource)(nil), FlexVolume:(*core.FlexVolumeSource)(nil), Cinder:(*core.CinderVolumeSource)(nil), CephFS:(*core.CephFSVolumeSource)(nil), Flocker:(*core.FlockerVolumeSource)(nil), DownwardAPI:(*core.DownwardAPIVolumeSource)(nil), FC:(*core.FCVolumeSource)(nil), AzureFile:(*core.AzureFileVolumeSource)(nil), ConfigMap:(*core.ConfigMapVolumeSource)(nil), VsphereVolume:(*core.VsphereVirtualDiskVolumeSource)(nil), AzureDisk:(*core.AzureDiskVolumeSource)(nil), PhotonPersistentDisk:(*core.PhotonPersistentDiskVolumeSource)(nil), Projected:(*core.ProjectedVolumeSource)(nil), PortworxVolume:(*core.PortworxVolumeSource)(nil), ScaleIO:(*core.ScaleIOVolumeSource)(nil), StorageOS:(*core.StorageOSVolumeSource)(nil), CSI:(*core.CSIVolumeSource)(nil), Ephemeral:(*core.EphemeralVolumeSource)(nil)}}, core.Volume{Name:\"tempdir\", VolumeSource:core.VolumeSource{HostPath:(*core.HostPathVolumeSource)(nil), EmptyDir:(*core.EmptyDirVolumeSource)(0xc0312492f0), GCEPersistentDisk:(*core.GCEPersistentDiskVolumeSource)(nil), AWSElasticBlockStore:(*core.AWSElasticBlockStoreVolumeSource)(nil), GitRepo:(*core.GitRepoVolumeSource)(nil), Secret:(*core.SecretVolumeSource)(nil), NFS:(*core.NFSVolumeSource)(nil), ISCSI:(*core.ISCSIVolumeSource)(nil), Glusterfs:(*core.GlusterfsVolumeSource)(nil), PersistentVolumeClaim:(*core.PersistentVolumeClaimVolumeSource)(nil), RBD:(*core.RBDVolumeSource)(nil), Quobyte:(*core.QuobyteVolumeSource)(nil), FlexVolume:(*core.FlexVolumeSource)(nil), Cinder:(*core.CinderVolumeSource)(nil), CephFS:(*core.CephFSVolumeSource)(nil), Flocker:(*core.FlockerVolumeSource)(nil), DownwardAPI:(*core.DownwardAPIVolumeSource)(nil), FC:(*core.FCVolumeSource)(nil), AzureFile:(*core.AzureFileVolumeSource)(nil), ConfigMap:(*core.ConfigMapVolumeSource)(nil), VsphereVolume:(*core.VsphereVirtualDiskVolumeSource)(nil), AzureDisk:(*core.AzureDiskVolumeSource)(nil), PhotonPersistentDisk:(*core.PhotonPersistentDiskVolumeSource)(nil), Projected:(*core.ProjectedVolumeSource)(nil), PortworxVolume:(*core.PortworxVolumeSource)(nil), ScaleIO:(*core.ScaleIOVolumeSource)(nil), StorageOS:(*core.StorageOSVolumeSource)(nil), CSI:(*core.CSIVolumeSource)(nil), Ephemeral:(*core.EphemeralVolumeSource)(nil)}}}, InitContainers:[]core.Container(nil), Containers:[]core.Container{core.Container{Name:\"restic\", Image:\"quay.io/backube/volsync:0.7.1\", Command:[]string{\"/mover-restic/entry.sh\"}, Args:[]string{\"backup\"}, WorkingDir:\"\", Ports:[]core.ContainerPort(nil), EnvFrom:[]core.EnvFromSource(nil), Env:[]core.EnvVar{core.EnvVar{Name:\"FORGET_OPTIONS\", Value:\" --keep-daily 10 --keep-weekly 6 --keep-monthly 4 --keep-within 3d\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"DATA_DIR\", Value:\"/data\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"RESTIC_CACHE_DIR\", Value:\"/cache\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"RESTORE_AS_OF\", Value:\"\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"SELECT_PREVIOUS\", Value:\"0\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"RESTIC_REPOSITORY\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02ad53fc0)}, core.EnvVar{Name:\"RESTIC_PASSWORD\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02ad53fe0)}, core.EnvVar{Name:\"RESTIC_COMPRESSION\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02c76c000)}, core.EnvVar{Name:\"RESTIC_PACK_SIZE\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02c76c020)}, core.EnvVar{Name:\"RESTIC_READ_CONCURRENCY\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02c76c040)}, core.EnvVar{Name:\"AWS_ACCESS_KEY_ID\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02c76c060)}, core.EnvVar{Name:\"AWS_SECRET_ACCESS_KEY\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02c76c080)}, core.EnvVar{Name:\"AWS_SESSION_TOKEN\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02c76c0a0)}, core.EnvVar{Name:\"AWS_DEFAULT_REGION\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02c76c0c0)}, core.EnvVar{Name:\"ST_AUTH\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02c76c0e0)}, core.EnvVar{Name:\"ST_USER\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02c76c100)}, core.EnvVar{Name:\"ST_KEY\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02c76c120)}, core.EnvVar{Name:\"OS_AUTH_URL\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02c76c140)}, core.EnvVar{Name:\"OS_REGION_NAME\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02c76c160)}, core.EnvVar{Name:\"OS_USERNAME\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02c76c180)}, core.EnvVar{Name:\"OS_USER_ID\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02c76c1a0)}, core.EnvVar{Name:\"OS_PASSWORD\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02c76c1c0)}, core.EnvVar{Name:\"OS_TENANT_ID\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02c76c1e0)}, core.EnvVar{Name:\"OS_TENANT_NAME\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02c76c200)}, core.EnvVar{Name:\"OS_USER_DOMAIN_NAME\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02c76c220)}, core.EnvVar{Name:\"OS_USER_DOMAIN_ID\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02c76c240)}, core.EnvVar{Name:\"OS_PROJECT_NAME\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02c76c260)}, core.EnvVar{Name:\"OS_PROJECT_DOMAIN_NAME\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02c76c280)}, core.EnvVar{Name:\"OS_PROJECT_DOMAIN_ID\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02c76c2a0)}, core.EnvVar{Name:\"OS_TRUST_ID\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02c76c2c0)}, core.EnvVar{Name:\"OS_APPLICATION_CREDENTIAL_ID\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02c76c2e0)}, core.EnvVar{Name:\"OS_APPLICATION_CREDENTIAL_NAME\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02c76c300)}, core.EnvVar{Name:\"OS_APPLICATION_CREDENTIAL_SECRET\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02c76c320)}, core.EnvVar{Name:\"OS_STORAGE_URL\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02c76c340)}, core.EnvVar{Name:\"OS_AUTH_TOKEN\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02c76c360)}, core.EnvVar{Name:\"B2_ACCOUNT_ID\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02c76c380)}, core.EnvVar{Name:\"B2_ACCOUNT_KEY\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02c76c3a0)}, core.EnvVar{Name:\"AZURE_ACCOUNT_NAME\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02c76c3c0)}, core.EnvVar{Name:\"AZURE_ACCOUNT_KEY\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02c76c3e0)}, core.EnvVar{Name:\"AZURE_ACCOUNT_SAS\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02c76c400)}, core.EnvVar{Name:\"GOOGLE_PROJECT_ID\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02c76c420)}, core.EnvVar{Name:\"RCLONE_CONFIG_NAS_PORT\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02c76c440)}, core.EnvVar{Name:\"RCLONE_CONFIG_NAS_SHELL_TYPE\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02c76c460)}, core.EnvVar{Name:\"RCLONE_CONFIG_NAS_USER\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02c76c480)}, core.EnvVar{Name:\"RCLONE_CONFIG_NAS_HOST\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02c76c4a0)}, core.EnvVar{Name:\"RCLONE_CONFIG_NAS_MD5SUM_COMMAND\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02c76c4c0)}, core.EnvVar{Name:\"RCLONE_CONFIG_NAS_TYPE\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02c76c4e0)}, core.EnvVar{Name:\"RCLONE_CONFIG_NAS_PASS\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02c76c500)}, core.EnvVar{Name:\"RCLONE_CONFIG_NAS_SHA1SUM_COMMAND\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02c76c520)}, core.EnvVar{Name:\"PRIVILEGED_MOVER\", Value:\"0\", ValueFrom:(*core.EnvVarSource)(nil)}}, Resources:core.ResourceRequirements{Limits:core.ResourceList(nil), Requests:core.ResourceList(nil), Claims:[]core.ResourceClaim(nil)}, VolumeMounts:[]core.VolumeMount{core.VolumeMount{Name:\"data\", ReadOnly:false, MountPath:\"/data\", SubPath:\"\", MountPropagation:(*core.MountPropagationMode)(nil), SubPathExpr:\"\"}, core.VolumeMount{Name:\"cache\", ReadOnly:false, MountPath:\"/cache\", SubPath:\"\", MountPropagation:(*core.MountPropagationMode)(nil), SubPathExpr:\"\"}, core.VolumeMount{Name:\"tempdir\", ReadOnly:false, MountPath:\"/tmp\", SubPath:\"\", MountPropagation:(*core.MountPropagationMode)(nil), SubPathExpr:\"\"}}, VolumeDevices:[]core.VolumeDevice(nil), LivenessProbe:(*core.Probe)(nil), ReadinessProbe:(*core.Probe)(nil), StartupProbe:(*core.Probe)(nil), Lifecycle:(*core.Lifecycle)(nil), TerminationMessagePath:\"/dev/termination-log\", TerminationMessagePolicy:\"File\", ImagePullPolicy:\"IfNotPresent\", SecurityContext:(*core.SecurityContext)(0xc0493ec3c0), Stdin:false, StdinOnce:false, TTY:false}}, EphemeralContainers:[]core.EphemeralContainer(nil), RestartPolicy:\"Never\", TerminationGracePeriodSeconds:(*int64)(0xc02fb8c3f0), ActiveDeadlineSeconds:(*int64)(nil), DNSPolicy:\"ClusterFirst\", NodeSelector:map[string]string(nil), ServiceAccountName:\"volsync-src-zwave-js-ui\", AutomountServiceAccountToken:(*bool)(nil), NodeName:\"\", SecurityContext:(*core.PodSecurityContext)(0xc02fc67cb0), ImagePullSecrets:[]core.LocalObjectReference(nil), Hostname:\"\", Subdomain:\"\", SetHostnameAsFQDN:(*bool)(nil), Affinity:(*core.Affinity)(nil), SchedulerName:\"default-scheduler\", Tolerations:[]core.Toleration(nil), HostAliases:[]core.HostAlias(nil), PriorityClassName:\"\", Priority:(*int32)(nil), PreemptionPolicy:(*core.PreemptionPolicy)(nil), DNSConfig:(*core.PodDNSConfig)(nil), ReadinessGates:[]core.PodReadinessGate(nil), RuntimeClassName:(*string)(nil), Overhead:core.ResourceList(nil), EnableServiceLinks:(*bool)(nil), TopologySpreadConstraints:[]core.TopologySpreadConstraint(nil), OS:(*core.PodOS)(nil), SchedulingGates:[]core.PodSchedulingGate(nil), ResourceClaims:[]core.PodResourceClaim(nil)}}: field is immutable"}
github.com/backube/volsync/controllers/utils.CreateOrUpdateDeleteOnImmutableErr
    /workspace/controllers/utils/reconcile.go:57
github.com/backube/volsync/controllers/mover/restic.(*Mover).ensureJob
    /workspace/controllers/mover/restic/mover.go:280
github.com/backube/volsync/controllers/mover/restic.(*Mover).Synchronize
    /workspace/controllers/mover/restic/mover.go:138
github.com/backube/volsync/controllers.(*rsMachine).Synchronize
    /workspace/controllers/replicationsource_controller.go:264
github.com/backube/volsync/controllers/statemachine.doSynchronizingState
    /workspace/controllers/statemachine/machine.go:102
github.com/backube/volsync/controllers/statemachine.Run
    /workspace/controllers/statemachine/machine.go:70
github.com/backube/volsync/controllers.(*ReplicationSourceReconciler).Reconcile
    /workspace/controllers/replicationsource_controller.go:138
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-28T21:00:06.032Z    ERROR   controllers.ReplicationSource   reconcile failed    {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "job": "hass/volsync-src-zwave-js-ui", "error": "unable to update object. Deleting object so it can be recreated"}
github.com/backube/volsync/controllers/mover/restic.(*Mover).ensureJob
    /workspace/controllers/mover/restic/mover.go:519
github.com/backube/volsync/controllers/mover/restic.(*Mover).Synchronize
    /workspace/controllers/mover/restic/mover.go:138
github.com/backube/volsync/controllers.(*rsMachine).Synchronize
    /workspace/controllers/replicationsource_controller.go:264
github.com/backube/volsync/controllers/statemachine.doSynchronizingState
    /workspace/controllers/statemachine/machine.go:102
github.com/backube/volsync/controllers/statemachine.Run
    /workspace/controllers/statemachine/machine.go:70
github.com/backube/volsync/controllers.(*ReplicationSourceReconciler).Reconcile
    /workspace/controllers/replicationsource_controller.go:138
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235

This is all that was on the screen when I stopped it. Did it catch the important part?... Seemed like the same error repeating.

I noticed it sat for like 2 minutes with no job.

Status:
  Conditions:
    Last Transition Time:  2023-04-28T20:59:33Z
    Message:               Timeout: request did not complete within requested timeout - context deadline exceeded
    Reason:                Error
    Status:                False
    Type:                  Synchronizing
  Last Sync Start Time:    2023-04-28T20:58:14Z
  Latest Mover Status:
  Restic:
Events:
  Type    Reason                        Age   From                Message
  ----    ------                        ----  ----                -------
  Normal  VolumeSnapshotCreated         77s   volsync-controller  created VolumeSnapshot/volsync-zwave-js-ui-src from PersistentVolumeClaim/zwave-pvc
  Normal  PersistentVolumeClaimCreated  51s   volsync-controller  created PersistentVolumeClaim/volsync-zwave-js-ui-src from VolumeSnapshot/volsync-zwave-js-ui-src
  Normal  PersistentVolumeClaimCreated  47s   volsync-controller  created PersistentVolumeClaim/volsync-zwave-js-ui-cache to receive incoming data

Then

Status:
  Conditions:
    Last Transition Time:  2023-04-28T20:59:53Z
    Message:               unable to update object. Deleting object so it can be recreated
    Reason:                Error
    Status:                False
    Type:                  Synchronizing
  Last Sync Start Time:    2023-04-28T20:58:14Z
  Latest Mover Status:
  Restic:
Events:
  Type    Reason                        Age   From                Message
  ----    ------                        ----  ----                -------
  Normal  VolumeSnapshotCreated         89s   volsync-controller  created VolumeSnapshot/volsync-zwave-js-ui-src from PersistentVolumeClaim/zwave-pvc
  Normal  PersistentVolumeClaimCreated  63s   volsync-controller  created PersistentVolumeClaim/volsync-zwave-js-ui-src from VolumeSnapshot/volsync-zwave-js-ui-src
  Normal  PersistentVolumeClaimCreated  59s   volsync-controller  created PersistentVolumeClaim/volsync-zwave-js-ui-cache to receive incoming data

And at this point the job IS created. Seconds later I have a wall of errors.

tesshuflower commented 1 year ago

I think there's another mismatch that I'm not thinking of at the moment - I might need to try a test myself next week to be sure - one thing I should've asked before giving you the image - what version of volsync are you using? This is based off main so would be equivalent to quay.io/backube/volsync:latest (+ the change in my branch)

Rojikku commented 1 year ago

I was using 0.7.1, which should be latest.

https://github.com/Rojikku/home-ops/blob/main/cluster/apps/volsync/volsync/app/helmrelease.yaml

There's my deploy. All my other rclone based syncs seem to be having no issues at all, it's only this restic one that's having issues. It's...possible I'm doing something wrong? But I wouldn't expect this sort of error from an issue on my end. Also, the backups appear to be happening, so I feel like it shouldn't be.

tesshuflower commented 1 year ago

I think the issue is that the job appears to keep the env var ordering, but my code to get the rclone keys dynamically from the secret is getting keys from the map - which will not be ordered consistently - I've pushed a new image you can try @Rojikku. Keep in mind it is based on the latest in main which has changed from 0.7.x, but I think for this test it should still work.

Rojikku commented 1 year ago

@tesshuflower That seems to have done the trick. I admit, very limited testing I have done.

Status:
  Conditions:
    Last Transition Time:  2023-05-02T00:50:02Z
    Message:               Synchronization in-progress
    Reason:                SyncInProgress
    Status:                True
    Type:                  Synchronizing
  Last Sync Duration:      18.070150054s
  Last Sync Start Time:    2023-05-02T00:50:00Z
  Last Sync Time:          2023-05-02T00:48:32Z
  Latest Mover Status:
    Logs:  using parent snapshot c0ec2bfc
Added to the repository: 1.435 MiB (374.648 KiB stored)
processed 2249 files, 6.956 MiB in 0:01
snapshot 1e150ae8 saved
Restic completed in 4s
    Result:        Successful
  Next Sync Time:  2023-05-02T00:50:00Z
  Restic:
Events:
  Type    Reason                        Age                From                Message
  ----    ------                        ----               ----                -------
  Normal  PersistentVolumeClaimCreated  110s               volsync-controller  created PersistentVolumeClaim/volsync-zwave-js-ui-cache to receive incoming data
  Normal  VolumeSnapshotCreated         7s (x2 over 113s)  volsync-controller  created VolumeSnapshot/volsync-zwave-js-ui-src from PersistentVolumeClaim/zwave-pvc
  Normal  PersistentVolumeClaimCreated  5s (x2 over 110s)  volsync-controller  created PersistentVolumeClaim/volsync-zwave-js-ui-src from VolumeSnapshot/volsync-zwave-js-ui-src

Mover log:

Starting container
VolSync restic container version: v0.7.1+8af0bf0
backup
restic 0.15.1 compiled with go1.19.7 on linux/amd64
Testing mandatory env variables
== Checking directory for content ===
== Initialize Dir =======
ID        Time                 Host        Tags        Paths
------------------------------------------------------------
c0ec2bfc  2023-04-28 21:02:08  volsync                 /data
------------------------------------------------------------
1 snapshots
=== Starting backup ===
/data /
rclone: 2023/05/02 00:48:25 NOTICE: Config file "/.rclone.conf" not found - using defaults
using parent snapshot c0ec2bfc

Files:           4 new,  2245 changed,     0 unmodified
Dirs:            0 new,   394 changed,     0 unmodified
Added to the repository: 1.435 MiB (374.648 KiB stored)

processed 2249 files, 6.956 MiB in 0:01
snapshot 1e150ae8 saved
/
=== Starting forget ===
rclone: 2023/05/02 00:48:27 NOTICE: Config file "/.rclone.conf" not found - using defaults
Applying Policy: keep 10 daily, 6 weekly, 4 monthly snapshots and all snapshots within 3d of the newest
keep 2 snapshots:
ID        Time                 Host        Tags        Reasons           Paths
------------------------------------------------------------------------------
c0ec2bfc  2023-04-28 21:02:08  volsync                 daily snapshot    /data
                                                       weekly snapshot
                                                       monthly snapshot
1e150ae8  2023-05-02 00:48:25  volsync                 within 3d         /data
                                                       daily snapshot
                                                       weekly snapshot
                                                       monthly snapshot
------------------------------------------------------------------------------
2 snapshots

Restic completed in 4s
=== Done ===

No random pods are being created anymore, the backup shows in restic. ls latest shows a bunch of files that look correct. All seems very promising to me.

Unfortunately I had a lot going on today, just figured I'd run this simple test before I went to bed, and it looks perfect to me!

Tomorrow I will work on migrating everything to it, and I'll see if I produce any bugs. But, it's doubtful I will, since it'll be the same config with different data. I've been running your previous image this whole time with no real issues that I've noticed, did do one backup restore in that time too with no issues. I'll keep running this test image until there's an official release, most likely. I will let you know if any issues pop up, but so far so good.

Thank you for working on this for me! Really looks like it'll be a quality of life improvement to have several versions of my data like this!

Rojikku commented 1 year ago

Update: Everything is still running perfectly. The retain is working as expected, all my backups have worked as expected, and I haven't seen any further issues.

tesshuflower commented 1 year ago

Update: Everything is still running perfectly. The retain is working as expected, all my backups have worked as expected, and I haven't seen any further issues.

Glad to hear it's working for you! Will do some cleanup/unit tests in my branch and try to get a PR up...

tesshuflower commented 1 year ago

Implemented in https://github.com/backube/volsync/pull/751