Closed Rojikku closed 1 year ago
You're correct that only specific env vars are passed through - the RCLONE_ ones are not. See https://github.com/backube/volsync/blob/ea2d597b7a5d9bf9976635b23a2590160cf421c9/controllers/mover/restic/mover.go#L335-L381
I think that to use rclone this is a bigger change than simply passing through env vars, as restic will also need to know the location of rclone (possibly other flags are required as well?). This wouldn't have been possible previously - but now that the movers are all consolidated into one volsync image, the rclone binary will now be present on the machine. Actually I looked and rclone is put into /usr/local/bin/rclone so maybe will already be present in the path so one less hurdle.
Would passing through all env vars that start with RCLONE_ be enough?
Hey, thanks for looking into this for me!
Yes, as far as I am aware, passing through all env vars that start with RCLONE_ WOULD be enough. The idea crossed my mind as well.
Not sure about other flags being required, but I think I can specify everything via the RCLONE_* variables. At least, everything that can be in the config, which should be enough.
If there's an image I can deploy, I can do testing if needed! Or you could copy my variables from the linked config above, if that helps you in your testing.
On Wed, Apr 26, 2023 at 12:07 PM Tesshu Flower @.***> wrote:
You're correct that only specific env vars are passed through - the RCLONE_ ones are not. See https://github.com/backube/volsync/blob/ea2d597b7a5d9bf9976635b23a2590160cf421c9/controllers/mover/restic/mover.go#L335-L381
I think that to use rclone this is a bigger change than simply passing through env vars, as restic will also need to know the location of rclone (possibly other flags are required as well?). This wouldn't have been possible previously - but now that the movers are all consolidated into one volsync image, the rclone binary will now be present on the machine. Actually I looked and rclone is put into /usr/local/bin/rclone so maybe will already be present in the path so one less hurdle.
Would passing through all env vars that start with RCLONE_ be enough?
— Reply to this email directly, view it on GitHub https://github.com/backube/volsync/issues/722#issuecomment-1523680505, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAMZGJD2NPRYMUFVXH7PVEDXDFB3ZANCNFSM6AAAAAAXHI72MY . You are receiving this because you authored the thread.Message ID: @.***>
@Rojikku - I've built an image (untested) you can try out: quay.io/tflower/volsync:restic-rclone-env
source is here: https://github.com/backube/volsync/compare/main...tesshuflower:volsync:restic_rclone_env_vars
@tesshuflower
TL;DR Seems to work, but there's errors on the replicationsource that cause it to re-make the job every second or so. Each backup succeeds, but I can't get logs because all pods are terminating. Kinda confused. Also I don't quite need a backup every second.
From describe replicationsource
Status:
Conditions:
Last Transition Time: 2023-04-28T19:46:30Z
Message: Operation cannot be fulfilled on jobs.batch "volsync-src-zwave-js-ui": the object has been modified; please apply your changes to the latest version and try again
Reason: Error
Status: False
Type: Synchronizing
Last Sync Start Time: 2023-04-28T19:42:33Z
Status:
Conditions:
Last Transition Time: 2023-04-28T19:47:05Z
Message: unable to update object. Deleting object so it can be recreated
Reason: Error
Status: False
Type: Synchronizing
Last Sync Start Time: 2023-04-28T19:42:33Z
From describe job:
Name: volsync-src-zwave-js-ui
Namespace: hass
Selector: controller-uid=3587d5a5-80c1-4c5b-a195-a056e7f7553b
Labels: app.kubernetes.io/created-by=volsync
volsync.backube/cleanup=43b9c3dc-3228-4106-a6d9-ec1ad9486c98
Annotations: batch.kubernetes.io/job-tracking:
Controlled By: ReplicationSource/zwave-js-ui
Parallelism: 1
Completions: <unset>
Completion Mode: NonIndexed
Start Time: Fri, 28 Apr 2023 15:48:17 -0400
Pods Statuses: 1 Active (0 Ready) / 0 Succeeded / 0 Failed
Pod Template:
Labels: app.kubernetes.io/created-by=volsync
controller-uid=3587d5a5-80c1-4c5b-a195-a056e7f7553b
job-name=volsync-src-zwave-js-ui
Service Account: volsync-src-zwave-js-ui
Containers:
restic:
Image: quay.io/backube/volsync:0.7.1
Port: <none>
Host Port: <none>
Command:
/mover-restic/entry.sh
Args:
backup
Environment:
FORGET_OPTIONS: --keep-daily 10 --keep-weekly 6 --keep-monthly 4 --keep-within 3d
DATA_DIR: /data
RESTIC_CACHE_DIR: /cache
RESTORE_AS_OF:
SELECT_PREVIOUS: 0
RESTIC_REPOSITORY: <set to the key 'RESTIC_REPOSITORY' in secret 'restic-conf-zwave'> Optional: false
RESTIC_PASSWORD: <set to the key 'RESTIC_PASSWORD' in secret 'restic-conf-zwave'> Optional: false
RESTIC_COMPRESSION: <set to the key 'RESTIC_COMPRESSION' in secret 'restic-conf-zwave'> Optional: true
RESTIC_PACK_SIZE: <set to the key 'RESTIC_PACK_SIZE' in secret 'restic-conf-zwave'> Optional: true
RESTIC_READ_CONCURRENCY: <set to the key 'RESTIC_READ_CONCURRENCY' in secret 'restic-conf-zwave'> Optional: true
AWS_ACCESS_KEY_ID: <set to the key 'AWS_ACCESS_KEY_ID' in secret 'restic-conf-zwave'> Optional: true
AWS_SECRET_ACCESS_KEY: <set to the key 'AWS_SECRET_ACCESS_KEY' in secret 'restic-conf-zwave'> Optional: true
AWS_SESSION_TOKEN: <set to the key 'AWS_SESSION_TOKEN' in secret 'restic-conf-zwave'> Optional: true
AWS_DEFAULT_REGION: <set to the key 'AWS_DEFAULT_REGION' in secret 'restic-conf-zwave'> Optional: true
ST_AUTH: <set to the key 'ST_AUTH' in secret 'restic-conf-zwave'> Optional: true
ST_USER: <set to the key 'ST_USER' in secret 'restic-conf-zwave'> Optional: true
ST_KEY: <set to the key 'ST_KEY' in secret 'restic-conf-zwave'> Optional: true
OS_AUTH_URL: <set to the key 'OS_AUTH_URL' in secret 'restic-conf-zwave'> Optional: true
OS_REGION_NAME: <set to the key 'OS_REGION_NAME' in secret 'restic-conf-zwave'> Optional: true
OS_USERNAME: <set to the key 'OS_USERNAME' in secret 'restic-conf-zwave'> Optional: true
OS_USER_ID: <set to the key 'OS_USER_ID' in secret 'restic-conf-zwave'> Optional: true
OS_PASSWORD: <set to the key 'OS_PASSWORD' in secret 'restic-conf-zwave'> Optional: true
OS_TENANT_ID: <set to the key 'OS_TENANT_ID' in secret 'restic-conf-zwave'> Optional: true
OS_TENANT_NAME: <set to the key 'OS_TENANT_NAME' in secret 'restic-conf-zwave'> Optional: true
OS_USER_DOMAIN_NAME: <set to the key 'OS_USER_DOMAIN_NAME' in secret 'restic-conf-zwave'> Optional: true
OS_USER_DOMAIN_ID: <set to the key 'OS_USER_DOMAIN_ID' in secret 'restic-conf-zwave'> Optional: true
OS_PROJECT_NAME: <set to the key 'OS_PROJECT_NAME' in secret 'restic-conf-zwave'> Optional: true
OS_PROJECT_DOMAIN_NAME: <set to the key 'OS_PROJECT_DOMAIN_NAME' in secret 'restic-conf-zwave'> Optional: true
OS_PROJECT_DOMAIN_ID: <set to the key 'OS_PROJECT_DOMAIN_ID' in secret 'restic-conf-zwave'> Optional: true
OS_TRUST_ID: <set to the key 'OS_TRUST_ID' in secret 'restic-conf-zwave'> Optional: true
OS_APPLICATION_CREDENTIAL_ID: <set to the key 'OS_APPLICATION_CREDENTIAL_ID' in secret 'restic-conf-zwave'> Optional: true
OS_APPLICATION_CREDENTIAL_NAME: <set to the key 'OS_APPLICATION_CREDENTIAL_NAME' in secret 'restic-conf-zwave'> Optional: true
OS_APPLICATION_CREDENTIAL_SECRET: <set to the key 'OS_APPLICATION_CREDENTIAL_SECRET' in secret 'restic-conf-zwave'> Optional: true
OS_STORAGE_URL: <set to the key 'OS_STORAGE_URL' in secret 'restic-conf-zwave'> Optional: true
OS_AUTH_TOKEN: <set to the key 'OS_AUTH_TOKEN' in secret 'restic-conf-zwave'> Optional: true
B2_ACCOUNT_ID: <set to the key 'B2_ACCOUNT_ID' in secret 'restic-conf-zwave'> Optional: true
B2_ACCOUNT_KEY: <set to the key 'B2_ACCOUNT_KEY' in secret 'restic-conf-zwave'> Optional: true
AZURE_ACCOUNT_NAME: <set to the key 'AZURE_ACCOUNT_NAME' in secret 'restic-conf-zwave'> Optional: true
AZURE_ACCOUNT_KEY: <set to the key 'AZURE_ACCOUNT_KEY' in secret 'restic-conf-zwave'> Optional: true
AZURE_ACCOUNT_SAS: <set to the key 'AZURE_ACCOUNT_SAS' in secret 'restic-conf-zwave'> Optional: true
GOOGLE_PROJECT_ID: <set to the key 'GOOGLE_PROJECT_ID' in secret 'restic-conf-zwave'> Optional: true
RCLONE_CONFIG_NAS_MD5SUM_COMMAND: <set to the key 'RCLONE_CONFIG_NAS_MD5SUM_COMMAND' in secret 'restic-conf-zwave'> Optional: true
RCLONE_CONFIG_NAS_PORT: <set to the key 'RCLONE_CONFIG_NAS_PORT' in secret 'restic-conf-zwave'> Optional: true
RCLONE_CONFIG_NAS_PASS: <set to the key 'RCLONE_CONFIG_NAS_PASS' in secret 'restic-conf-zwave'> Optional: true
RCLONE_CONFIG_NAS_SHA1SUM_COMMAND: <set to the key 'RCLONE_CONFIG_NAS_SHA1SUM_COMMAND' in secret 'restic-conf-zwave'> Optional: true
RCLONE_CONFIG_NAS_SHELL_TYPE: <set to the key 'RCLONE_CONFIG_NAS_SHELL_TYPE' in secret 'restic-conf-zwave'> Optional: true
RCLONE_CONFIG_NAS_TYPE: <set to the key 'RCLONE_CONFIG_NAS_TYPE' in secret 'restic-conf-zwave'> Optional: true
RCLONE_CONFIG_NAS_USER: <set to the key 'RCLONE_CONFIG_NAS_USER' in secret 'restic-conf-zwave'> Optional: true
RCLONE_CONFIG_NAS_HOST: <set to the key 'RCLONE_CONFIG_NAS_HOST' in secret 'restic-conf-zwave'> Optional: true
PRIVILEGED_MOVER: 0
Mounts:
/cache from cache (rw)
/data from data (rw)
/tmp from tempdir (rw)
Volumes:
data:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: volsync-zwave-js-ui-src
ReadOnly: false
cache:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: volsync-zwave-js-ui-cache
ReadOnly: false
tempdir:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium: Memory
SizeLimit: <unset>
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal SuccessfulCreate 0s job-controller Created pod: volsync-src-zwave-js-ui-bvmsl
So, from what I can tell, it's angry at the job (for some reason?) and therefore deletes and remakes it. A lot. As a result I have a wall of terminating pods in my namespace.
Looks like they have a storage mount error because so many exist they can't share the same storage. Also I have 363 snapshots in restic now. My cron is set to once a half hour. So that's interesting.
So I figured I'd prune those, but:
repository 46b9c8a0 opened (version 2, compression level auto)
Fatal: unable to create lock in backend: repository is already locked by PID 19 on volsync-src-zwave-js-ui-w8gp7 by (UID 0, GID 0)
lock was created at 2023-04-28 19:36:16 (17m16.777376427s ago)
storage ID caea602d
Looks like this might be part of the issue, maybe?
I was able to fix this with
restic -r sftp:k8s@nas:/k8s/r/hass/zwave unlock --remove-all
Also I suspended it, deleted it, waiting for it to clear out the pods, then let it make it again.
However, creating the replicationsource again produces the same issue... Describe of pod that's terminating:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 13s default-scheduler Successfully assigned hass/volsync-src-zwave-js-ui-zmnnf to k8s-3
Warning FailedAttachVolume 14s attachdetach-controller Multi-Attach error for volume "pvc-07434137-a269-4927-92f9-53ab07aa76c3" Volume is already exclusively attached to one node and can't be attached to another
Warning FailedAttachVolume 14s attachdetach-controller Multi-Attach error for volume "pvc-80159c94-525d-4298-9c5f-2f320fc5bfc4" Volume is already exclusively attached to one node and can't be attached to another
Normal Pulled 6s kubelet Container image "quay.io/backube/volsync:0.7.1" already present on machine
Normal Created 5s kubelet Created container restic
Normal Started 4s kubelet Started container restic
Normal Killing 4s kubelet Stopping container restic
Snapshots list:
repository 46b9c8a0 opened (version 2, compression level auto)
ID Time Host Tags Paths
------------------------------------------------------------
f2d2dcc4 2023-04-28 15:50:29 volsync /data
96171e9d 2023-04-28 15:58:31 volsync /data
906f1d58 2023-04-28 15:58:33 volsync /data
2f11445d 2023-04-28 15:58:34 volsync /data
cf563533 2023-04-28 15:58:34 volsync /data
fc9893fd 2023-04-28 15:58:39 volsync /data
9c71ef6e 2023-04-28 15:58:42 volsync /data
a0a69085 2023-04-28 15:58:45 volsync /data
261d65aa 2023-04-28 15:58:45 volsync /data
d0d97009 2023-04-28 15:58:45 volsync /data
7304d206 2023-04-28 15:58:47 volsync /data
081d9cef 2023-04-28 15:58:49 volsync /data
cf53d49f 2023-04-28 15:58:49 volsync /data
8f8845ca 2023-04-28 15:58:53 volsync /data
f34b0aa1 2023-04-28 15:58:54 volsync /data
075114da 2023-04-28 15:58:55 volsync /data
ac592a90 2023-04-28 15:58:56 volsync /data
270b90b5 2023-04-28 15:58:58 volsync /data
1b52061f 2023-04-28 15:59:02 volsync /data
f872a95c 2023-04-28 15:59:02 volsync /data
5b9ee331 2023-04-28 15:59:08 volsync /data
3d199aad 2023-04-28 15:59:08 volsync /data
18cd328e 2023-04-28 15:59:10 volsync /data
3f30f6e8 2023-04-28 15:59:11 volsync /data
6e06198c 2023-04-28 15:59:14 volsync /data
228d6fa3 2023-04-28 15:59:18 volsync /data
2dbed603 2023-04-28 15:59:20 volsync /data
57a9ddc3 2023-04-28 15:59:21 volsync /data
b76adb34 2023-04-28 15:59:22 volsync /data
fccfea00 2023-04-28 15:59:23 volsync /data
83866d42 2023-04-28 15:59:24 volsync /data
4f8ab7ae 2023-04-28 15:59:24 volsync /data
91aa304d 2023-04-28 15:59:25 volsync /data
61039727 2023-04-28 15:59:25 volsync /data
289cef69 2023-04-28 15:59:25 volsync /data
------------------------------------------------------------
35 snapshots
As far as I can tell, the backups are succeeding... ?
Does this behavior make any sense to you? What debug output would you like?
First guess was that you managed to patch it when the job was still running, and the reconciles had trouble updating the job (originally started with old env vars) with the new env vars. It may have gotten into a loop getting errors because the job has changed (perhaps finished) while thinking it needed to be updated with the new vars?
but - I think you're saying you can get this every time? Even with a new replicationsource? Maybe something is wrong with the patch - can you send me the errors you see in the volsync controller logs when it's trying to create/update the job?
@tesshuflower Nah, I deleted it entirely and re-created it, so wasn't that first thing.
Oh. That controller log got angry really fast...
r{Name:\"OS_PASSWORD\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc026895e20)}, core.EnvVar{Name:\"OS_TENANT_ID\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc026895e40)}, core.EnvVar{Name:\"OS_TENANT_NAME\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc026895e60)}, core.EnvVar{Name:\"OS_USER_DOMAIN_NAME\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc026895e80)}, core.EnvVar{Name:\"OS_USER_DOMAIN_ID\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc026895ea0)}, core.EnvVar{Name:\"OS_PROJECT_NAME\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc026895ec0)}, core.EnvVar{Name:\"OS_PROJECT_DOMAIN_NAME\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc026895ee0)}, core.EnvVar{Name:\"OS_PROJECT_DOMAIN_ID\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc026895f00)}, core.EnvVar{Name:\"OS_TRUST_ID\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc026895f20)}, core.EnvVar{Name:\"OS_APPLICATION_CREDENTIAL_ID\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc026895f40)}, core.EnvVar{Name:\"OS_APPLICATION_CREDENTIAL_NAME\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc026895f60)}, core.EnvVar{Name:\"OS_APPLICATION_CREDENTIAL_SECRET\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc026895f80)}, core.EnvVar{Name:\"OS_STORAGE_URL\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc026895fa0)}, core.EnvVar{Name:\"OS_AUTH_TOKEN\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc026895fc0)}, core.EnvVar{Name:\"B2_ACCOUNT_ID\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc026895fe0)}, core.EnvVar{Name:\"B2_ACCOUNT_KEY\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc026e58000)}, core.EnvVar{Name:\"AZURE_ACCOUNT_NAME\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc026e58020)}, core.EnvVar{Name:\"AZURE_ACCOUNT_KEY\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc026e58040)}, core.EnvVar{Name:\"AZURE_ACCOUNT_SAS\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc026e58060)}, core.EnvVar{Name:\"GOOGLE_PROJECT_ID\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc026e58080)}, core.EnvVar{Name:\"RCLONE_CONFIG_NAS_HOST\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc026e580a0)}, core.EnvVar{Name:\"RCLONE_CONFIG_NAS_SHA1SUM_COMMAND\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc026e580c0)}, core.EnvVar{Name:\"RCLONE_CONFIG_NAS_USER\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc026e580e0)}, core.EnvVar{Name:\"RCLONE_CONFIG_NAS_MD5SUM_COMMAND\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc026e58100)}, core.EnvVar{Name:\"RCLONE_CONFIG_NAS_PORT\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc026e58120)}, core.EnvVar{Name:\"RCLONE_CONFIG_NAS_PASS\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc026e58140)}, core.EnvVar{Name:\"RCLONE_CONFIG_NAS_SHELL_TYPE\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc026e58160)}, core.EnvVar{Name:\"RCLONE_CONFIG_NAS_TYPE\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc026e58180)}, core.EnvVar{Name:\"PRIVILEGED_MOVER\", Value:\"0\", ValueFrom:(*core.EnvVarSource)(nil)}}, Resources:core.ResourceRequirements{Limits:core.ResourceList(nil), Requests:core.ResourceList(nil), Claims:[]core.ResourceClaim(nil)}, VolumeMounts:[]core.VolumeMount{core.VolumeMount{Name:\"data\", ReadOnly:false, MountPath:\"/data\", SubPath:\"\", MountPropagation:(*core.MountPropagationMode)(nil), SubPathExpr:\"\"}, core.VolumeMount{Name:\"cache\", ReadOnly:false, MountPath:\"/cache\", SubPath:\"\", MountPropagation:(*core.MountPropagationMode)(nil), SubPathExpr:\"\"}, core.VolumeMount{Name:\"tempdir\", ReadOnly:false, MountPath:\"/tmp\", SubPath:\"\", MountPropagation:(*core.MountPropagationMode)(nil), SubPathExpr:\"\"}}, VolumeDevices:[]core.VolumeDevice(nil), LivenessProbe:(*core.Probe)(nil), ReadinessProbe:(*core.Probe)(nil), StartupProbe:(*core.Probe)(nil), Lifecycle:(*core.Lifecycle)(nil), TerminationMessagePath:\"/dev/termination-log\", TerminationMessagePolicy:\"File\", ImagePullPolicy:\"IfNotPresent\", SecurityContext:(*core.SecurityContext)(0xc012d77e60), Stdin:false, StdinOnce:false, TTY:false}}, EphemeralContainers:[]core.EphemeralContainer(nil), RestartPolicy:\"Never\", TerminationGracePeriodSeconds:(*int64)(0xc018dffdb0), ActiveDeadlineSeconds:(*int64)(nil), DNSPolicy:\"ClusterFirst\", NodeSelector:map[string]string(nil), ServiceAccountName:\"volsync-src-zwave-js-ui\", AutomountServiceAccountToken:(*bool)(nil), NodeName:\"\", SecurityContext:(*core.PodSecurityContext)(0xc04dbefb90), ImagePullSecrets:[]core.LocalObjectReference(nil), Hostname:\"\", Subdomain:\"\", SetHostnameAsFQDN:(*bool)(nil), Affinity:(*core.Affinity)(nil), SchedulerName:\"default-scheduler\", Tolerations:[]core.Toleration(nil), HostAliases:[]core.HostAlias(nil), PriorityClassName:\"\", Priority:(*int32)(nil), PreemptionPolicy:(*core.PreemptionPolicy)(nil), DNSConfig:(*core.PodDNSConfig)(nil), ReadinessGates:[]core.PodReadinessGate(nil), RuntimeClassName:(*string)(nil), Overhead:core.ResourceList(nil), EnableServiceLinks:(*bool)(nil), TopologySpreadConstraints:[]core.TopologySpreadConstraint(nil), OS:(*core.PodOS)(nil), SchedulingGates:[]core.PodSchedulingGate(nil), ResourceClaims:[]core.PodResourceClaim(nil)}}: field is immutable"}
github.com/backube/volsync/controllers/utils.CreateOrUpdateDeleteOnImmutableErr
/workspace/controllers/utils/reconcile.go:57
github.com/backube/volsync/controllers/mover/restic.(*Mover).ensureJob
/workspace/controllers/mover/restic/mover.go:280
github.com/backube/volsync/controllers/mover/restic.(*Mover).Synchronize
/workspace/controllers/mover/restic/mover.go:138
github.com/backube/volsync/controllers.(*rsMachine).Synchronize
/workspace/controllers/replicationsource_controller.go:264
github.com/backube/volsync/controllers/statemachine.doSynchronizingState
/workspace/controllers/statemachine/machine.go:102
github.com/backube/volsync/controllers/statemachine.Run
/workspace/controllers/statemachine/machine.go:70
github.com/backube/volsync/controllers.(*ReplicationSourceReconciler).Reconcile
/workspace/controllers/replicationsource_controller.go:138
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-28T21:00:05.424Z ERROR controllers.ReplicationSource reconcile failed {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "job": "hass/volsync-src-zwave-js-ui", "error": "unable to update object. Deleting object so it can be recreated"}
github.com/backube/volsync/controllers/mover/restic.(*Mover).ensureJob
/workspace/controllers/mover/restic/mover.go:519
github.com/backube/volsync/controllers/mover/restic.(*Mover).Synchronize
/workspace/controllers/mover/restic/mover.go:138
github.com/backube/volsync/controllers.(*rsMachine).Synchronize
/workspace/controllers/replicationsource_controller.go:264
github.com/backube/volsync/controllers/statemachine.doSynchronizingState
/workspace/controllers/statemachine/machine.go:102
github.com/backube/volsync/controllers/statemachine.Run
/workspace/controllers/statemachine/machine.go:70
github.com/backube/volsync/controllers.(*ReplicationSourceReconciler).Reconcile
/workspace/controllers/replicationsource_controller.go:138
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-28T21:00:05.460Z ERROR Reconciler error {"controller": "replicationsource", "controllerGroup": "volsync.backube", "controllerKind": "ReplicationSource", "ReplicationSource": {"name":"zwave-js-ui","namespace":"hass"}, "namespace": "hass", "name": "zwave-js-ui", "reconcileID": "892edf23-029d-4453-9b11-3d768109785e", "error": "unable to update object. Deleting object so it can be recreated"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-28T21:00:05.460Z INFO controllers.ReplicationSource Namespace does not indicate volsync privileged movers are allowed, will assume privileged movers are not allowed. Annotation missing or not equal to 'true' {"replicationsource": "hass/zwave-js-ui", "namespace": "hass", "Annotation": "volsync.backube/privileged-movers"}
2023-04-28T21:00:05.460Z DEBUG controllers.ReplicationSource temporary snapshot reconciled {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "snapshot": "hass/volsync-zwave-js-ui-src", "operation": "unchanged"}
2023-04-28T21:00:05.460Z DEBUG controllers.ReplicationSource pvc from snap reconciled {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "pvc": "hass/volsync-zwave-js-ui-src", "operation": "unchanged"}
2023-04-28T21:00:05.460Z INFO controllers.ReplicationSource allocating cache volume {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "PVC": "volsync-zwave-js-ui-cache"}
2023-04-28T21:00:05.460Z DEBUG controllers.ReplicationSource PVC reconciled {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "PVC": "volsync-zwave-js-ui-cache", "operation": "unchanged"}
2023-04-28T21:00:05.461Z DEBUG controllers.ReplicationSource ServiceAccount reconciled {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "ServiceAccount": "hass/volsync-src-zwave-js-ui", "operation": "unchanged"}
2023-04-28T21:00:05.461Z DEBUG controllers.ReplicationSource Role reconciled {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "Role": "hass/volsync-src-zwave-js-ui", "operation": "unchanged"}
2023-04-28T21:00:05.461Z DEBUG controllers.ReplicationSource RoleBinding reconciled {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "RoleBinding": "hass/volsync-src-zwave-js-ui", "operation": "unchanged"}
2023-04-28T21:00:05.461Z INFO controllers.ReplicationSource job actions {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "job": "hass/volsync-src-zwave-js-ui", "actions": ["backup"]}
2023-04-28T21:00:05.520Z INFO controllers.ReplicationSource Namespace does not indicate volsync privileged movers are allowed, will assume privileged movers are not allowed. Annotation missing or not equal to 'true' {"replicationsource": "hass/zwave-js-ui", "namespace": "hass", "Annotation": "volsync.backube/privileged-movers"}
2023-04-28T21:00:05.520Z DEBUG controllers.ReplicationSource temporary snapshot reconciled {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "snapshot": "hass/volsync-zwave-js-ui-src", "operation": "unchanged"}
2023-04-28T21:00:05.521Z DEBUG controllers.ReplicationSource pvc from snap reconciled {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "pvc": "hass/volsync-zwave-js-ui-src", "operation": "unchanged"}
2023-04-28T21:00:05.521Z INFO controllers.ReplicationSource allocating cache volume {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "PVC": "volsync-zwave-js-ui-cache"}
2023-04-28T21:00:05.521Z DEBUG controllers.ReplicationSource PVC reconciled {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "PVC": "volsync-zwave-js-ui-cache", "operation": "unchanged"}
2023-04-28T21:00:05.521Z DEBUG controllers.ReplicationSource ServiceAccount reconciled {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "ServiceAccount": "hass/volsync-src-zwave-js-ui", "operation": "unchanged"}
2023-04-28T21:00:05.521Z DEBUG controllers.ReplicationSource Role reconciled {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "Role": "hass/volsync-src-zwave-js-ui", "operation": "unchanged"}
2023-04-28T21:00:05.521Z DEBUG controllers.ReplicationSource RoleBinding reconciled {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "RoleBinding": "hass/volsync-src-zwave-js-ui", "operation": "unchanged"}
2023-04-28T21:00:05.521Z INFO controllers.ReplicationSource job actions {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "job": "hass/volsync-src-zwave-js-ui", "actions": ["backup"]}
2023-04-28T21:00:05.532Z ERROR controllers.ReplicationSource Immutable error updating the object. Will delete so it can be recreated {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "job": "hass/volsync-src-zwave-js-ui", "error": "Job.batch \"volsync-src-zwave-js-ui\" is invalid: spec.template: Invalid value: core.PodTemplateSpec{ObjectMeta:v1.ObjectMeta{Name:\"volsync-src-zwave-js-ui\", GenerateName:\"\", Namespace:\"\", SelfLink:\"\", UID:\"\", ResourceVersion:\"\", Generation:0, CreationTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeletionTimestamp:<nil>, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{\"app.kubernetes.io/created-by\":\"volsync\", \"controller-uid\":\"2a1354ae-fb90-4fee-9c68-9240b0f93942\", \"job-name\":\"volsync-src-zwave-js-ui\"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:core.PodSpec{Volumes:[]core.Volume{core.Volume{Name:\"data\", VolumeSource:core.VolumeSource{HostPath:(*core.HostPathVolumeSource)(nil), EmptyDir:(*core.EmptyDirVolumeSource)(nil), GCEPersistentDisk:(*core.GCEPersistentDiskVolumeSource)(nil), AWSElasticBlockStore:(*core.AWSElasticBlockStoreVolumeSource)(nil), GitRepo:(*core.GitRepoVolumeSource)(nil), Secret:(*core.SecretVolumeSource)(nil), NFS:(*core.NFSVolumeSource)(nil), ISCSI:(*core.ISCSIVolumeSource)(nil), Glusterfs:(*core.GlusterfsVolumeSource)(nil), PersistentVolumeClaim:(*core.PersistentVolumeClaimVolumeSource)(0xc0084c19c8), RBD:(*core.RBDVolumeSource)(nil), Quobyte:(*core.QuobyteVolumeSource)(nil), FlexVolume:(*core.FlexVolumeSource)(nil), Cinder:(*core.CinderVolumeSource)(nil), CephFS:(*core.CephFSVolumeSource)(nil), Flocker:(*core.FlockerVolumeSource)(nil), DownwardAPI:(*core.DownwardAPIVolumeSource)(nil), FC:(*core.FCVolumeSource)(nil), AzureFile:(*core.AzureFileVolumeSource)(nil), ConfigMap:(*core.ConfigMapVolumeSource)(nil), VsphereVolume:(*core.VsphereVirtualDiskVolumeSource)(nil), AzureDisk:(*core.AzureDiskVolumeSource)(nil), PhotonPersistentDisk:(*core.PhotonPersistentDiskVolumeSource)(nil), Projected:(*core.ProjectedVolumeSource)(nil), PortworxVolume:(*core.PortworxVolumeSource)(nil), ScaleIO:(*core.ScaleIOVolumeSource)(nil), StorageOS:(*core.StorageOSVolumeSource)(nil), CSI:(*core.CSIVolumeSource)(nil), Ephemeral:(*core.EphemeralVolumeSource)(nil)}}, core.Volume{Name:\"cache\", VolumeSource:core.VolumeSource{HostPath:(*core.HostPathVolumeSource)(nil), EmptyDir:(*core.EmptyDirVolumeSource)(nil), GCEPersistentDisk:(*core.GCEPersistentDiskVolumeSource)(nil), AWSElasticBlockStore:(*core.AWSElasticBlockStoreVolumeSource)(nil), GitRepo:(*core.GitRepoVolumeSource)(nil), Secret:(*core.SecretVolumeSource)(nil), NFS:(*core.NFSVolumeSource)(nil), ISCSI:(*core.ISCSIVolumeSource)(nil), Glusterfs:(*core.GlusterfsVolumeSource)(nil), PersistentVolumeClaim:(*core.PersistentVolumeClaimVolumeSource)(0xc0084c19e0), RBD:(*core.RBDVolumeSource)(nil), Quobyte:(*core.QuobyteVolumeSource)(nil), FlexVolume:(*core.FlexVolumeSource)(nil), Cinder:(*core.CinderVolumeSource)(nil), CephFS:(*core.CephFSVolumeSource)(nil), Flocker:(*core.FlockerVolumeSource)(nil), DownwardAPI:(*core.DownwardAPIVolumeSource)(nil), FC:(*core.FCVolumeSource)(nil), AzureFile:(*core.AzureFileVolumeSource)(nil), ConfigMap:(*core.ConfigMapVolumeSource)(nil), VsphereVolume:(*core.VsphereVirtualDiskVolumeSource)(nil), AzureDisk:(*core.AzureDiskVolumeSource)(nil), PhotonPersistentDisk:(*core.PhotonPersistentDiskVolumeSource)(nil), Projected:(*core.ProjectedVolumeSource)(nil), PortworxVolume:(*core.PortworxVolumeSource)(nil), ScaleIO:(*core.ScaleIOVolumeSource)(nil), StorageOS:(*core.StorageOSVolumeSource)(nil), CSI:(*core.CSIVolumeSource)(nil), Ephemeral:(*core.EphemeralVolumeSource)(nil)}}, core.Volume{Name:\"tempdir\", VolumeSource:core.VolumeSource{HostPath:(*core.HostPathVolumeSource)(nil), EmptyDir:(*core.EmptyDirVolumeSource)(0xc0084c19f8), GCEPersistentDisk:(*core.GCEPersistentDiskVolumeSource)(nil), AWSElasticBlockStore:(*core.AWSElasticBlockStoreVolumeSource)(nil), GitRepo:(*core.GitRepoVolumeSource)(nil), Secret:(*core.SecretVolumeSource)(nil), NFS:(*core.NFSVolumeSource)(nil), ISCSI:(*core.ISCSIVolumeSource)(nil), Glusterfs:(*core.GlusterfsVolumeSource)(nil), PersistentVolumeClaim:(*core.PersistentVolumeClaimVolumeSource)(nil), RBD:(*core.RBDVolumeSource)(nil), Quobyte:(*core.QuobyteVolumeSource)(nil), FlexVolume:(*core.FlexVolumeSource)(nil), Cinder:(*core.CinderVolumeSource)(nil), CephFS:(*core.CephFSVolumeSource)(nil), Flocker:(*core.FlockerVolumeSource)(nil), DownwardAPI:(*core.DownwardAPIVolumeSource)(nil), FC:(*core.FCVolumeSource)(nil), AzureFile:(*core.AzureFileVolumeSource)(nil), ConfigMap:(*core.ConfigMapVolumeSource)(nil), VsphereVolume:(*core.VsphereVirtualDiskVolumeSource)(nil), AzureDisk:(*core.AzureDiskVolumeSource)(nil), PhotonPersistentDisk:(*core.PhotonPersistentDiskVolumeSource)(nil), Projected:(*core.ProjectedVolumeSource)(nil), PortworxVolume:(*core.PortworxVolumeSource)(nil), ScaleIO:(*core.ScaleIOVolumeSource)(nil), StorageOS:(*core.StorageOSVolumeSource)(nil), CSI:(*core.CSIVolumeSource)(nil), Ephemeral:(*core.EphemeralVolumeSource)(nil)}}}, InitContainers:[]core.Container(nil), Containers:[]core.Container{core.Container{Name:\"restic\", Image:\"quay.io/backube/volsync:0.7.1\", Command:[]string{\"/mover-restic/entry.sh\"}, Args:[]string{\"backup\"}, WorkingDir:\"\", Ports:[]core.ContainerPort(nil), EnvFrom:[]core.EnvFromSource(nil), Env:[]core.EnvVar{core.EnvVar{Name:\"FORGET_OPTIONS\", Value:\" --keep-daily 10 --keep-weekly 6 --keep-monthly 4 --keep-within 3d\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"DATA_DIR\", Value:\"/data\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"RESTIC_CACHE_DIR\", Value:\"/cache\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"RESTORE_AS_OF\", Value:\"\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"SELECT_PREVIOUS\", Value:\"0\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"RESTIC_REPOSITORY\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc026cdbfa0)}, core.EnvVar{Name:\"RESTIC_PASSWORD\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc026cdbfc0)}, core.EnvVar{Name:\"RESTIC_COMPRESSION\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc026cdbfe0)}, core.EnvVar{Name:\"RESTIC_PACK_SIZE\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc027eba000)}, core.EnvVar{Name:\"RESTIC_READ_CONCURRENCY\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc027eba020)}, core.EnvVar{Name:\"AWS_ACCESS_KEY_ID\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc027eba040)}, core.EnvVar{Name:\"AWS_SECRET_ACCESS_KEY\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc027eba060)}, core.EnvVar{Name:\"AWS_SESSION_TOKEN\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc027eba080)}, core.EnvVar{Name:\"AWS_DEFAULT_REGION\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc027eba0a0)}, core.EnvVar{Name:\"ST_AUTH\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc027eba0c0)}, core.EnvVar{Name:\"ST_USER\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc027eba0e0)}, core.EnvVar{Name:\"ST_KEY\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc027eba100)}, core.EnvVar{Name:\"OS_AUTH_URL\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc027eba120)}, core.EnvVar{Name:\"OS_REGION_NAME\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc027eba140)}, core.EnvVar{Name:\"OS_USERNAME\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc027eba160)}, core.EnvVar{Name:\"OS_USER_ID\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc027eba180)}, core.EnvVar{Name:\"OS_PASSWORD\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc027eba1a0)}, core.EnvVar{Name:\"OS_TENANT_ID\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc027eba1c0)}, core.EnvVar{Name:\"OS_TENANT_NAME\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc027eba1e0)}, core.EnvVar{Name:\"OS_USER_DOMAIN_NAME\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc027eba200)}, core.EnvVar{Name:\"OS_USER_DOMAIN_ID\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc027eba220)}, core.EnvVar{Name:\"OS_PROJECT_NAME\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc027eba240)}, core.EnvVar{Name:\"OS_PROJECT_DOMAIN_NAME\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc027eba260)}, core.EnvVar{Name:\"OS_PROJECT_DOMAIN_ID\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc027eba280)}, core.EnvVar{Name:\"OS_TRUST_ID\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc027eba2a0)}, core.EnvVar{Name:\"OS_APPLICATION_CREDENTIAL_ID\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc027eba2c0)}, core.EnvVar{Name:\"OS_APPLICATION_CREDENTIAL_NAME\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc027eba2e0)}, core.EnvVar{Name:\"OS_APPLICATION_CREDENTIAL_SECRET\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc027eba300)}, core.EnvVar{Name:\"OS_STORAGE_URL\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc027eba320)}, core.EnvVar{Name:\"OS_AUTH_TOKEN\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc027eba340)}, core.EnvVar{Name:\"B2_ACCOUNT_ID\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc027eba360)}, core.EnvVar{Name:\"B2_ACCOUNT_KEY\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc027eba380)}, core.EnvVar{Name:\"AZURE_ACCOUNT_NAME\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc027eba3a0)}, core.EnvVar{Name:\"AZURE_ACCOUNT_KEY\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc027eba3c0)}, core.EnvVar{Name:\"AZURE_ACCOUNT_SAS\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc027eba3e0)}, core.EnvVar{Name:\"GOOGLE_PROJECT_ID\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc027eba400)}, core.EnvVar{Name:\"RCLONE_CONFIG_NAS_SHELL_TYPE\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc027eba420)}, core.EnvVar{Name:\"RCLONE_CONFIG_NAS_TYPE\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc027eba440)}, core.EnvVar{Name:\"RCLONE_CONFIG_NAS_USER\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc027eba460)}, core.EnvVar{Name:\"RCLONE_CONFIG_NAS_PORT\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc027eba480)}, core.EnvVar{Name:\"RCLONE_CONFIG_NAS_MD5SUM_COMMAND\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc027eba4a0)}, core.EnvVar{Name:\"RCLONE_CONFIG_NAS_PASS\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc027eba4c0)}, core.EnvVar{Name:\"RCLONE_CONFIG_NAS_SHA1SUM_COMMAND\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc027eba4e0)}, core.EnvVar{Name:\"RCLONE_CONFIG_NAS_HOST\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc027eba500)}, core.EnvVar{Name:\"PRIVILEGED_MOVER\", Value:\"0\", ValueFrom:(*core.EnvVarSource)(nil)}}, Resources:core.ResourceRequirements{Limits:core.ResourceList(nil), Requests:core.ResourceList(nil), Claims:[]core.ResourceClaim(nil)}, VolumeMounts:[]core.VolumeMount{core.VolumeMount{Name:\"data\", ReadOnly:false, MountPath:\"/data\", SubPath:\"\", MountPropagation:(*core.MountPropagationMode)(nil), SubPathExpr:\"\"}, core.VolumeMount{Name:\"cache\", ReadOnly:false, MountPath:\"/cache\", SubPath:\"\", MountPropagation:(*core.MountPropagationMode)(nil), SubPathExpr:\"\"}, core.VolumeMount{Name:\"tempdir\", ReadOnly:false, MountPath:\"/tmp\", SubPath:\"\", MountPropagation:(*core.MountPropagationMode)(nil), SubPathExpr:\"\"}}, VolumeDevices:[]core.VolumeDevice(nil), LivenessProbe:(*core.Probe)(nil), ReadinessProbe:(*core.Probe)(nil), StartupProbe:(*core.Probe)(nil), Lifecycle:(*core.Lifecycle)(nil), TerminationMessagePath:\"/dev/termination-log\", TerminationMessagePolicy:\"File\", ImagePullPolicy:\"IfNotPresent\", SecurityContext:(*core.SecurityContext)(0xc00dac0c60), Stdin:false, StdinOnce:false, TTY:false}}, EphemeralContainers:[]core.EphemeralContainer(nil), RestartPolicy:\"Never\", TerminationGracePeriodSeconds:(*int64)(0xc03badbd90), ActiveDeadlineSeconds:(*int64)(nil), DNSPolicy:\"ClusterFirst\", NodeSelector:map[string]string(nil), ServiceAccountName:\"volsync-src-zwave-js-ui\", AutomountServiceAccountToken:(*bool)(nil), NodeName:\"\", SecurityContext:(*core.PodSecurityContext)(0xc01d0398c0), ImagePullSecrets:[]core.LocalObjectReference(nil), Hostname:\"\", Subdomain:\"\", SetHostnameAsFQDN:(*bool)(nil), Affinity:(*core.Affinity)(nil), SchedulerName:\"default-scheduler\", Tolerations:[]core.Toleration(nil), HostAliases:[]core.HostAlias(nil), PriorityClassName:\"\", Priority:(*int32)(nil), PreemptionPolicy:(*core.PreemptionPolicy)(nil), DNSConfig:(*core.PodDNSConfig)(nil), ReadinessGates:[]core.PodReadinessGate(nil), RuntimeClassName:(*string)(nil), Overhead:core.ResourceList(nil), EnableServiceLinks:(*bool)(nil), TopologySpreadConstraints:[]core.TopologySpreadConstraint(nil), OS:(*core.PodOS)(nil), SchedulingGates:[]core.PodSchedulingGate(nil), ResourceClaims:[]core.PodResourceClaim(nil)}}: field is immutable"}
github.com/backube/volsync/controllers/utils.CreateOrUpdateDeleteOnImmutableErr
/workspace/controllers/utils/reconcile.go:57
github.com/backube/volsync/controllers/mover/restic.(*Mover).ensureJob
/workspace/controllers/mover/restic/mover.go:280
github.com/backube/volsync/controllers/mover/restic.(*Mover).Synchronize
/workspace/controllers/mover/restic/mover.go:138
github.com/backube/volsync/controllers.(*rsMachine).Synchronize
/workspace/controllers/replicationsource_controller.go:264
github.com/backube/volsync/controllers/statemachine.doSynchronizingState
/workspace/controllers/statemachine/machine.go:102
github.com/backube/volsync/controllers/statemachine.Run
/workspace/controllers/statemachine/machine.go:70
github.com/backube/volsync/controllers.(*ReplicationSourceReconciler).Reconcile
/workspace/controllers/replicationsource_controller.go:138
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-28T21:00:05.550Z ERROR controllers.ReplicationSource reconcile failed {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "job": "hass/volsync-src-zwave-js-ui", "error": "unable to update object. Deleting object so it can be recreated"}
github.com/backube/volsync/controllers/mover/restic.(*Mover).ensureJob
/workspace/controllers/mover/restic/mover.go:519
github.com/backube/volsync/controllers/mover/restic.(*Mover).Synchronize
/workspace/controllers/mover/restic/mover.go:138
github.com/backube/volsync/controllers.(*rsMachine).Synchronize
/workspace/controllers/replicationsource_controller.go:264
github.com/backube/volsync/controllers/statemachine.doSynchronizingState
/workspace/controllers/statemachine/machine.go:102
github.com/backube/volsync/controllers/statemachine.Run
/workspace/controllers/statemachine/machine.go:70
github.com/backube/volsync/controllers.(*ReplicationSourceReconciler).Reconcile
/workspace/controllers/replicationsource_controller.go:138
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-28T21:00:05.579Z ERROR Reconciler error {"controller": "replicationsource", "controllerGroup": "volsync.backube", "controllerKind": "ReplicationSource", "ReplicationSource": {"name":"zwave-js-ui","namespace":"hass"}, "namespace": "hass", "name": "zwave-js-ui", "reconcileID": "eb62eea0-6e60-4f20-ab56-7e0798dbe17d", "error": "unable to update object. Deleting object so it can be recreated"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-28T21:00:05.579Z INFO controllers.ReplicationSource Namespace does not indicate volsync privileged movers are allowed, will assume privileged movers are not allowed. Annotation missing or not equal to 'true' {"replicationsource": "hass/zwave-js-ui", "namespace": "hass", "Annotation": "volsync.backube/privileged-movers"}
2023-04-28T21:00:05.579Z DEBUG controllers.ReplicationSource temporary snapshot reconciled {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "snapshot": "hass/volsync-zwave-js-ui-src", "operation": "unchanged"}
2023-04-28T21:00:05.579Z DEBUG controllers.ReplicationSource pvc from snap reconciled {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "pvc": "hass/volsync-zwave-js-ui-src", "operation": "unchanged"}
2023-04-28T21:00:05.579Z INFO controllers.ReplicationSource allocating cache volume {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "PVC": "volsync-zwave-js-ui-cache"}
2023-04-28T21:00:05.580Z DEBUG controllers.ReplicationSource PVC reconciled {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "PVC": "volsync-zwave-js-ui-cache", "operation": "unchanged"}
2023-04-28T21:00:05.580Z DEBUG controllers.ReplicationSource ServiceAccount reconciled {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "ServiceAccount": "hass/volsync-src-zwave-js-ui", "operation": "unchanged"}
2023-04-28T21:00:05.580Z DEBUG controllers.ReplicationSource Role reconciled {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "Role": "hass/volsync-src-zwave-js-ui", "operation": "unchanged"}
2023-04-28T21:00:05.580Z DEBUG controllers.ReplicationSource RoleBinding reconciled {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "RoleBinding": "hass/volsync-src-zwave-js-ui", "operation": "unchanged"}
2023-04-28T21:00:05.580Z INFO controllers.ReplicationSource job actions {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "job": "hass/volsync-src-zwave-js-ui", "actions": ["backup"]}
2023-04-28T21:00:05.658Z INFO controllers.ReplicationSource Namespace does not indicate volsync privileged movers are allowed, will assume privileged movers are not allowed. Annotation missing or not equal to 'true' {"replicationsource": "hass/zwave-js-ui", "namespace": "hass", "Annotation": "volsync.backube/privileged-movers"}
2023-04-28T21:00:05.658Z DEBUG controllers.ReplicationSource temporary snapshot reconciled {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "snapshot": "hass/volsync-zwave-js-ui-src", "operation": "unchanged"}
2023-04-28T21:00:05.658Z DEBUG controllers.ReplicationSource pvc from snap reconciled {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "pvc": "hass/volsync-zwave-js-ui-src", "operation": "unchanged"}
2023-04-28T21:00:05.658Z INFO controllers.ReplicationSource allocating cache volume {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "PVC": "volsync-zwave-js-ui-cache"}
2023-04-28T21:00:05.659Z DEBUG controllers.ReplicationSource PVC reconciled {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "PVC": "volsync-zwave-js-ui-cache", "operation": "unchanged"}
2023-04-28T21:00:05.659Z DEBUG controllers.ReplicationSource ServiceAccount reconciled {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "ServiceAccount": "hass/volsync-src-zwave-js-ui", "operation": "unchanged"}
2023-04-28T21:00:05.659Z DEBUG controllers.ReplicationSource Role reconciled {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "Role": "hass/volsync-src-zwave-js-ui", "operation": "unchanged"}
2023-04-28T21:00:05.659Z DEBUG controllers.ReplicationSource RoleBinding reconciled {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "RoleBinding": "hass/volsync-src-zwave-js-ui", "operation": "unchanged"}
2023-04-28T21:00:05.659Z INFO controllers.ReplicationSource job actions {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "job": "hass/volsync-src-zwave-js-ui", "actions": ["backup"]}
2023-04-28T21:00:05.669Z ERROR controllers.ReplicationSource Immutable error updating the object. Will delete so it can be recreated {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "job": "hass/volsync-src-zwave-js-ui", "error": "Job.batch \"volsync-src-zwave-js-ui\" is invalid: spec.template: Invalid value: core.PodTemplateSpec{ObjectMeta:v1.ObjectMeta{Name:\"volsync-src-zwave-js-ui\", GenerateName:\"\", Namespace:\"\", SelfLink:\"\", UID:\"\", ResourceVersion:\"\", Generation:0, CreationTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeletionTimestamp:<nil>, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{\"app.kubernetes.io/created-by\":\"volsync\", \"controller-uid\":\"fd66afa7-370b-405a-98f5-49ae8d08f4bd\", \"job-name\":\"volsync-src-zwave-js-ui\"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:core.PodSpec{Volumes:[]core.Volume{core.Volume{Name:\"data\", VolumeSource:core.VolumeSource{HostPath:(*core.HostPathVolumeSource)(nil), EmptyDir:(*core.EmptyDirVolumeSource)(nil), GCEPersistentDisk:(*core.GCEPersistentDiskVolumeSource)(nil), AWSElasticBlockStore:(*core.AWSElasticBlockStoreVolumeSource)(nil), GitRepo:(*core.GitRepoVolumeSource)(nil), Secret:(*core.SecretVolumeSource)(nil), NFS:(*core.NFSVolumeSource)(nil), ISCSI:(*core.ISCSIVolumeSource)(nil), Glusterfs:(*core.GlusterfsVolumeSource)(nil), PersistentVolumeClaim:(*core.PersistentVolumeClaimVolumeSource)(0xc00e82b4a0), RBD:(*core.RBDVolumeSource)(nil), Quobyte:(*core.QuobyteVolumeSource)(nil), FlexVolume:(*core.FlexVolumeSource)(nil), Cinder:(*core.CinderVolumeSource)(nil), CephFS:(*core.CephFSVolumeSource)(nil), Flocker:(*core.FlockerVolumeSource)(nil), DownwardAPI:(*core.DownwardAPIVolumeSource)(nil), FC:(*core.FCVolumeSource)(nil), AzureFile:(*core.AzureFileVolumeSource)(nil), ConfigMap:(*core.ConfigMapVolumeSource)(nil), VsphereVolume:(*core.VsphereVirtualDiskVolumeSource)(nil), AzureDisk:(*core.AzureDiskVolumeSource)(nil), PhotonPersistentDisk:(*core.PhotonPersistentDiskVolumeSource)(nil), Projected:(*core.ProjectedVolumeSource)(nil), PortworxVolume:(*core.PortworxVolumeSource)(nil), ScaleIO:(*core.ScaleIOVolumeSource)(nil), StorageOS:(*core.StorageOSVolumeSource)(nil), CSI:(*core.CSIVolumeSource)(nil), Ephemeral:(*core.EphemeralVolumeSource)(nil)}}, core.Volume{Name:\"cache\", VolumeSource:core.VolumeSource{HostPath:(*core.HostPathVolumeSource)(nil), EmptyDir:(*core.EmptyDirVolumeSource)(nil), GCEPersistentDisk:(*core.GCEPersistentDiskVolumeSource)(nil), AWSElasticBlockStore:(*core.AWSElasticBlockStoreVolumeSource)(nil), GitRepo:(*core.GitRepoVolumeSource)(nil), Secret:(*core.SecretVolumeSource)(nil), NFS:(*core.NFSVolumeSource)(nil), ISCSI:(*core.ISCSIVolumeSource)(nil), Glusterfs:(*core.GlusterfsVolumeSource)(nil), PersistentVolumeClaim:(*core.PersistentVolumeClaimVolumeSource)(0xc00e82b4b8), RBD:(*core.RBDVolumeSource)(nil), Quobyte:(*core.QuobyteVolumeSource)(nil), FlexVolume:(*core.FlexVolumeSource)(nil), Cinder:(*core.CinderVolumeSource)(nil), CephFS:(*core.CephFSVolumeSource)(nil), Flocker:(*core.FlockerVolumeSource)(nil), DownwardAPI:(*core.DownwardAPIVolumeSource)(nil), FC:(*core.FCVolumeSource)(nil), AzureFile:(*core.AzureFileVolumeSource)(nil), ConfigMap:(*core.ConfigMapVolumeSource)(nil), VsphereVolume:(*core.VsphereVirtualDiskVolumeSource)(nil), AzureDisk:(*core.AzureDiskVolumeSource)(nil), PhotonPersistentDisk:(*core.PhotonPersistentDiskVolumeSource)(nil), Projected:(*core.ProjectedVolumeSource)(nil), PortworxVolume:(*core.PortworxVolumeSource)(nil), ScaleIO:(*core.ScaleIOVolumeSource)(nil), StorageOS:(*core.StorageOSVolumeSource)(nil), CSI:(*core.CSIVolumeSource)(nil), Ephemeral:(*core.EphemeralVolumeSource)(nil)}}, core.Volume{Name:\"tempdir\", VolumeSource:core.VolumeSource{HostPath:(*core.HostPathVolumeSource)(nil), EmptyDir:(*core.EmptyDirVolumeSource)(0xc00e82b4d0), GCEPersistentDisk:(*core.GCEPersistentDiskVolumeSource)(nil), AWSElasticBlockStore:(*core.AWSElasticBlockStoreVolumeSource)(nil), GitRepo:(*core.GitRepoVolumeSource)(nil), Secret:(*core.SecretVolumeSource)(nil), NFS:(*core.NFSVolumeSource)(nil), ISCSI:(*core.ISCSIVolumeSource)(nil), Glusterfs:(*core.GlusterfsVolumeSource)(nil), PersistentVolumeClaim:(*core.PersistentVolumeClaimVolumeSource)(nil), RBD:(*core.RBDVolumeSource)(nil), Quobyte:(*core.QuobyteVolumeSource)(nil), FlexVolume:(*core.FlexVolumeSource)(nil), Cinder:(*core.CinderVolumeSource)(nil), CephFS:(*core.CephFSVolumeSource)(nil), Flocker:(*core.FlockerVolumeSource)(nil), DownwardAPI:(*core.DownwardAPIVolumeSource)(nil), FC:(*core.FCVolumeSource)(nil), AzureFile:(*core.AzureFileVolumeSource)(nil), ConfigMap:(*core.ConfigMapVolumeSource)(nil), VsphereVolume:(*core.VsphereVirtualDiskVolumeSource)(nil), AzureDisk:(*core.AzureDiskVolumeSource)(nil), PhotonPersistentDisk:(*core.PhotonPersistentDiskVolumeSource)(nil), Projected:(*core.ProjectedVolumeSource)(nil), PortworxVolume:(*core.PortworxVolumeSource)(nil), ScaleIO:(*core.ScaleIOVolumeSource)(nil), StorageOS:(*core.StorageOSVolumeSource)(nil), CSI:(*core.CSIVolumeSource)(nil), Ephemeral:(*core.EphemeralVolumeSource)(nil)}}}, InitContainers:[]core.Container(nil), Containers:[]core.Container{core.Container{Name:\"restic\", Image:\"quay.io/backube/volsync:0.7.1\", Command:[]string{\"/mover-restic/entry.sh\"}, Args:[]string{\"backup\"}, WorkingDir:\"\", Ports:[]core.ContainerPort(nil), EnvFrom:[]core.EnvFromSource(nil), Env:[]core.EnvVar{core.EnvVar{Name:\"FORGET_OPTIONS\", Value:\" --keep-daily 10 --keep-weekly 6 --keep-monthly 4 --keep-within 3d\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"DATA_DIR\", Value:\"/data\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"RESTIC_CACHE_DIR\", Value:\"/cache\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"RESTORE_AS_OF\", Value:\"\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"SELECT_PREVIOUS\", Value:\"0\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"RESTIC_REPOSITORY\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02934e7a0)}, core.EnvVar{Name:\"RESTIC_PASSWORD\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02934e7c0)}, core.EnvVar{Name:\"RESTIC_COMPRESSION\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02934e7e0)}, core.EnvVar{Name:\"RESTIC_PACK_SIZE\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02934e800)}, core.EnvVar{Name:\"RESTIC_READ_CONCURRENCY\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02934e820)}, core.EnvVar{Name:\"AWS_ACCESS_KEY_ID\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02934e840)}, core.EnvVar{Name:\"AWS_SECRET_ACCESS_KEY\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02934e860)}, core.EnvVar{Name:\"AWS_SESSION_TOKEN\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02934e880)}, core.EnvVar{Name:\"AWS_DEFAULT_REGION\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02934e8a0)}, core.EnvVar{Name:\"ST_AUTH\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02934e8c0)}, core.EnvVar{Name:\"ST_USER\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02934e8e0)}, core.EnvVar{Name:\"ST_KEY\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02934e900)}, core.EnvVar{Name:\"OS_AUTH_URL\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02934e920)}, core.EnvVar{Name:\"OS_REGION_NAME\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02934e940)}, core.EnvVar{Name:\"OS_USERNAME\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02934e960)}, core.EnvVar{Name:\"OS_USER_ID\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02934e980)}, core.EnvVar{Name:\"OS_PASSWORD\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02934e9a0)}, core.EnvVar{Name:\"OS_TENANT_ID\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02934e9c0)}, core.EnvVar{Name:\"OS_TENANT_NAME\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02934e9e0)}, core.EnvVar{Name:\"OS_USER_DOMAIN_NAME\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02934ea00)}, core.EnvVar{Name:\"OS_USER_DOMAIN_ID\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02934ea20)}, core.EnvVar{Name:\"OS_PROJECT_NAME\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02934ea40)}, core.EnvVar{Name:\"OS_PROJECT_DOMAIN_NAME\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02934ea60)}, core.EnvVar{Name:\"OS_PROJECT_DOMAIN_ID\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02934ea80)}, core.EnvVar{Name:\"OS_TRUST_ID\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02934eaa0)}, core.EnvVar{Name:\"OS_APPLICATION_CREDENTIAL_ID\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02934eac0)}, core.EnvVar{Name:\"OS_APPLICATION_CREDENTIAL_NAME\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02934eae0)}, core.EnvVar{Name:\"OS_APPLICATION_CREDENTIAL_SECRET\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02934eb00)}, core.EnvVar{Name:\"OS_STORAGE_URL\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02934eb20)}, core.EnvVar{Name:\"OS_AUTH_TOKEN\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02934eb40)}, core.EnvVar{Name:\"B2_ACCOUNT_ID\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02934eb60)}, core.EnvVar{Name:\"B2_ACCOUNT_KEY\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02934eb80)}, core.EnvVar{Name:\"AZURE_ACCOUNT_NAME\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02934eba0)}, core.EnvVar{Name:\"AZURE_ACCOUNT_KEY\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02934ebc0)}, core.EnvVar{Name:\"AZURE_ACCOUNT_SAS\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02934ebe0)}, core.EnvVar{Name:\"GOOGLE_PROJECT_ID\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02934ec00)}, core.EnvVar{Name:\"RCLONE_CONFIG_NAS_SHELL_TYPE\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02934ec20)}, core.EnvVar{Name:\"RCLONE_CONFIG_NAS_USER\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02934ec40)}, core.EnvVar{Name:\"RCLONE_CONFIG_NAS_HOST\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02934ec60)}, core.EnvVar{Name:\"RCLONE_CONFIG_NAS_PORT\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02934ec80)}, core.EnvVar{Name:\"RCLONE_CONFIG_NAS_MD5SUM_COMMAND\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02934eca0)}, core.EnvVar{Name:\"RCLONE_CONFIG_NAS_PASS\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02934ecc0)}, core.EnvVar{Name:\"RCLONE_CONFIG_NAS_SHA1SUM_COMMAND\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02934ece0)}, core.EnvVar{Name:\"RCLONE_CONFIG_NAS_TYPE\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02934ed00)}, core.EnvVar{Name:\"PRIVILEGED_MOVER\", Value:\"0\", ValueFrom:(*core.EnvVarSource)(nil)}}, Resources:core.ResourceRequirements{Limits:core.ResourceList(nil), Requests:core.ResourceList(nil), Claims:[]core.ResourceClaim(nil)}, VolumeMounts:[]core.VolumeMount{core.VolumeMount{Name:\"data\", ReadOnly:false, MountPath:\"/data\", SubPath:\"\", MountPropagation:(*core.MountPropagationMode)(nil), SubPathExpr:\"\"}, core.VolumeMount{Name:\"cache\", ReadOnly:false, MountPath:\"/cache\", SubPath:\"\", MountPropagation:(*core.MountPropagationMode)(nil), SubPathExpr:\"\"}, core.VolumeMount{Name:\"tempdir\", ReadOnly:false, MountPath:\"/tmp\", SubPath:\"\", MountPropagation:(*core.MountPropagationMode)(nil), SubPathExpr:\"\"}}, VolumeDevices:[]core.VolumeDevice(nil), LivenessProbe:(*core.Probe)(nil), ReadinessProbe:(*core.Probe)(nil), StartupProbe:(*core.Probe)(nil), Lifecycle:(*core.Lifecycle)(nil), TerminationMessagePath:\"/dev/termination-log\", TerminationMessagePolicy:\"File\", ImagePullPolicy:\"IfNotPresent\", SecurityContext:(*core.SecurityContext)(0xc0334973e0), Stdin:false, StdinOnce:false, TTY:false}}, EphemeralContainers:[]core.EphemeralContainer(nil), RestartPolicy:\"Never\", TerminationGracePeriodSeconds:(*int64)(0xc036845990), ActiveDeadlineSeconds:(*int64)(nil), DNSPolicy:\"ClusterFirst\", NodeSelector:map[string]string(nil), ServiceAccountName:\"volsync-src-zwave-js-ui\", AutomountServiceAccountToken:(*bool)(nil), NodeName:\"\", SecurityContext:(*core.PodSecurityContext)(0xc02b4fc990), ImagePullSecrets:[]core.LocalObjectReference(nil), Hostname:\"\", Subdomain:\"\", SetHostnameAsFQDN:(*bool)(nil), Affinity:(*core.Affinity)(nil), SchedulerName:\"default-scheduler\", Tolerations:[]core.Toleration(nil), HostAliases:[]core.HostAlias(nil), PriorityClassName:\"\", Priority:(*int32)(nil), PreemptionPolicy:(*core.PreemptionPolicy)(nil), DNSConfig:(*core.PodDNSConfig)(nil), ReadinessGates:[]core.PodReadinessGate(nil), RuntimeClassName:(*string)(nil), Overhead:core.ResourceList(nil), EnableServiceLinks:(*bool)(nil), TopologySpreadConstraints:[]core.TopologySpreadConstraint(nil), OS:(*core.PodOS)(nil), SchedulingGates:[]core.PodSchedulingGate(nil), ResourceClaims:[]core.PodResourceClaim(nil)}}: field is immutable"}
github.com/backube/volsync/controllers/utils.CreateOrUpdateDeleteOnImmutableErr
/workspace/controllers/utils/reconcile.go:57
github.com/backube/volsync/controllers/mover/restic.(*Mover).ensureJob
/workspace/controllers/mover/restic/mover.go:280
github.com/backube/volsync/controllers/mover/restic.(*Mover).Synchronize
/workspace/controllers/mover/restic/mover.go:138
github.com/backube/volsync/controllers.(*rsMachine).Synchronize
/workspace/controllers/replicationsource_controller.go:264
github.com/backube/volsync/controllers/statemachine.doSynchronizingState
/workspace/controllers/statemachine/machine.go:102
github.com/backube/volsync/controllers/statemachine.Run
/workspace/controllers/statemachine/machine.go:70
github.com/backube/volsync/controllers.(*ReplicationSourceReconciler).Reconcile
/workspace/controllers/replicationsource_controller.go:138
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-28T21:00:05.758Z ERROR controllers.ReplicationSource reconcile failed {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "job": "hass/volsync-src-zwave-js-ui", "error": "unable to update object. Deleting object so it can be recreated"}
github.com/backube/volsync/controllers/mover/restic.(*Mover).ensureJob
/workspace/controllers/mover/restic/mover.go:519
github.com/backube/volsync/controllers/mover/restic.(*Mover).Synchronize
/workspace/controllers/mover/restic/mover.go:138
github.com/backube/volsync/controllers.(*rsMachine).Synchronize
/workspace/controllers/replicationsource_controller.go:264
github.com/backube/volsync/controllers/statemachine.doSynchronizingState
/workspace/controllers/statemachine/machine.go:102
github.com/backube/volsync/controllers/statemachine.Run
/workspace/controllers/statemachine/machine.go:70
github.com/backube/volsync/controllers.(*ReplicationSourceReconciler).Reconcile
/workspace/controllers/replicationsource_controller.go:138
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-28T21:00:05.793Z INFO controllers.ReplicationSource Namespace does not indicate volsync privileged movers are allowed, will assume privileged movers are not allowed. Annotation missing or not equal to 'true' {"replicationsource": "media/jellyfin", "namespace": "media", "Annotation": "volsync.backube/privileged-movers"}
2023-04-28T21:00:05.793Z DEBUG controllers.ReplicationSource Initiate Rclone Spec validation {"replicationsource": "media/jellyfin", "method": "Rclone"}
2023-04-28T21:00:05.793Z DEBUG controllers.ReplicationSource Rclone Spec validation complete. {"replicationsource": "media/jellyfin", "method": "Rclone"}
2023-04-28T21:00:05.793Z INFO controllers.ReplicationSource RcloneConfig reconciled {"replicationsource": "media/jellyfin", "method": "Rclone"}
2023-04-28T21:00:05.793Z ERROR Reconciler error {"controller": "replicationsource", "controllerGroup": "volsync.backube", "controllerKind": "ReplicationSource", "ReplicationSource": {"name":"zwave-js-ui","namespace":"hass"}, "namespace": "hass", "name": "zwave-js-ui", "reconcileID": "48452cdc-4641-41fe-b946-a5e8a1ee6eaa", "error": "unable to update object. Deleting object so it can be recreated"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-28T21:00:05.793Z INFO controllers.ReplicationSource Namespace does not indicate volsync privileged movers are allowed, will assume privileged movers are not allowed. Annotation missing or not equal to 'true' {"replicationsource": "hass/zwave-js-ui", "namespace": "hass", "Annotation": "volsync.backube/privileged-movers"}
2023-04-28T21:00:05.793Z DEBUG controllers.ReplicationSource waiting for snapshot to be ready {"replicationsource": "media/jellyfin", "method": "Rclone", "snapshot": "media/volsync-jellyfin-src"}
2023-04-28T21:00:05.793Z DEBUG controllers.ReplicationSource temporary snapshot reconciled {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "snapshot": "hass/volsync-zwave-js-ui-src", "operation": "unchanged"}
2023-04-28T21:00:05.794Z DEBUG controllers.ReplicationSource pvc from snap reconciled {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "pvc": "hass/volsync-zwave-js-ui-src", "operation": "unchanged"}
2023-04-28T21:00:05.794Z INFO controllers.ReplicationSource allocating cache volume {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "PVC": "volsync-zwave-js-ui-cache"}
2023-04-28T21:00:05.794Z DEBUG controllers.ReplicationSource PVC reconciled {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "PVC": "volsync-zwave-js-ui-cache", "operation": "unchanged"}
2023-04-28T21:00:05.794Z DEBUG controllers.ReplicationSource ServiceAccount reconciled {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "ServiceAccount": "hass/volsync-src-zwave-js-ui", "operation": "unchanged"}
2023-04-28T21:00:05.794Z DEBUG controllers.ReplicationSource Role reconciled {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "Role": "hass/volsync-src-zwave-js-ui", "operation": "unchanged"}
2023-04-28T21:00:05.794Z DEBUG controllers.ReplicationSource RoleBinding reconciled {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "RoleBinding": "hass/volsync-src-zwave-js-ui", "operation": "unchanged"}
2023-04-28T21:00:05.794Z INFO controllers.ReplicationSource job actions {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "job": "hass/volsync-src-zwave-js-ui", "actions": ["backup"]}
2023-04-28T21:00:05.922Z INFO controllers.ReplicationSource Namespace does not indicate volsync privileged movers are allowed, will assume privileged movers are not allowed. Annotation missing or not equal to 'true' {"replicationsource": "hass/zwave-js-ui", "namespace": "hass", "Annotation": "volsync.backube/privileged-movers"}
2023-04-28T21:00:05.922Z DEBUG controllers.ReplicationSource temporary snapshot reconciled {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "snapshot": "hass/volsync-zwave-js-ui-src", "operation": "unchanged"}
2023-04-28T21:00:05.922Z DEBUG controllers.ReplicationSource pvc from snap reconciled {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "pvc": "hass/volsync-zwave-js-ui-src", "operation": "unchanged"}
2023-04-28T21:00:05.922Z INFO controllers.ReplicationSource allocating cache volume {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "PVC": "volsync-zwave-js-ui-cache"}
2023-04-28T21:00:05.923Z DEBUG controllers.ReplicationSource PVC reconciled {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "PVC": "volsync-zwave-js-ui-cache", "operation": "unchanged"}
2023-04-28T21:00:05.923Z DEBUG controllers.ReplicationSource ServiceAccount reconciled {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "ServiceAccount": "hass/volsync-src-zwave-js-ui", "operation": "unchanged"}
2023-04-28T21:00:05.923Z DEBUG controllers.ReplicationSource Role reconciled {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "Role": "hass/volsync-src-zwave-js-ui", "operation": "unchanged"}
2023-04-28T21:00:05.923Z DEBUG controllers.ReplicationSource RoleBinding reconciled {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "RoleBinding": "hass/volsync-src-zwave-js-ui", "operation": "unchanged"}
2023-04-28T21:00:05.923Z INFO controllers.ReplicationSource job actions {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "job": "hass/volsync-src-zwave-js-ui", "actions": ["backup"]}
2023-04-28T21:00:05.964Z ERROR controllers.ReplicationSource reconcile failed {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "job": "hass/volsync-src-zwave-js-ui", "error": "Operation cannot be fulfilled on jobs.batch \"volsync-src-zwave-js-ui\": the object has been modified; please apply your changes to the latest version and try again"}
github.com/backube/volsync/controllers/mover/restic.(*Mover).ensureJob
/workspace/controllers/mover/restic/mover.go:519
github.com/backube/volsync/controllers/mover/restic.(*Mover).Synchronize
/workspace/controllers/mover/restic/mover.go:138
github.com/backube/volsync/controllers.(*rsMachine).Synchronize
/workspace/controllers/replicationsource_controller.go:264
github.com/backube/volsync/controllers/statemachine.doSynchronizingState
/workspace/controllers/statemachine/machine.go:102
github.com/backube/volsync/controllers/statemachine.Run
/workspace/controllers/statemachine/machine.go:70
github.com/backube/volsync/controllers.(*ReplicationSourceReconciler).Reconcile
/workspace/controllers/replicationsource_controller.go:138
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-28T21:00:05.995Z ERROR Reconciler error {"controller": "replicationsource", "controllerGroup": "volsync.backube", "controllerKind": "ReplicationSource", "ReplicationSource": {"name":"zwave-js-ui","namespace":"hass"}, "namespace": "hass", "name": "zwave-js-ui", "reconcileID": "fa38c547-e541-4ec4-8143-7dc012d7c719", "error": "Operation cannot be fulfilled on jobs.batch \"volsync-src-zwave-js-ui\": the object has been modified; please apply your changes to the latest version and try again"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-28T21:00:05.995Z INFO controllers.ReplicationSource Namespace does not indicate volsync privileged movers are allowed, will assume privileged movers are not allowed. Annotation missing or not equal to 'true' {"replicationsource": "hass/zwave-js-ui", "namespace": "hass", "Annotation": "volsync.backube/privileged-movers"}
2023-04-28T21:00:05.995Z DEBUG controllers.ReplicationSource temporary snapshot reconciled {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "snapshot": "hass/volsync-zwave-js-ui-src", "operation": "unchanged"}
2023-04-28T21:00:05.996Z DEBUG controllers.ReplicationSource pvc from snap reconciled {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "pvc": "hass/volsync-zwave-js-ui-src", "operation": "unchanged"}
2023-04-28T21:00:05.996Z INFO controllers.ReplicationSource allocating cache volume {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "PVC": "volsync-zwave-js-ui-cache"}
2023-04-28T21:00:05.997Z DEBUG controllers.ReplicationSource PVC reconciled {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "PVC": "volsync-zwave-js-ui-cache", "operation": "unchanged"}
2023-04-28T21:00:05.997Z DEBUG controllers.ReplicationSource ServiceAccount reconciled {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "ServiceAccount": "hass/volsync-src-zwave-js-ui", "operation": "unchanged"}
2023-04-28T21:00:05.998Z DEBUG controllers.ReplicationSource Role reconciled {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "Role": "hass/volsync-src-zwave-js-ui", "operation": "unchanged"}
2023-04-28T21:00:05.998Z DEBUG controllers.ReplicationSource RoleBinding reconciled {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "RoleBinding": "hass/volsync-src-zwave-js-ui", "operation": "unchanged"}
2023-04-28T21:00:05.998Z INFO controllers.ReplicationSource job actions {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "job": "hass/volsync-src-zwave-js-ui", "actions": ["backup"]}
2023-04-28T21:00:06.013Z ERROR controllers.ReplicationSource Immutable error updating the object. Will delete so it can be recreated {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "job": "hass/volsync-src-zwave-js-ui", "error": "Job.batch \"volsync-src-zwave-js-ui\" is invalid: spec.template: Invalid value: core.PodTemplateSpec{ObjectMeta:v1.ObjectMeta{Name:\"volsync-src-zwave-js-ui\", GenerateName:\"\", Namespace:\"\", SelfLink:\"\", UID:\"\", ResourceVersion:\"\", Generation:0, CreationTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeletionTimestamp:<nil>, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{\"app.kubernetes.io/created-by\":\"volsync\", \"controller-uid\":\"f8dd9358-58c6-4ee5-8457-b4b5883dcd84\", \"job-name\":\"volsync-src-zwave-js-ui\"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:core.PodSpec{Volumes:[]core.Volume{core.Volume{Name:\"data\", VolumeSource:core.VolumeSource{HostPath:(*core.HostPathVolumeSource)(nil), EmptyDir:(*core.EmptyDirVolumeSource)(nil), GCEPersistentDisk:(*core.GCEPersistentDiskVolumeSource)(nil), AWSElasticBlockStore:(*core.AWSElasticBlockStoreVolumeSource)(nil), GitRepo:(*core.GitRepoVolumeSource)(nil), Secret:(*core.SecretVolumeSource)(nil), NFS:(*core.NFSVolumeSource)(nil), ISCSI:(*core.ISCSIVolumeSource)(nil), Glusterfs:(*core.GlusterfsVolumeSource)(nil), PersistentVolumeClaim:(*core.PersistentVolumeClaimVolumeSource)(0xc0312492c0), RBD:(*core.RBDVolumeSource)(nil), Quobyte:(*core.QuobyteVolumeSource)(nil), FlexVolume:(*core.FlexVolumeSource)(nil), Cinder:(*core.CinderVolumeSource)(nil), CephFS:(*core.CephFSVolumeSource)(nil), Flocker:(*core.FlockerVolumeSource)(nil), DownwardAPI:(*core.DownwardAPIVolumeSource)(nil), FC:(*core.FCVolumeSource)(nil), AzureFile:(*core.AzureFileVolumeSource)(nil), ConfigMap:(*core.ConfigMapVolumeSource)(nil), VsphereVolume:(*core.VsphereVirtualDiskVolumeSource)(nil), AzureDisk:(*core.AzureDiskVolumeSource)(nil), PhotonPersistentDisk:(*core.PhotonPersistentDiskVolumeSource)(nil), Projected:(*core.ProjectedVolumeSource)(nil), PortworxVolume:(*core.PortworxVolumeSource)(nil), ScaleIO:(*core.ScaleIOVolumeSource)(nil), StorageOS:(*core.StorageOSVolumeSource)(nil), CSI:(*core.CSIVolumeSource)(nil), Ephemeral:(*core.EphemeralVolumeSource)(nil)}}, core.Volume{Name:\"cache\", VolumeSource:core.VolumeSource{HostPath:(*core.HostPathVolumeSource)(nil), EmptyDir:(*core.EmptyDirVolumeSource)(nil), GCEPersistentDisk:(*core.GCEPersistentDiskVolumeSource)(nil), AWSElasticBlockStore:(*core.AWSElasticBlockStoreVolumeSource)(nil), GitRepo:(*core.GitRepoVolumeSource)(nil), Secret:(*core.SecretVolumeSource)(nil), NFS:(*core.NFSVolumeSource)(nil), ISCSI:(*core.ISCSIVolumeSource)(nil), Glusterfs:(*core.GlusterfsVolumeSource)(nil), PersistentVolumeClaim:(*core.PersistentVolumeClaimVolumeSource)(0xc0312492d8), RBD:(*core.RBDVolumeSource)(nil), Quobyte:(*core.QuobyteVolumeSource)(nil), FlexVolume:(*core.FlexVolumeSource)(nil), Cinder:(*core.CinderVolumeSource)(nil), CephFS:(*core.CephFSVolumeSource)(nil), Flocker:(*core.FlockerVolumeSource)(nil), DownwardAPI:(*core.DownwardAPIVolumeSource)(nil), FC:(*core.FCVolumeSource)(nil), AzureFile:(*core.AzureFileVolumeSource)(nil), ConfigMap:(*core.ConfigMapVolumeSource)(nil), VsphereVolume:(*core.VsphereVirtualDiskVolumeSource)(nil), AzureDisk:(*core.AzureDiskVolumeSource)(nil), PhotonPersistentDisk:(*core.PhotonPersistentDiskVolumeSource)(nil), Projected:(*core.ProjectedVolumeSource)(nil), PortworxVolume:(*core.PortworxVolumeSource)(nil), ScaleIO:(*core.ScaleIOVolumeSource)(nil), StorageOS:(*core.StorageOSVolumeSource)(nil), CSI:(*core.CSIVolumeSource)(nil), Ephemeral:(*core.EphemeralVolumeSource)(nil)}}, core.Volume{Name:\"tempdir\", VolumeSource:core.VolumeSource{HostPath:(*core.HostPathVolumeSource)(nil), EmptyDir:(*core.EmptyDirVolumeSource)(0xc0312492f0), GCEPersistentDisk:(*core.GCEPersistentDiskVolumeSource)(nil), AWSElasticBlockStore:(*core.AWSElasticBlockStoreVolumeSource)(nil), GitRepo:(*core.GitRepoVolumeSource)(nil), Secret:(*core.SecretVolumeSource)(nil), NFS:(*core.NFSVolumeSource)(nil), ISCSI:(*core.ISCSIVolumeSource)(nil), Glusterfs:(*core.GlusterfsVolumeSource)(nil), PersistentVolumeClaim:(*core.PersistentVolumeClaimVolumeSource)(nil), RBD:(*core.RBDVolumeSource)(nil), Quobyte:(*core.QuobyteVolumeSource)(nil), FlexVolume:(*core.FlexVolumeSource)(nil), Cinder:(*core.CinderVolumeSource)(nil), CephFS:(*core.CephFSVolumeSource)(nil), Flocker:(*core.FlockerVolumeSource)(nil), DownwardAPI:(*core.DownwardAPIVolumeSource)(nil), FC:(*core.FCVolumeSource)(nil), AzureFile:(*core.AzureFileVolumeSource)(nil), ConfigMap:(*core.ConfigMapVolumeSource)(nil), VsphereVolume:(*core.VsphereVirtualDiskVolumeSource)(nil), AzureDisk:(*core.AzureDiskVolumeSource)(nil), PhotonPersistentDisk:(*core.PhotonPersistentDiskVolumeSource)(nil), Projected:(*core.ProjectedVolumeSource)(nil), PortworxVolume:(*core.PortworxVolumeSource)(nil), ScaleIO:(*core.ScaleIOVolumeSource)(nil), StorageOS:(*core.StorageOSVolumeSource)(nil), CSI:(*core.CSIVolumeSource)(nil), Ephemeral:(*core.EphemeralVolumeSource)(nil)}}}, InitContainers:[]core.Container(nil), Containers:[]core.Container{core.Container{Name:\"restic\", Image:\"quay.io/backube/volsync:0.7.1\", Command:[]string{\"/mover-restic/entry.sh\"}, Args:[]string{\"backup\"}, WorkingDir:\"\", Ports:[]core.ContainerPort(nil), EnvFrom:[]core.EnvFromSource(nil), Env:[]core.EnvVar{core.EnvVar{Name:\"FORGET_OPTIONS\", Value:\" --keep-daily 10 --keep-weekly 6 --keep-monthly 4 --keep-within 3d\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"DATA_DIR\", Value:\"/data\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"RESTIC_CACHE_DIR\", Value:\"/cache\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"RESTORE_AS_OF\", Value:\"\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"SELECT_PREVIOUS\", Value:\"0\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"RESTIC_REPOSITORY\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02ad53fc0)}, core.EnvVar{Name:\"RESTIC_PASSWORD\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02ad53fe0)}, core.EnvVar{Name:\"RESTIC_COMPRESSION\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02c76c000)}, core.EnvVar{Name:\"RESTIC_PACK_SIZE\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02c76c020)}, core.EnvVar{Name:\"RESTIC_READ_CONCURRENCY\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02c76c040)}, core.EnvVar{Name:\"AWS_ACCESS_KEY_ID\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02c76c060)}, core.EnvVar{Name:\"AWS_SECRET_ACCESS_KEY\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02c76c080)}, core.EnvVar{Name:\"AWS_SESSION_TOKEN\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02c76c0a0)}, core.EnvVar{Name:\"AWS_DEFAULT_REGION\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02c76c0c0)}, core.EnvVar{Name:\"ST_AUTH\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02c76c0e0)}, core.EnvVar{Name:\"ST_USER\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02c76c100)}, core.EnvVar{Name:\"ST_KEY\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02c76c120)}, core.EnvVar{Name:\"OS_AUTH_URL\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02c76c140)}, core.EnvVar{Name:\"OS_REGION_NAME\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02c76c160)}, core.EnvVar{Name:\"OS_USERNAME\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02c76c180)}, core.EnvVar{Name:\"OS_USER_ID\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02c76c1a0)}, core.EnvVar{Name:\"OS_PASSWORD\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02c76c1c0)}, core.EnvVar{Name:\"OS_TENANT_ID\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02c76c1e0)}, core.EnvVar{Name:\"OS_TENANT_NAME\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02c76c200)}, core.EnvVar{Name:\"OS_USER_DOMAIN_NAME\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02c76c220)}, core.EnvVar{Name:\"OS_USER_DOMAIN_ID\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02c76c240)}, core.EnvVar{Name:\"OS_PROJECT_NAME\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02c76c260)}, core.EnvVar{Name:\"OS_PROJECT_DOMAIN_NAME\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02c76c280)}, core.EnvVar{Name:\"OS_PROJECT_DOMAIN_ID\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02c76c2a0)}, core.EnvVar{Name:\"OS_TRUST_ID\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02c76c2c0)}, core.EnvVar{Name:\"OS_APPLICATION_CREDENTIAL_ID\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02c76c2e0)}, core.EnvVar{Name:\"OS_APPLICATION_CREDENTIAL_NAME\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02c76c300)}, core.EnvVar{Name:\"OS_APPLICATION_CREDENTIAL_SECRET\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02c76c320)}, core.EnvVar{Name:\"OS_STORAGE_URL\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02c76c340)}, core.EnvVar{Name:\"OS_AUTH_TOKEN\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02c76c360)}, core.EnvVar{Name:\"B2_ACCOUNT_ID\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02c76c380)}, core.EnvVar{Name:\"B2_ACCOUNT_KEY\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02c76c3a0)}, core.EnvVar{Name:\"AZURE_ACCOUNT_NAME\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02c76c3c0)}, core.EnvVar{Name:\"AZURE_ACCOUNT_KEY\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02c76c3e0)}, core.EnvVar{Name:\"AZURE_ACCOUNT_SAS\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02c76c400)}, core.EnvVar{Name:\"GOOGLE_PROJECT_ID\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02c76c420)}, core.EnvVar{Name:\"RCLONE_CONFIG_NAS_PORT\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02c76c440)}, core.EnvVar{Name:\"RCLONE_CONFIG_NAS_SHELL_TYPE\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02c76c460)}, core.EnvVar{Name:\"RCLONE_CONFIG_NAS_USER\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02c76c480)}, core.EnvVar{Name:\"RCLONE_CONFIG_NAS_HOST\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02c76c4a0)}, core.EnvVar{Name:\"RCLONE_CONFIG_NAS_MD5SUM_COMMAND\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02c76c4c0)}, core.EnvVar{Name:\"RCLONE_CONFIG_NAS_TYPE\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02c76c4e0)}, core.EnvVar{Name:\"RCLONE_CONFIG_NAS_PASS\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02c76c500)}, core.EnvVar{Name:\"RCLONE_CONFIG_NAS_SHA1SUM_COMMAND\", Value:\"\", ValueFrom:(*core.EnvVarSource)(0xc02c76c520)}, core.EnvVar{Name:\"PRIVILEGED_MOVER\", Value:\"0\", ValueFrom:(*core.EnvVarSource)(nil)}}, Resources:core.ResourceRequirements{Limits:core.ResourceList(nil), Requests:core.ResourceList(nil), Claims:[]core.ResourceClaim(nil)}, VolumeMounts:[]core.VolumeMount{core.VolumeMount{Name:\"data\", ReadOnly:false, MountPath:\"/data\", SubPath:\"\", MountPropagation:(*core.MountPropagationMode)(nil), SubPathExpr:\"\"}, core.VolumeMount{Name:\"cache\", ReadOnly:false, MountPath:\"/cache\", SubPath:\"\", MountPropagation:(*core.MountPropagationMode)(nil), SubPathExpr:\"\"}, core.VolumeMount{Name:\"tempdir\", ReadOnly:false, MountPath:\"/tmp\", SubPath:\"\", MountPropagation:(*core.MountPropagationMode)(nil), SubPathExpr:\"\"}}, VolumeDevices:[]core.VolumeDevice(nil), LivenessProbe:(*core.Probe)(nil), ReadinessProbe:(*core.Probe)(nil), StartupProbe:(*core.Probe)(nil), Lifecycle:(*core.Lifecycle)(nil), TerminationMessagePath:\"/dev/termination-log\", TerminationMessagePolicy:\"File\", ImagePullPolicy:\"IfNotPresent\", SecurityContext:(*core.SecurityContext)(0xc0493ec3c0), Stdin:false, StdinOnce:false, TTY:false}}, EphemeralContainers:[]core.EphemeralContainer(nil), RestartPolicy:\"Never\", TerminationGracePeriodSeconds:(*int64)(0xc02fb8c3f0), ActiveDeadlineSeconds:(*int64)(nil), DNSPolicy:\"ClusterFirst\", NodeSelector:map[string]string(nil), ServiceAccountName:\"volsync-src-zwave-js-ui\", AutomountServiceAccountToken:(*bool)(nil), NodeName:\"\", SecurityContext:(*core.PodSecurityContext)(0xc02fc67cb0), ImagePullSecrets:[]core.LocalObjectReference(nil), Hostname:\"\", Subdomain:\"\", SetHostnameAsFQDN:(*bool)(nil), Affinity:(*core.Affinity)(nil), SchedulerName:\"default-scheduler\", Tolerations:[]core.Toleration(nil), HostAliases:[]core.HostAlias(nil), PriorityClassName:\"\", Priority:(*int32)(nil), PreemptionPolicy:(*core.PreemptionPolicy)(nil), DNSConfig:(*core.PodDNSConfig)(nil), ReadinessGates:[]core.PodReadinessGate(nil), RuntimeClassName:(*string)(nil), Overhead:core.ResourceList(nil), EnableServiceLinks:(*bool)(nil), TopologySpreadConstraints:[]core.TopologySpreadConstraint(nil), OS:(*core.PodOS)(nil), SchedulingGates:[]core.PodSchedulingGate(nil), ResourceClaims:[]core.PodResourceClaim(nil)}}: field is immutable"}
github.com/backube/volsync/controllers/utils.CreateOrUpdateDeleteOnImmutableErr
/workspace/controllers/utils/reconcile.go:57
github.com/backube/volsync/controllers/mover/restic.(*Mover).ensureJob
/workspace/controllers/mover/restic/mover.go:280
github.com/backube/volsync/controllers/mover/restic.(*Mover).Synchronize
/workspace/controllers/mover/restic/mover.go:138
github.com/backube/volsync/controllers.(*rsMachine).Synchronize
/workspace/controllers/replicationsource_controller.go:264
github.com/backube/volsync/controllers/statemachine.doSynchronizingState
/workspace/controllers/statemachine/machine.go:102
github.com/backube/volsync/controllers/statemachine.Run
/workspace/controllers/statemachine/machine.go:70
github.com/backube/volsync/controllers.(*ReplicationSourceReconciler).Reconcile
/workspace/controllers/replicationsource_controller.go:138
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-28T21:00:06.032Z ERROR controllers.ReplicationSource reconcile failed {"replicationsource": "hass/zwave-js-ui", "method": "Restic", "job": "hass/volsync-src-zwave-js-ui", "error": "unable to update object. Deleting object so it can be recreated"}
github.com/backube/volsync/controllers/mover/restic.(*Mover).ensureJob
/workspace/controllers/mover/restic/mover.go:519
github.com/backube/volsync/controllers/mover/restic.(*Mover).Synchronize
/workspace/controllers/mover/restic/mover.go:138
github.com/backube/volsync/controllers.(*rsMachine).Synchronize
/workspace/controllers/replicationsource_controller.go:264
github.com/backube/volsync/controllers/statemachine.doSynchronizingState
/workspace/controllers/statemachine/machine.go:102
github.com/backube/volsync/controllers/statemachine.Run
/workspace/controllers/statemachine/machine.go:70
github.com/backube/volsync/controllers.(*ReplicationSourceReconciler).Reconcile
/workspace/controllers/replicationsource_controller.go:138
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
This is all that was on the screen when I stopped it. Did it catch the important part?... Seemed like the same error repeating.
I noticed it sat for like 2 minutes with no job.
Status:
Conditions:
Last Transition Time: 2023-04-28T20:59:33Z
Message: Timeout: request did not complete within requested timeout - context deadline exceeded
Reason: Error
Status: False
Type: Synchronizing
Last Sync Start Time: 2023-04-28T20:58:14Z
Latest Mover Status:
Restic:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal VolumeSnapshotCreated 77s volsync-controller created VolumeSnapshot/volsync-zwave-js-ui-src from PersistentVolumeClaim/zwave-pvc
Normal PersistentVolumeClaimCreated 51s volsync-controller created PersistentVolumeClaim/volsync-zwave-js-ui-src from VolumeSnapshot/volsync-zwave-js-ui-src
Normal PersistentVolumeClaimCreated 47s volsync-controller created PersistentVolumeClaim/volsync-zwave-js-ui-cache to receive incoming data
Then
Status:
Conditions:
Last Transition Time: 2023-04-28T20:59:53Z
Message: unable to update object. Deleting object so it can be recreated
Reason: Error
Status: False
Type: Synchronizing
Last Sync Start Time: 2023-04-28T20:58:14Z
Latest Mover Status:
Restic:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal VolumeSnapshotCreated 89s volsync-controller created VolumeSnapshot/volsync-zwave-js-ui-src from PersistentVolumeClaim/zwave-pvc
Normal PersistentVolumeClaimCreated 63s volsync-controller created PersistentVolumeClaim/volsync-zwave-js-ui-src from VolumeSnapshot/volsync-zwave-js-ui-src
Normal PersistentVolumeClaimCreated 59s volsync-controller created PersistentVolumeClaim/volsync-zwave-js-ui-cache to receive incoming data
And at this point the job IS created. Seconds later I have a wall of errors.
I think there's another mismatch that I'm not thinking of at the moment - I might need to try a test myself next week to be sure - one thing I should've asked before giving you the image - what version of volsync are you using? This is based off main so would be equivalent to quay.io/backube/volsync:latest (+ the change in my branch)
I was using 0.7.1, which should be latest.
https://github.com/Rojikku/home-ops/blob/main/cluster/apps/volsync/volsync/app/helmrelease.yaml
There's my deploy. All my other rclone based syncs seem to be having no issues at all, it's only this restic one that's having issues. It's...possible I'm doing something wrong? But I wouldn't expect this sort of error from an issue on my end. Also, the backups appear to be happening, so I feel like it shouldn't be.
I think the issue is that the job appears to keep the env var ordering, but my code to get the rclone keys dynamically from the secret is getting keys from the map - which will not be ordered consistently - I've pushed a new image you can try @Rojikku. Keep in mind it is based on the latest in main which has changed from 0.7.x, but I think for this test it should still work.
@tesshuflower That seems to have done the trick. I admit, very limited testing I have done.
Status:
Conditions:
Last Transition Time: 2023-05-02T00:50:02Z
Message: Synchronization in-progress
Reason: SyncInProgress
Status: True
Type: Synchronizing
Last Sync Duration: 18.070150054s
Last Sync Start Time: 2023-05-02T00:50:00Z
Last Sync Time: 2023-05-02T00:48:32Z
Latest Mover Status:
Logs: using parent snapshot c0ec2bfc
Added to the repository: 1.435 MiB (374.648 KiB stored)
processed 2249 files, 6.956 MiB in 0:01
snapshot 1e150ae8 saved
Restic completed in 4s
Result: Successful
Next Sync Time: 2023-05-02T00:50:00Z
Restic:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal PersistentVolumeClaimCreated 110s volsync-controller created PersistentVolumeClaim/volsync-zwave-js-ui-cache to receive incoming data
Normal VolumeSnapshotCreated 7s (x2 over 113s) volsync-controller created VolumeSnapshot/volsync-zwave-js-ui-src from PersistentVolumeClaim/zwave-pvc
Normal PersistentVolumeClaimCreated 5s (x2 over 110s) volsync-controller created PersistentVolumeClaim/volsync-zwave-js-ui-src from VolumeSnapshot/volsync-zwave-js-ui-src
Mover log:
Starting container
VolSync restic container version: v0.7.1+8af0bf0
backup
restic 0.15.1 compiled with go1.19.7 on linux/amd64
Testing mandatory env variables
== Checking directory for content ===
== Initialize Dir =======
ID Time Host Tags Paths
------------------------------------------------------------
c0ec2bfc 2023-04-28 21:02:08 volsync /data
------------------------------------------------------------
1 snapshots
=== Starting backup ===
/data /
rclone: 2023/05/02 00:48:25 NOTICE: Config file "/.rclone.conf" not found - using defaults
using parent snapshot c0ec2bfc
Files: 4 new, 2245 changed, 0 unmodified
Dirs: 0 new, 394 changed, 0 unmodified
Added to the repository: 1.435 MiB (374.648 KiB stored)
processed 2249 files, 6.956 MiB in 0:01
snapshot 1e150ae8 saved
/
=== Starting forget ===
rclone: 2023/05/02 00:48:27 NOTICE: Config file "/.rclone.conf" not found - using defaults
Applying Policy: keep 10 daily, 6 weekly, 4 monthly snapshots and all snapshots within 3d of the newest
keep 2 snapshots:
ID Time Host Tags Reasons Paths
------------------------------------------------------------------------------
c0ec2bfc 2023-04-28 21:02:08 volsync daily snapshot /data
weekly snapshot
monthly snapshot
1e150ae8 2023-05-02 00:48:25 volsync within 3d /data
daily snapshot
weekly snapshot
monthly snapshot
------------------------------------------------------------------------------
2 snapshots
Restic completed in 4s
=== Done ===
No random pods are being created anymore, the backup shows in restic. ls latest
shows a bunch of files that look correct. All seems very promising to me.
Unfortunately I had a lot going on today, just figured I'd run this simple test before I went to bed, and it looks perfect to me!
Tomorrow I will work on migrating everything to it, and I'll see if I produce any bugs. But, it's doubtful I will, since it'll be the same config with different data. I've been running your previous image this whole time with no real issues that I've noticed, did do one backup restore in that time too with no issues. I'll keep running this test image until there's an official release, most likely. I will let you know if any issues pop up, but so far so good.
Thank you for working on this for me! Really looks like it'll be a quality of life improvement to have several versions of my data like this!
Update: Everything is still running perfectly. The retain is working as expected, all my backups have worked as expected, and I haven't seen any further issues.
Update: Everything is still running perfectly. The retain is working as expected, all my backups have worked as expected, and I haven't seen any further issues.
Glad to hear it's working for you! Will do some cleanup/unit tests in my branch and try to get a PR up...
Implemented in https://github.com/backube/volsync/pull/751
Describe the feature you'd like to have. I would like to be able to pass environmental variables to the mover pod.
What is the value to the end user? (why is it a priority?) Restic allows you to use rclone as a backup method, but there's not really an rclone config option unless you're using rclone. Rclone allows you to do a full config via variables
How will we know we have a good solution? (acceptance criteria) Ability to pass arbitrarily named ENV variables to mover pod so that restic/rclone will recognize them.
Additional context Example I believe I configured that correctly, but I think the issue is that I took "Variables will correspond" from the docs too literally...and it doesn't just pass through everything.
Honestly, the more I look at this, the more I'm confused why it doesn't work. RESTIC_PASSWORD and RESTIC_REPOSITORY are both environment variables... Did you MANUALLY pass them through instead of passing through all portions of the secret as variables as a blanket? Pretty sure a blanket pass through would work fine for this use case. Honestly, either way. I'd be happy with anything as long as I can pass through my RCLONE config options.