Open ball-hayden opened 2 years ago
This one had me deleting the airbyte server pod multiple times until it gets scheduled on the same node as the airbyte-cron pod (ReadWriteOnce is accessible by multiple pods if they are on the same node). It makes upgrading the airbyte instance complicated.
I can't recommend using ReadWriteMany beacause of so few providers supporting it and neither using node affinity constraints because it makes the cluster more rigid. If both airbyte-cron and server need write access to it we can:
Weirdly enough the Helm chart doesn't seem to have this airbyte-volume-configs
volume at all... https://github.com/airbytehq/airbyte/blob/master/charts/airbyte-cron/templates/deployment.yaml
What's this PVC used for anyway?
Weirdly enough the Helm chart doesn't seem to have this
airbyte-volume-configs
volume at all... https://github.com/airbytehq/airbyte/blob/master/charts/airbyte-cron/templates/deployment.yaml
But I see it here: https://github.com/airbytehq/airbyte/blob/master/kube/resources/volume-configs.yaml.
And we have the same problems now. If a restart happens we need to do a lot of workarounds to make it work. Any insights on why we need it at all?
Yep have that problem too
Normal Scheduled 45m default-scheduler Successfully assigned airbyte/airbyte-cron-765ccb6b44-9lb2j to gke-ase1-glo-data-1-e2-std-14-36-de-s-3e48975c-4xw8
Warning FailedAttachVolume 45m attachdetach-controller Multi-Attach error for volume "pvc-cf321101-735f-47e1-ae5a-607a8c26fa1a" Volume is already used by pod(s) airbyte-server-8c464cd5f-tbc96
Warning FailedMount 16m (x3 over 40m) kubelet Unable to attach or mount volumes: unmounted volumes=[airbyte-volume-configs], unattached volumes=[gcs-log-creds-volume kube-api-access-lkqfk airbyte-volume-configs]: timed out waiting for the condition
Warning FailedMount 13m (x3 over 34m) kubelet Unable to attach or mount volumes: unmounted volumes=[airbyte-volume-configs], unattached volumes=[kube-api-access-lkqfk airbyte-volume-configs gcs-log-creds-volume]: timed out waiting for the condition
Warning FailedMount 20s (x14 over 43m) kubelet Unable to attach or mount volumes: unmounted volumes=[airbyte-volume-configs], unattached volumes=[airbyte-volume-configs gcs-log-creds-volume kube-api-access-lkqfk]: timed out waiting for the condition
Currently i delete that volumt mount
Is it related CSI Driver (GKE) here https://airbyte.gitbook.io/airbyte/deploying-airbyte/on-kubernetes#persistent-storage-on-gke-regional-cluster
Hey, we got the same issue with kustomize deployment. Any ideas? Maybe Deployment should be changed to StatefulSet as it was proposed here https://github.com/airbytehq/airbyte/issues/9133#issuecomment-1127667156?
Is there any update to this?
Facing same issue here. So, I ended up commenting out the whole volumeMounts
and volume
section in cron.yaml. So now airbyte-cron doesn't go and fight with airbyte-server for the volume. The volume is likely not useful anyway.
# volumeMounts:
# - name: airbyte-volume-configs
# mountPath: /configs
# - name: gcs-log-creds-volume
# mountPath: /secrets/gcs-log-creds
# readOnly: true
# volumes:
# - name: airbyte-volume-configs
# persistentVolumeClaim:
# claimName: airbyte-volume-configs
# - name: gcs-log-creds-volume
# secret:
# secretName: gcs-log-creds
Environment
Current Behavior
airbyte-cron
deployment pods fail to start, becauseairbyte-volume-configs
is already mounted byairbyte-server
podExpected Behavior
A
ReadWriteOnce
PVC should not be used by multiple pods.Logs
Steps to Reproduce
Are you willing to submit a PR?
I'd need guidance as to how the team would like to handle this.
One option would be to use a
ReadWriteMany
, although this doesn't seem to be supported by many providers. Another option might be to haveairbyte-cron
contactairbyte-server
to fetch config?