search

gsemet / kube-airflow

Helm chart to run production Airflow/Celery on Kubernetes
20 stars 8 forks source link

Persistent volume claims would have to be unique for ReadWriteOnce #8

Open Sedward opened 6 years ago

Sedward commented 6 years ago

With a access policy of ReadWriteOnce, which is the only access policy supported by aws ebs volumes. This will mean that each pvc will need to be given a unique name, like "fullname"-scheduler, etc.

An alternative proposal would be to using a pvc that supports ReadWriteMany. Users on AWS could possible use the elastic file system available from the external storage project https://github.com/kubernetes-incubator/external-storage/tree/master/aws/efs

If this were the case, only one deployment would need to run the optional git-sync which may be desirable.

Personally, I think the second approach is more desireable

rolanddb commented 6 years ago

I agree that the second approach is more desireable. The contents of the DAGs directory should be the same on all Pods, so sharing a volume makes more sense in my opinion.

We are on Azure, and I'm planning to use Azure File Storage which is similar to AWS EFS.

gsemet commented 6 years ago

pull request welcome !

rolanddb commented 6 years ago

@gsemet Happy to help, but what would have to change?

The chart works as-is if you specify accessMode=ReadWriteMany for the PV. I expect it also to work with accessMode=ReadOnlyMany.

Would it be enough to update the documentation / comments in the chart to point out the supported accessModes?

gsemet commented 6 years ago

I like when pvc are unique so when they use fullname