[Docs] Document how to mount and dynamically create persistent volume

[SmritiSatyanV]

Description

Slack conversation

User_1: Hello flyte team. I have a question regarding using persistent volumes (PVs) and persistent volume claims (PVCs) with flyte tasks and workflows. More specifically, I’m interested in using an AWS efs volume in my eks cluster as a shared mount for ReadWriteMany. I have looked at [this example](https://docs.flyte.org/projects/cookbook/en/latest/auto/integrations/kubernetes/pod/pod.html) and see the V1Volume being passed into the pod spec. I’m looking through [these docs](https://github.com/kubernetes-client/python/blob/master/kubernetes/docs/V1Volume.md) and see that you can create a V1Volume from a persistent_volume_claim . So in theory, I can just create a PV and PVC in my cluster, include them in my pod spec, and attach the spec to a flyte task. But I noticed that PVCs are namespace specific and flyte uses the project-domain namespace for tasks/workflows that are executing. Two questions.
Are PVCs the right solution here and if so, how can I dynamically create PVCs for my project-domain s? Is this something flyte could be configured to do for us or would we  be responsible for ensuring any referenced PVCs and PVs exist.
What other options are available for mounting shared persistent volumes to my flyte tasks?

User_2: I have considered exactly the same problem in my team.
here are some solution I do.
you can create pvc per namespace using IaC platform like Terraform, Pulumi.
if you are using IaC , then you don't have to create each PVC per namespace manually.
or you should make shell script to build all PVC per namespace at once.
how about using other path in V1Volume? you can use NFS or hostpath to access some specific volume. these are also be provided in V1Volume (kubernetes client, indeed)

Yuvraj [Response]: You guys can use cluster_resource_manager , For creating PVC for each project-domain namespace like this https://github.com/flyteorg/flyte/blob/master/charts/flyte-core/values-eks.yaml#L360
[@Mike Zhong](https://flyte-org.slack.com/team/U03BXNGV65A)
 There are two ways to mount volume in your flyte task,
First is using [pod plugin](https://docs.flyte.org/projects/cookbook/en/latest/auto/integrations/kubernetes/pod/pod.html)
Second is propeller’s pod template  https://docs.flyte.org/en/latest/deployment/cluster_config/general.html#using-default-k8s-podtemplates

User_1: The cluster_resource_manager block is optional, but if we define namespace PVCs there, we would need to add them as new projects are created and redeploy to keep things updated. I was able to seamlessly get EFS working with flyte tasks so that was nice, just trying to figure out the best way to grant and manage access to the PV from other projects
other flyte projects (-p). Since the namespaces are {project}-{domain} and PVCs exist per namespace, any project that wants to use an efs backed PV would need the PVC created for its namespace. Please correct me if this is not correct

Ketan [Response]: That is true. But, if you are using EFS just create PVC in every namespace and then use podtemplates to mount them.
Now to provision them you can use ClusterResourceManager, or
you can use flytectl (gitops) style to create namesapces and manage resources in that namespaces through your regular CI/CD tooling. and you can have a centralized way to onboard onto Flyte - Creating a project has to go through a central way

Document :

1. How to mount shared persistent volumes to Flyte tasks
2. How to dynamically create Persistent Volume Claims (PVCs) for project-domains

[samhita-alla]

Hey Smriti! Can you paste the slack convo here, just in case the message vanishes when we hit the limit?

[hamersaw]

This is an interesting issue, I think there are two things to answer here: (1) How do we mount volumes in Flyte tasks? (2) How can we programmatically create persistent volumes and claims (PVCs) for new Flyte projects.

I think the former must be documented - as discussed in the google doc we can either use the pod plugin to define a VolumeMount in the PodSpec for the task or use the default PodTemplate so the volume is mounted in every task.

For the later, I'm wondering if it's something we need to address in the documentation. It seems to be more of a CI/CD issue - when a new project is created use k8s clusterresourcemanager to create a PVC. My concern is that if we get too specific with use-cases our docs will bloat and finding things will be extremely difficult

@samhita-alla @cosmicBboy thoughts?

[samhita-alla]

@hamersaw, I agree with you. (2) relies on the CI/CD setup. If you think documenting (2) isn't a good idea, let's skip it. But should we be guiding the user somehow without being specific? Do you have any ideas on how we can do that?

[hamersaw]

Unfortunately I don't have a great idea. In our recent OSS planning meeting there was some push towards github discussions? That way these things are searchable, it's basically what stackoverflow was designed for. I don't know if that is the correct direction either - it would be perfect to have some way that these issues don't bloat the docs but are still available if necessary.

[github-actions[bot]]

Hello 👋, This issue has been inactive for over 9 months. To help maintain a clean and focused backlog, we'll be marking this issue as stale and will close the issue if we detect no activity in the next 7 days. Thank you for your contribution and understanding! 🙏

[github-actions[bot]]

Hello 👋, This issue has been inactive for over 9 months and hasn't received any updates since it was marked as stale. We'll be closing this issue for now, but if you believe this issue is still relevant, please feel free to reopen it. Thank you for your contribution and understanding! 🙏

[github-actions[bot]]

Hello 👋, this issue has been inactive for over 9 months. To help maintain a clean and focused backlog, we'll be marking this issue as stale and will engage on it to decide if it is still applicable. Thank you for your contribution and understanding! 🙏