Create minio-local-config script for drenv

[matancarmeli7]

I tried to use the Minio script in the drenv and I got an error in the setup job that it uses minio-config-pvc PVC which is being used by another pod and the storage provider that I used for creating the PVCs only supports RWO so I couldn't drenv to create a minio in the DR clusters. In order to solve it I changed the minio.yaml so that it won't create minio-config-pvc PVC and both the deployment and the job will run on the same host using node selector and the I used hostPath to provide persistent storage for config directory. It can be a good quick solution for testing environments.

[nirs]

@matancarmeli7 If I understand correctly, the issue is that the config pvc has access mode ReadWriteOnce but it is accessed by both the minio container and the setup job at the same time.

Maybe minikube storage allows RWO access mode on the same host, but your storage (network based?) does not allow 2 connections to the same volume?

I think we need to check what is the documented way to deploy minio before we try to solve this using hostPath. I'm guessing that what we do is not the documented way and this can explain why we keep our own minio yaml and not using upstream example (like we do for rook).

[nirs]

Looking in minio docs, the yaml is much simpler and it does not have a config pvc: https://raw.githubusercontent.com/minio/docs/master/source/extra/examples/minio-dev.yaml

[matancarmeli7]

@matancarmeli7 If I understand correctly, the issue is that the config pvc has access mode ReadWriteOnce but it is accessed by both the minio container and the setup job at the same time.

Maybe minikube storage allows RWO access mode on the same host, but your storage (network based?) does not allow 2 connections to the same volume?

I think we need to check what is the documented way to deploy minio before we try to solve this using hostPath. I'm guessing that what we do is not the documented way and this can explain why we keep our own minio yaml and not using upstream example (like we do for rook).

Our CSI driver supports only RWO so we can't use PVC with two different pods, probably the storage provider in Minikube supports RWM/RWX

[matancarmeli7]

minio.yaml this is an example to a minio with hostPath in the config

[nirs]

I think the best way to solve this is to use simpler minio yaml from minio docs that does not have a config pvc, so it does not have the issue of accessing the cofnig pvc concurrently.