search

galaxyproject / galaxy-helm

Minimal setup required to run Galaxy under Kubernetes
MIT License
41 stars 38 forks source link

Galaxy with CVMFS seems to work on newer kubernetes #342

Open cyrilcros opened 2 years ago

cyrilcros commented 2 years ago

* This isn't a bug, I just want to point out that CVMFS seems to work better now....

Hi, I saw this pull request https://github.com/cernops/cvmfs-csi/pull/35 on the cernops/cvmfs-csi and decided to try it. I added a few edits in a new_provisioner branch https://github.com/cyrilcros/cvmfs-csi/blob/new_provisioner/deployments/helm/cvmfs-csi/values.yaml

That's been working well for me, notably because it has a different csi-cvmfsplugin image that solves some issues (https://github.com/cernops/cvmfs-csi/issues/26). I just added some quick other changes like in this pull request https://github.com/cernops/cvmfs-csi/pull/22/files , or removing the attach component as in https://github.com/cernops/cvmfs-csi/issues/29 .

I am getting 504 timeout errors from nginx I am now trying to fix, but now the PVCs are created as they should on Kubernetes 1.21...

almahmoud commented 2 years ago

Thank you for pointing out that PR!! I'll work on porting the new version on our end to have Galaxy's CVMFS work post-1.19.

nuwang commented 2 years ago

This is super helpful, thanks @cyrilcros. It still looks like the repo is generally behind on maintenance, and the last merged PR was in May, 2020. Perhaps we need to maintain a fork ourselves at https://github.com/CloudVE/cvmfs-csi that will generally work for this helm chart. Another, longer term alternative that was contemplated was to sync all the CVMFS data over to S3 and to switch to S3FS as an alternative using something like: https://github.com/ctrox/csi-s3.

cyrilcros commented 2 years ago

Yes, from what I have seen the Helm chart needed just minor work, but the CSI plugin itself needs an update. The fbarreir/csi-cvmfsplugin:9 Docker image (from this guy I assume: Fernando Barreiro ) seems to use a modernized version, ideally you may get the source code from him. I don't see it on his public Github....

ksuderman commented 2 years ago

@cyrilcros I am trying to get all of these changes merged into our helm charts, but the fbarreir/csi-cvmfsplugin:9 Docker image does not seem to exist on Docker Hub or quay.io and I can not find any contact information for Fernando Barreiro. I was hoping the brinkmanlab/csi-cvmfsplugin:1.2.0 Docker image would be a drop in replacement. The PVCa and PVa are created, but I get the following error when launching Galaxy:

MountVolume.MountDevice failed for volume "pvc-2f420977-f878-4e0b-ae6a-95c387b4c426" : rpc error: code = Internal desc = mount failed: exit status 32 Mounting command: mount Mounting arguments: -t cvmfs -o config=/etc/cvmfs/config-csi-cvmfs-pvc-2f420977-f878-4e0b-ae6a-95c387b4c426-a48189e8-d068-11ec-a3a6-fa163e3c7d6b data.galaxyproject.org /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-2f420977-f878-4e0b-ae6a-95c387b4c426/globalmount Output: CernVM-FS: running with credentials 999:997 CernVM-FS: loading Fuse module... Failed to fetch manifest (22 - history init failure)

Is this still working for you?