Closed millingw closed 2 weeks ago
this resembles an error I've seen when there is a ceph user mismatch.
@DP-B21 this resembles the problem you were seeing on the EIDF system, I believe. Can I add you to this ticket to take a look, please, if I don't manage to resolve it tomorrow?
Unfortunately couldn't get to this today.
@DP-B21 Not sure if you are familair with Manila and cephfs. If not, could you give it a go, look over the documentation (e.g. https://docs.openstack.org/manila/latest/admin/shared-file-systems-crud-share.html, you can create shares from the Horizon web gui under project>share>shares) and try to see if you can mount it on a test instance please?
Hi @GregBlow after looking at it I believe it may be a networking issue since export locations gives us:
path = 10.19.4.17:6789,10.19.4.16:6789,10.19.4.18:6789:/volumes/_nogroup/06ed9f64-0806-411b-8dfe-81a36fa1a8b5/aaf96b18-8ac1-44af-8e19-f64d3d1db969
when the network that the VM is in is 10.10.0.0
Is this a mistake on my part, or something that needs to be further investigated?
Hello @millingw , could you clarify which VM is the subject of this issue? Apologies, but I've gone back over the ticket and can't see where is was identified.
I can see that one of the VMs in the gaia project (bob-test-1) has an interface on the CephFS network, however none of the others do. Though the share is managed through the same project, the CephFS has it's own network.
You should be able to add such an interface directly to the instance in question through openstack to check, though depending on the image you've used it may take some work to configure it correctly.
I was trying from the gaia_dataset_one VM. I'll try again with the ceph network explicitly attached to the VM. One thing that puzzles me slightly is that I was apparently able to attach a shared ceph volume ("gaia_dataset_shared_volume") without explicitly being on the ceph network.
At this point it is worth drawing the distinction between Ceph and CephFS. It sounds like attaching a shared ceph volume uses ceph block storage, mediated by Cinder (an openstack service), whereas CephFS is a POSIX-compliant file system, mediated by Manila.
https://docs.ceph.com/en/reef/rbd/ https://docs.ceph.com/en/reef/cephfs/
The latter (as far as I am aware) will always require an interface on the cephfs
network on any instance that is to access it.
Confirmed, with the cephfs network correctly attached to the VM, the ceph share can be mounted successfully.
I've created a ceph share in the somerville-jade project, added an access credential for it, then tried to mount the share in a VM within the same project, having installed ceph-common on the VM and added the access key to ceph.secret
I ran the following command:
I'm not quite sure where I've gone wrong - possibly networking? I assumed that if the VM and share are within the same project there wouldn't be additional networking steps to configure