edmc-ss
commented
5 years ago Welcome to ProxyFS, jnamdar!
Apologies about the lack of the CentOS box... indeed, I believe the path to it needs tweaking. That particular box has the vbox tools installed allowing "synced folders" to work. I trust you found a workable substitute. In any event, we should be updating the CentOS box version we are referencing.
Nice to see you were able to get your Swift Proxy "NoAuth" pipeline up and working. One caution about that pipeline. As you can imagine, it doesn't want to have the pfs_middleware installed (but does need meta_middleware installed)... the reverse of the "normal" Swift Proxy. Another caution... if you enable encryption in your cluster, you need to be sure the encryption middleware is in both pipelines (and after pfs_middleware in the "normal" pipeline).
Keystone Auth for the Swift API pipeline should "just work"... but I must admit to never having tried it. The pfs_middleware that enables ProxyFS sits after the Auth step in the pipeline, so we should be just fine. But if you are asking about Keystone Auth being integrated on the SMB side, I don't have any experience with that alas. Is that the need for you?
Your attempt to get SMB mounting working to your modified setup is very interesting. I was first thinking that one of the authorizing steps was missed so I tried a couple of things.
1) Samba, due to its history of leveraging Linux "authorization" by mapping SMB users to Linux users, means that the SMB user needs to also exist in the Linux user database. I noted, however, that your smbpasswd -a step would have failed if it didn't.
2) I then note that if you haven't added your SMB user to the valid users = line in /etc/samba/smb.conf - and restarted or otherwise "triggered" a refresh - it won't (yet) know about your added SMB user. Alas, what you'd get in this case is a mount error(13): Permission denied, not the mount error(5): Input/output error you received.
What I note has usually happened when getting the error you received I just re-verified. I sent a SIGKILL to the 'proxyfsdprocess. Without something restarting it, the mount will fail in precisely the way you see... and I strongly suspect that has happened. What's going on there is Samba has been configured (for this volume anyway) to invoke the vfs plug-in included in a couple of submodules of ProxyFS (I believe you must have done agit submodule update --recursive` at some point, so you should be fine). Anyway, the jrpcclient submodule is actually the one connecting to the proxyfsd process over a couple of TCP ports (the supplied .conf's have those as ports 12345 & 32345 I believe) on the PrivateIPAddr interface. Port 32345 is used for the read and write data path (only coming from the adjacent on the node Samba instance in your case)... while port 12345 is for all other RPCs... coming from both the adjacent Samba instance as well as all of the pfs_middleware instances in your collection of Swift Proxy nodes.
Anyway, I'm thinking the proxyfsd process has failed. It would be very interesting to see the log from that process (/var/log/proxyfsd/proxyfsd.log).
Next, it looks like you got a "500 Internal Error" when later attempting a Swift API method targeting this Swift Account. This makes total sense. What's going on here is there is a Header in the Swift Account that indicates this is a so-called "BiModal" Account... meaning that ProxyFS manages its content. In your proxy-server.conf's [filter:pfs] section, there are instructions for which proxyfsd instance (IPAddr & Port#) to contact (note the above comments about TCP Port 12345). The pfs_middleware attempts to contact this proxyfsd_host (this can actually be a list btw) asking ProxyFS for the IPAddr of the proxyfsd instance actually serving the Volume. In this case, it should be told "hey, yes, it's me - you've come to the right place". Anyway, it tries really hard to contact the proxyfsd process indicated...and if it fails, you'll see this "500 Internal Error".
So with that, I believe all signs point to a failure of your proxyfsd process. If you can post the log file I mentioned, perhaps we can get to the bottom of this quickly.
Now just one more thing to check out. As you noted, you needed to configure what we call the "NoAuth" Swift Proxy instance adjacent to your proxyfsd process. It should be serving localhost:8090 I believe. If you log onto the "node" running proxyfsd, you should be able to talk to the underlying Swift Account. Just make sure this is possible. Do a HEAD on the "BiModal" Swift Account... you should see the BiModal Header I mentioned earlier indicating it is, in fact, BiModal. If you do a GET on it, you should see at least one Container likely named ".checkpoint". This is the Container that receives all of the file system's metadata for that Volume in the form of a "write log" consisting of LogSegment objects numbered in ascending order (starting with 0000000000000002 or so). There will be holes as the same 64-bit "Nonce" number sequence is used for all sorts of things that must be named uniquely (e.g. Inode#'s, Container names, etc...).
If you don't see the BiModal Header on the Account... or the .checkpoint Container in the Account, then what it sounds like to me is the Volume needs to be formatted. If you look in the start_and_mount_pfs script, you should see a function named format_volume_if_necessary that invokes a tool called mkproxyfs. It takes just a few options... this script uses "-I" to say "format it if it's not already formatted". That's probably what you want... There are other options to say either "only format if the Account is empty" or "hey, empty it first before formatting" (obviously dangerous).
One note about formatting. ProxyFS requires a bit more guarantee than an Eventually Consistent storage system such as Swift provides. It cannot handle getting back "stale" information that an EC system may supply if, say, you did a 2nd PUT of an Object (a subsequent GET could return nothing, the first version, or the 2nd version). To avoid this, ProxyFS never PUTs the same Object (or Container) twice. That's why I mentioned the "Nonce" above. As such, the only way it could "know" that it had never used a given Nonce Value before is if the Account starts out completely empty. So just a heads up that you should either never reuse an Account... or empty if first (e.g. with mkproxyfs -F).
Hope all of this discussion gives you things to look at to get your system running again. I don't know which of these are going to help identify the issue... but hopefully at least one of them will.
And, again, welcome to ProxyFS!
jnamdar
Hi,
Firstly, thanks for this application and for giving us the opportunity to use it.
To try it out, I deployed ProxyFS on a CentOS 7.4 VM using the
Vagrantfilein thesaiosubfolder. By the way, the referenced vagrant box in this file seems to be down, I used this box (config.vm.box = "CentosBox/Centos-7-v7.4-Minimal-CLI"with thevirtualboxprovider to continue.After the
vagrant_provision.shfinished running, I compiled the ProxyFS project usingmake: everything went well. I then used the scriptstart_and_mount_pfsto mount the NFS and SMB share. I can create folders/files in both shares without issues, and then view everything with theswiftCLI:I've been looking for a way to use ProxyFS in my existing OpenStack Swift/Keystone installation:
rocky(installed with this link)rocky(installed using this link)So far, I have been able to deploy a CentOS 7.4 VM using the
Vagrantfilein thesaiosubfolder. I removed everything regarding the installation of Swift (including the creation of the userswift) since I already have one installed.I then fiddled with the ProxyFS configuration on this VM to point to my existing Swift Proxy server. I installed the
pfsandmetamiddlewares on the machine hosting my Swift Proxy server, added them to the pipeline. I also launched another instance of the Proxy server listening on port 8090 with the/etc/swift/proxy-server/proxy-noauth.cond.d/20_settings.conffile:/usr/bin/python2 /usr/bin/swift-proxy-server /etc/swift/proxy-server/proxy-noauth.cond.dFinally I used the script
start_and_mount_pfs, after removing the lines about starting Swift, to launch ProxyFS and mount the NFS and SMB network shares.The NFS share seems to work well (I can create folders and write files), but I'm getting an error trying to mount the SMB one. Relevant info: since I haven't created a
swiftuser, I replaced it with thevagrantuser that was already existing in the VM in thesmb.conffile, and usedsmbpasswd -a vagrant. Command line error:What I find in
/var/log/samba/log.smbdafter addinglog level = 3 passdb:5 auth:5insmb.conf:It looks like the samba authentication went well, but the relevant error to me are the following lines:
I tried troubleshooting this, but no luck so far. Would anyone be able to help on this? Here's my
df -Houtput if needed:I also tried to get containers and objects I created via the NFS share with the Object Storage API, but I got the following error on my Swift Proxy server:
Relevant lines in
/var/log/messagesregarding the error:On another subject, does ProxyFS support Keystone authentication, instead of the
tempauthused in the main pipeline?More broadly, has anyone tried to connect ProxyFS to an existing OpenStack Swift/Keystone installation?
Regards