enable FUSE mounting remote storage - pre-study

[rokroskar]

To better understand our options when it comes to enabling access to remote storage, we should understand the limitations of using FUSE in various scenarios:

• S3
• NFS
• SFTP

Specifically we need to understand whether there are circumstances under which FUSE may simply not be an option for our users, e.g. under:

• Mac OS
• Windows
• certain Linux installations
• restricted environments like HPC clusters

[olevski]

@rokroskar for testing this out on mac/linux/windows is it enough to spin up VMs with the OS and see if I can mount a S3, SFTP and NFS drives on each using the appropriate fuse-based utility? Or are you expecting that we should test how this would potentially work from within the renku CLI in more details?

As for the HPC cluster I guess we would do the same. Just spinning up a HPC cluster is a lot more complicated os it would be nice to just get access to an existing one.

[rokroskar]

I think VMs are fine. Just keep track of any potentially invasive use-access issues that need to be resolved to make fuse work.

We do have access to HPC clusters at EPFL and ETH. For ETH, anyone with a NETHZ account can log in to euler.ethz.ch (need to be on the VPN).

[olevski]

Ok cool. Here is a list then of publicly available buckets, NFS drives, SFTP that I will use to test:

• S3 bucket:

  • bucket name: giab
  • endpoint: http://s3.amazonaws.com (us-east-1 location)
  • publicly available

• NFS drive:

  • See below - easiest so far is to run the NFS server in a Docker container

• FTP:

  • host: demo@test.rebex.net
  • user: demo
  • password: password
  • sftp demo@test.rebex.net
  • source: https://test.rebex.net/
  • an alternative is https://www.ngdc.noaa.gov/ftp.html

[olevski]

NFS server setup

Dockerfile

Please note that you should create a text file called test-nfs-share-file.txt that will be shared.

FROM erichough/nfs-server
COPY test-nfs-share-file.txt /nfs_exports/
VOLUME /nfs_exports
ENV NFS_EXPORT_0='/nfs_exports    *(rw,no_subtree_check,anonuid=1001,anongid=1001)'

Build and run commands

docker build -t test-nfs-server:0.0.1 .
docker run -ti --rm --privileged --name nfs_server test-nfs-server:0.0.1

Mounting

1. Find the ip of the docker container by running docker inspect <container_id> and looking under Networks.IPAddress
2. Run `mount nfs :/nfs_exports /local_folder_to_mount

[olevski]

Linux observations

NFS:

• Mounting NFS drives requires root and it requires the nfs-common library to be installed. I.e. running the mount command would not work without sudo mount ....
• The way to "bypass" the NFS root requirement for mounting is to edit the /etc/fstab file in the NFS client as root with an entry like <container_ip>:/nfs_exports /client/mount/location nfs rw,relatime,user,noauto 0 0. After this only the specified share from the specified server can be mounted without root privileges. I do not think this is easy to do on a random user machine from within the Renku CLI.
• Potentially fuse-nfs can be used to do this. But it seems the library has to be compiled and installed and only perhaps Red Hat's RPM provides a package that can be installed. But even here the library explains that root access is needed to access the NFS files. See the readme in the repo.
• Requires root

FTP:

• Requires a library called curlftpfs which uses fuse under the hood. Run sudo apt install -y curlftpfs
• Mount by doing curlftpfs test.rebex.net mount_location -o user=demo:password
• Does not require root

S3:

• Several candidates (i.e. s3fs or goofys are some fuse-based cli options)
• For example s3fs can be installed via apt sudo apt install s3fs, goofys can be downloaded as an executable
• Mount by doing s3fs -f giab test_mount -o url=http://s3.amazonaws.com -o public_bucket=1
• Does not require root

[olevski]

Windows

• It seems that for a CLI rclone is the only option, rclone additionally requies https://winfsp.dev/rel/ to be installed
• FTP and S3 should be supported
• Looking at NFS support

Since we do not support "regular" Windows I have decided to not pursue this further.

One slight problem with rclone that is that it requires a configuration to be written in a file so that it can be used. But if we decide to use rclone I do not think this is a show-stopper but rather an annoyance.

WSL2

• I tried to setup a windows VM to try this out on GCP but it would not run WSL2 because it did not have proper virtualization settings enabled. I have no other computer with windows so I did not test this.
• According to this Microsoft dev blog WSL2 is supposed to support FUSE essentially out of the box, if this is the case then mounting stuff on WSL2 is essentially the same as Linux

WSL1

• Seems to be a pre-release of a project that is supposed to make this possible. But considering this project is not led by Microsoft and that we have WSL2 I would say that relying on this is extremely risky.
• Not doable.

[olevski]

Mac

NFS:

• I did not have a chance to try this because Docker on Mac cannot run an NFS server (the docker VM on Mac needs the NFS kernel module to be loaded but this not a simple thing to do it seems).
• In theory and by looking at the mac os mount cli command docs this should work exactly the same as in Linux, and the problem is that this also brings the same limitations as Linux
• Requires root

FTP:

• curlftpfs can be installed through macports
• There is a brew version for it here but is being deprecated

S3:

• MacFuse (the library needed to use fuse on mac) is kinda broken on my computer, and I have been trying without success to fix it. But Before my MacFuse installation broke I could successfully use goofys or s3fs to mount a s3 bucket without root privileges
• The commands here are the same as Linux
• Does not require root

The biggest problem with Mac is that MacFuse is not open source and brew has stopped maintaining all formulae that require fuse. That means that installing tools to mount things through FUSE in Mac is not that simple and will probably not improve in the near future.

In addition installing MacFuse can be a pain. I currently have it in some weird broken state that I cannot fix. No matter how many times I delete and install it. I am not sure if this is simply me being extremely unlucky or other people have similar experiences. The weirdest part is that after you install fuse I think you need to restart your computer but right after your computer comes up you have to go into the security settings and approve the use of it. If you miss this then fuse will not work.

[olevski]

In summary:

• NFS needs root privileges across the board. A fuse-nfs project exists but it can only be installed by RPM (i.e. Redhat distros) otherwise it has to be compiled and according to its docs it still will probably need root
• For windows the only option is WSL2 (WSL1 will not work as it does not support Fuse)
• We need MacFuse for Mac
• Assuming that Fuse (on WSL2, Linux or Mac) is installed and functioning then we can mount FTP and S3
• Rclone is promising as it supports many OS's and can handle S3 and FTP but requires a file-based configuration. I think this is a small price to pay for the convenience of not having to install different tools for each data source. I guess that the Rclone config file that Renku CLI will use can live somewhere in the users .renku folder
• If we do not use Rclone then we would need to install specific tools for each data source, in addition to this Rclone supports many different data sources in addition to S3 and FTP already and we get these for free

After doing this excercise I am still very worried that adding this feature would require us to troubleshoot Rclone and Fuse installations for different users across many different OS's.

[olevski]

@rokroskar let me know what you think. The TLDR is right above this comment ☝

[rokroskar]

Thanks for this summary @olevski ... it definitely looks like relying on FUSE on the user installations is going to be a huge liability. Another thing that occurred to me is that for most users they probably have their networked drive mounted by some other means, e.g. for labs they would probably have something like the NAS available via an SMB share or something like that. So in those cases, we don't need to handle the mounting itself but only keep track of where the data is mounted to map it to what is being used for the given project. In the hosted sessions we can probably take care of the mounting and the book-keeping automatically.

So for working locally it should be

1. copy the data, if possible
2. user provides an existing location of the data (in the case of networked drives)
3. ? not sure what other option there is - we could "fuse mount at your own risk"?