Implementing multiple backends by re-using snakemake.remote or pyfilesystem2

[Avsecz]

Would it be possible to wrap the classes implementing snakemake.remote.AbstractRemoteObject (snakemake.remote, AbstractRemoteObject) into lazydata.remote.RemoteStorage class?

This would allow to implement the following remote storage providers in one go (https://snakemake.readthedocs.io/en/stable/snakefiles/remote_files.html):

• Amazon Simple Storage Service (AWS S3): snakemake.remote.S3
• Google Cloud Storage (GS): snakemake.remote.GS
• File transfer over SSH (SFTP): snakemake.remote.SFTP
• Read-only web (HTTP[S]): snakemake.remote.HTTP
• File transfer protocol (FTP): snakemake.remote.FTP
• Dropbox: snakemake.remote.dropbox
• XRootD: snakemake.remote.XRootD
• GenBank / NCBI Entrez: snakemake.remote.NCBI
• WebDAV: snakemake.remote.webdav
• GFAL: snakemake.remote.gfal
• GridFTP: snakemake.remote.gridftp
• iRODS: snakemake.remote.iRODS
• EGA: snakemake.remote.EGA

Pyfilesystem2

Another alternative would be to write a wrapper around pyfilesystem2: https://github.com/PyFilesystem/pyfilesystem2. It supports the following filesystems: https://www.pyfilesystem.org/page/index-of-filesystems/

Builtin

• FTPFS File Transfer Protocol.
• ...

Official

Filesystems in the PyFilesystem organisation on GitHub.

• S3FS Amazon S3 Filesystem.
• WebDavFS WebDav Filesystem.

Third Party

• fs.archive Enhanced archive filesystems.
• fs.dropboxfs Dropbox Filesystem.
• fs-gcsfs Google Cloud Storage Filesystem.
• fs.googledrivefs Google Drive Filesystem.
• fs.onedrivefs Microsoft OneDrive Filesystem.
• fs.smbfs A filesystem running over the SMB protocol.
• fs.sshfs A filesystem running over the SSH protocol.
• fs.youtube A filesystem for accessing YouTube Videos and Playlists.
• fs.dnla A filesystem for accessing accessing DLNA Servers

[rstojnic]

Good idea, this would be great!

[Avsecz]

Can you explain how to implement something like this? E.g. where to put the class, how to name it, which methods to implement and what are the potential caveats?

[rstojnic]

Some thoughts:

1) I would probably keep the interface of the remote.RemoteStorage class unchanged. 2) I would create a new class remote.SnakeMakeRemoteStorage that inherits remote.RemoteStorage and takes at least two parameters: a snakemake backend name + any other necessary parameters (e.g. the access keys). I probably wouldn't want to reimplement the S3 and other existing backends. 3) In remote.RemoteStorage.get_from_url() and remote.RemoteStorage.get_from_config() make sure the remote storage backend is correctly parsed from the lazydata.yml config file and the correct child class SnakeMakeRemoteStorage instantiated. 4) In cli.commands.config I would allow for configuration of any necessary additional access keys 5) In cli.commands.add_remote I would allow for any of the snakemake backends to be specified.

And that should be it. Some unit tests would be welcome as well :)