search

audeering / audbackend

Manage file storage on different backends
https://audeering.github.io/audbackend/
Other
3 stars 0 forks source link

Check if we can integrate fsspec #230

Open hagenw opened 2 weeks ago

hagenw commented 2 weeks ago

There exists a similar package to audbackend, which provides abstractions for backends: https://filesystem-spec.readthedocs.io/

It comes already with a bunch of implementations:

>>> import fsspec
>>> fsspec.available_protocols()
['abfs',
 'adl',
 'arrow_hdfs',
 'asynclocal',
 'az',    
 'blockcache',
 'box',
 'cached',
 'dask',
 'data',
 'dbfs',
 'dir',
 'dropbox',
 'dvc',
 'file',
 'filecache',
 'ftp',
 'gcs',
 'gdrive',
 'generic',
 'git',
 'github',
 'gs',
 'hdfs',
 'hf',
 'http',
 'https',
 'jlab',
 'jupyter',
 'lakefs',
 'libarchive',
 'local',
 'memory',
 'oci',
 'ocilake',
 'oss',
 'reference',
 'root',
 's3',
 's3a',
 'sftp',
 'simplecache',
 'smb',
 'ssh',
 'tar',
 'wandb',
 'webdav',
 'webhdfs',
 'zip']

One way to make audbackend compatible, so we could use all those already existing backends, would be to make audbackend.backend.Base compatible with the fsspec specifications.