search

fsspec / sshfs

sshfs - SSH/SFTP implementation for fsspec
Apache License 2.0
58 stars 14 forks source link

`get_file` behaves differently using `SSHFileSystem` vs `LocalFileSystem ` #20

Open sguldemond opened 2 years ago

sguldemond commented 2 years ago

I've implemented the AbstractFileSystem in my code in order to direct my application either to the local file system or file system over SSH. Only I noticed that the behavior of get_file is different in both. I wrote this little script to test and demonstrate.

from fsspec.implementations.local import LocalFileSystem
from sshfs import SSHFileSystem

ssh_fs = SSHFileSystem(
    "localhost",
    username="foobar",
    password="foobar",
)
ssh_fs.get_file("/tmp/foobar", ".")

local_fs = LocalFileSystem()
local_fs.get_file("/tmp/foobar2", ".")

I would expect that calling get_file on either with similar parameters would result the copying of the requested file to the local folder. Only the LocalFileSystem implementation results in an error:

IsADirectoryError: [Errno 21] Is a directory: '/home/west/Research/sshfs/.'

The LocalFileSystem implementation requires a full file path:

local_fs.get_file("/tmp/foobar2", "./foobar2")

I seems to me that the implemenation of get_file in SSHFileSystem does not follow the fsspec API. Or am I missing something?

efiop commented 2 years ago

Hi @sguldemond !

Good catch! Indeed, it is an inconsistecy with LocalFileSystem. I'm not 100% sure how all other implementations behave though. My guess is that it probably depends on the underlying library that is used in particular implementation. One could possibly consider this an extended feature support, so I'm not sure if this is really worth going around and unifying across all implementations. This needs a bit more research to see if there is a more general solution that is feasible.