search

fsspec / sshfs

sshfs - SSH/SFTP implementation for fsspec
Apache License 2.0
58 stars 14 forks source link

Use sftp RENAME instead of copy and delete in case of moving a file #40

Open florian-dacosta opened 9 months ago

florian-dacosta commented 9 months ago

Hello, I use fsspec to have a single api to work with multiple different storage backend. I ran into some issues with the sftp implementation because the move method is used and fails with the error : asyncssh.misc.ChannelOpenError: Session request failed trying to do a cp command here https://github.com/fsspec/sshfs/blob/main/sshfs/spec.py#L162

From what I understand the cp command is not always (rarely ?) possible on sftp protocol and the posix_rename is not always supported either. Would it be possible to replace this to copy and then remove the file by a sftp rename ? In my case, that did fix the issue and this is the approach used for example in the fs.sshfs implementation. (https://github.com/althonos/fs.sshfs/blob/master/fs/sshfs/sshfs.py#L279) And it seems quite easy since the rename is already implemented in asyncssh. I could do a PR about it is an acceptable change.

efiop commented 9 months ago

Hey Florian,

We are trying to cp over ssh, not over sftp, so I wonder if your server (what is it btw?) doesn't like/support us trying to run commands through ssh. It would be great to look around to understand what is specifically wrong there.

Regarding rename, I don't remember all the specifics about its differences with posix_rename from the top of my head (with exception of atomicity, IIRC), but it indeed shouldn't be worse than cp cmd that we are trying to run that is not atomic anyways, so we should indeed be able eto fallback to it one way or another (or maybe even just replace cp, I haven't thought deeply about it though). Obviously, PR is welcome.