search

byteskeptical / sftpretty

Provides multi-threaded routines and high level protocol abstractions for a pretty quick & simple file transfer experience. Super duper close to a drop in replacement for pysftp.
https://docs.sftpretty.com/
BSD 3-Clause "New" or "Revised" License
34 stars 8 forks source link

Can't catch IOError when retry is enabled during uploading a file #40

Closed mawaliya closed 1 year ago

mawaliya commented 1 year ago

python: 3.10.6 sftpretty:1.1.1 paramiko: 3.3.1 Ubuntu 22.04.3 LTS

When trying to catch error with a retry enabled, when the IOError is thrown, it will not be caught and it will give this stack trace:

.venv/lib/python3.10/site-packages/sftpretty/helpers.py:149: in _retry
    return f(*args, **kwargs)
.venv/lib/python3.10/site-packages/sftpretty/__init__.py:603: in _put
    remote_attributes = channel.put(localfile,
.venv/lib/python3.10/site-packages/paramiko/sftp_client.py:759: in put
    return self.putfo(fl, remotepath, file_size, callback, confirm)
.venv/lib/python3.10/site-packages/paramiko/sftp_client.py:714: in putfo
    with self.file(remotepath, "wb") as fr:
.venv/lib/python3.10/site-packages/paramiko/sftp_client.py:372: in open
    t, msg = self._request(CMD_OPEN, filename, imode, attrblock)
.venv/lib/python3.10/site-packages/paramiko/sftp_client.py:822: in _request
    return self._read_response(num)
.venv/lib/python3.10/site-packages/paramiko/sftp_client.py:874: in _read_response
    self._convert_status(msg)
.venv/lib/python3.10/site-packages/paramiko/sftp_client.py:905: in _convert_status
    raise IOError(errno.EACCES, text)
E   PermissionError: [Errno 13] Permission denied

During handling of the above exception, another exception occurred:
tests/ephemeral_performance/conftest.py:492: in run_sod_schedule
    sftp_client.upload_file(
src/common/clients/sftp.py:41: in upload_file
    sftp.put(localfile, remotedir, logger=logger, **kwargs)
.venv/lib/python3.10/site-packages/sftpretty/__init__.py:614: in put
    return _put(self, localfile, remotepath=remotepath, callback=callback,
.venv/lib/python3.10/site-packages/sftpretty/helpers.py:150: in _retry
    except all_exception_types as e:
E   TypeError: catching classes that do not inherit from BaseException is not allowed

Steps to reproduce:

run a similar code like this, but the file exists already in the remote directory so it will give us permission denied error.

try:
    sftp.put(localfile, remotefile, tries=3)
except IOError:
    print("Error caught")
byteskeptical commented 1 year ago

Not sure if I'm missing something but I don't see where you are passing in IOError to the exceptions argument of the put function. So in your example to reproduce I would expect:

try:
    sftp.put(localfile, remotefile, exceptions=IOError, tries=3)
except IOError:
    print("Error caught")

Can you confirm that this fixes the behavior? To be clear I should be dealing with the lack of an exceptions argument a bit more concisely than what you got so it's clear to the user what they are missing.

mawaliya commented 1 year ago

the suggestion works. thank you for the clarification. it wasn't very clear to me that we needed to pass on the exceptions on which we want to retry with tries argument.