fsspec / s3fs

S3 Filesystem
http://s3fs.readthedocs.io/en/latest/
BSD 3-Clause "New" or "Revised" License
892 stars 274 forks source link

s3fs rm fails silently #838

Open dhirschfeld opened 11 months ago

dhirschfeld commented 11 months ago

I'm observing that rm is swallowing underlying errors and not deleting the files it was requested to delete:

>>> fs.rm(path)
[]
>>> fs.rm_file(path)
<snip>
File C:\python\envs\dev-py310\lib\site-packages\s3fs\core.py:1910, in S3FileSystem._rm_file(self, path, **kwargs)
   1907 self.invalidate_cache(path)
   1909 try:
-> 1910     await self._call_s3("delete_object", Bucket=bucket, Key=key)
   1911 except ClientError as e:
   1912     raise translate_boto_error(e)

PermissionError: Access Denied

i.e. rm is failing silently.

My expectation when I call rm is that only one of 2 things will happen - either:

  1. All files were deleted successfully
  2. Or an error is raised

I'm fine with some of the files having been deleted and some not so long as an exception is raised telling me that not all of the files/folders I requested to be deleted were able to be deleted.

martindurant commented 11 months ago

rm() expands its inputs (globs and maybe recursive) and returns the list of deleted files. If no file were found, it should raise FileNotFound I suppose.

In _bulk_delete there is a comment to extract errors from files found but not deleted. This should be filled out.