improve error message of operations on a file if it is 'StorageClass': 'DEEP_ARCHIVE'

[raybellwaves]

I was trying to get a file today and I realized I could not as it was 'StorageClass': 'DEEP_ARCHIVE'.

Doing fs.get("s3://BUCKET/FILE", "FILE") gave PermissionError: The operation is not valid for the object's storage class (full trackback below).

In doing fs.info("s3://BUCKET/FILE") I see 'StorageClass': 'DEEP_ARCHIVE'

It would be good if the error message added this as well e.g. "PermissionError: .... storage class is: {}".(fs.info("s3://BUCKET/FILE")['StorageClass'])

ClientError                               Traceback (most recent call last)
/opt/envs/shared/ds/lib/python3.8/site-packages/s3fs/core.py in _fetch_range(client, bucket, key, version_id, start, end, max_attempts, req_kw)
   1332         try:
-> 1333             resp = client.get_object(Bucket=bucket, Key=key,
   1334                                      Range='bytes=%i-%i' % (start, end - 1),

/opt/envs/shared/ds/lib/python3.8/site-packages/botocore/client.py in _api_call(self, *args, **kwargs)
    315             # The "self" in this scope is referring to the BaseClient.
--> 316             return self._make_api_call(operation_name, kwargs)
    317 

/opt/envs/shared/ds/lib/python3.8/site-packages/botocore/client.py in _make_api_call(self, operation_name, api_params)
    634             error_class = self.exceptions.from_code(error_code)
--> 635             raise error_class(parsed_response, operation_name)
    636         else:

ClientError: An error occurred (InvalidObjectState) when calling the GetObject operation: The operation is not valid for the object's storage class

During handling of the above exception, another exception occurred:

PermissionError                           Traceback (most recent call last)
<ipython-input-26-e6fe1581ad35> in <module>
----> 1 fs.get("s3://BUCKET/FILE", "FILE")

/opt/envs/shared/ds/lib/python3.8/site-packages/fsspec/spec.py in get(self, rpath, lpath, recursive, **kwargs)
    724         lpaths = other_paths(rpaths, lpath)
    725         for lpath, rpath in zip(lpaths, rpaths):
--> 726             self.get_file(rpath, lpath, **kwargs)
    727 
    728     def put_file(self, lpath, rpath, **kwargs):

/opt/envs/shared/ds/lib/python3.8/site-packages/fsspec/spec.py in get_file(self, rpath, lpath, **kwargs)
    704                     data = True
    705                     while data:
--> 706                         data = f1.read(self.blocksize)
    707                         f2.write(data)
    708 

/opt/envs/shared/ds/lib/python3.8/site-packages/fsspec/spec.py in read(self, length)
   1430             # don't even bother calling fetch
   1431             return b""
-> 1432         out = self.cache._fetch(self.loc, self.loc + length)
   1433         self.loc += len(out)
   1434         return out

/opt/envs/shared/ds/lib/python3.8/site-packages/fsspec/caching.py in _fetch(self, start, end)
    341         ):
    342             # First read, or extending both before and after
--> 343             self.cache = self.fetcher(start, bend)
    344             self.start = start
    345         elif start < self.start:

/opt/envs/shared/ds/lib/python3.8/site-packages/s3fs/core.py in _fetch_range(self, start, end)
   1198 
   1199     def _fetch_range(self, start, end):
-> 1200         return _fetch_range(self.fs.s3, self.bucket, self.key, self.version_id, start, end, req_kw=self.req_kw)
   1201 
   1202     def _upload_chunk(self, final=False):

/opt/envs/shared/ds/lib/python3.8/site-packages/s3fs/core.py in _fetch_range(client, bucket, key, version_id, start, end, max_attempts, req_kw)
   1350                                                               'InvalidRange']:
   1351                 return b''
-> 1352             raise translate_boto_error(e)
   1353         except Exception as e:
   1354             if 'time' in str(e).lower():  # Actual exception type changes often

PermissionError: The operation is not valid for the object's storage class

[martindurant]

Yes please! An info is called on file opening, so that's the best place to put this.

[martindurant]

Actually, I completely agree with your original version - but only call info if "storage class" is in the error message.