[BUG]: in morpheus/utils/file_utils.py, date_extractor() errors with access denied on s3

tgrunzweig-cpacket commented 1 year ago

Version

23.07

Which installation method(s) does this occur on?

Docker

Describe the bug.

Run code similar to dfp_duo_training.ipynb but with input file names that do not have explicit data time, so data_extractror executes the following line in file_utils.py:

ts_object = file_object.fs.modified(file_object.path)

getting error:

/opt/conda/envs/morpheus/lib/python3.10/site-packages/botocore/utils.py:1785: FutureWarning: The S3RegionRedirector class has been deprecated for a new internal replacement. A future version of botocore may remove this class.
  warnings.warn(
---------------------------------------------------------------------------
ClientError                               Traceback (most recent call last)
File /opt/conda/envs/morpheus/lib/python3.10/site-packages/s3fs/core.py:110, in _error_wrapper(func, args, kwargs, retries)
    109 try:
--> 110     return await func(*args, **kwargs)
    111 except S3_RETRYABLE_ERRORS as e:

File /opt/conda/envs/morpheus/lib/python3.10/site-packages/aiobotocore/client.py:358, in AioBaseClient._make_api_call(self, operation_name, api_params)
    357     error_class = self.exceptions.from_code(error_code)
--> 358     raise error_class(parsed_response, operation_name)
    359 else:

ClientError: An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied

Minimum reproducible example

# s3_access.py
# This snippet illustrate the  fsspec access method that fails 
# (without the need to run the whole morpheus stack)
# run this in the morpheus container
# you may need to pip install boto3 and botocore
# the boto3 lines illustrates that permission are set on s3 
# you should see all the attributes via the s3.head_objects
# function call,
# but the fsspec.core.OpenFiles.fs.modified() call still fails.

# boto way (not morpheus):
import boto3
import botocore

# fsspec way (morpheus):
import fsspec
import fsspec.utils

session = boto3.Session()
s3 = session.client('s3')

bucket_name = "some-bucket"
key = "some-key.parquet"
print("---all this should work fine---")
print(f"bucket_name =    {bucket_name}")
print(f"key         =    {key}")
print("-------")
print(s3.head_object(Bucket=bucket_name, Key=key))
print("-------")

files: fsspec.core.OpenFiles = fsspec.open_files(f"s3://{bucket_name}/{key}")
filepath = files[0].path
print("---this also should work fine---")
print(f"files={files}")
print("----")
print(f"files[0].path={files[0].path}")
print("----")

# this errors out:
files[0].fs.modified(files[0].path)


### Relevant log output

```shell
---all this should work fine---
bucket_name =    some-bucket
key         =    some-key.parquet
-------
{'ResponseMetadata': {#I removed some sensitive stuff   'last-modified': 'Thu, 29 Jun 2023 20:36:40 GMT' # I removed some more stuff here}
-------
---this also should work fine---
files=<List of 1 OpenFile instances>
----
files[0].path=some-bucket/some-key.parquet
----
/opt/conda/envs/morpheus/lib/python3.10/site-packages/botocore/utils.py:1785: FutureWarning: The S3RegionRedirector class has been deprecated for a new internal replacement. A future version of botocore may remove this class.
  warnings.warn(
Traceback (most recent call last):
  File "/opt/conda/envs/morpheus/lib/python3.10/site-packages/s3fs/core.py", line 110, in _error_wrapper
    return await func(*args, **kwargs)
  File "/opt/conda/envs/morpheus/lib/python3.10/site-packages/aiobotocore/client.py", line 358, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/workspace/examples/digital_fingerprinting/production/morpheus/s3_access.py", line 25, in <module>
    attrib = files[0].fs.modified(files[0].path)
  File "/opt/conda/envs/morpheus/lib/python3.10/site-packages/s3fs/core.py", line 1806, in modified
    info = self.info(path=path, version_id=version_id, refresh=refresh)
  File "/opt/conda/envs/morpheus/lib/python3.10/site-packages/fsspec/asyn.py", line 111, in wrapper
    return sync(self.loop, func, *args, **kwargs)
  File "/opt/conda/envs/morpheus/lib/python3.10/site-packages/fsspec/asyn.py", line 96, in sync
    raise return_result
  File "/opt/conda/envs/morpheus/lib/python3.10/site-packages/fsspec/asyn.py", line 53, in _runner
    result[0] = await coro
  File "/opt/conda/envs/morpheus/lib/python3.10/site-packages/s3fs/core.py", line 1166, in _info
    out = await self._call_s3(
  File "/opt/conda/envs/morpheus/lib/python3.10/site-packages/s3fs/core.py", line 332, in _call_s3
    return await _error_wrapper(
  File "/opt/conda/envs/morpheus/lib/python3.10/site-packages/s3fs/core.py", line 137, in _error_wrapper
    raise err
PermissionError: Access Denied


### Full env printout

```shell
s3fs and fsspec are both  version 22.08.2 in this container.

in an environment where

s3fs and fsspec are  23.06 this runs smoothly with no errors.

Other/Misc.

No response

Code of Conduct

[X] I agree to follow Morpheus' Code of Conduct
[X] I have searched the open bugs and have found no duplicates for this bug report

jarmak-nv commented 1 year ago

Hi @Tzahi-cpacket!

Thanks for submitting this issue - our team has been notified and we'll get back to you as soon as we can! In the mean time, feel free to add any relevant information to this issue.

dagardner-nv commented 1 year ago

The issue is as you pointed out, that the default date_conversion_func used by the DFPFileBatcherStage (https://github.com/nv-morpheus/Morpheus/blob/branch-23.07/docs/source/developer_guide/guides/6_digital_fingerprinting_reference.md#file-batcher-stage-dfpfilebatcherstage) assumes that the creation time of the file is encoded in the filename as an ISO 8601 formatted date string. If that cannot be obtained this way then the method falls-back to attempting to retrieve the modification date from the filesystem (via https://filesystem-spec.readthedocs.io/en/latest/api.html?highlight=modified#fsspec.spec.AbstractFileSystem.modified).

My guess is that this wasn't supported in the versions we currently have in the container, but was added more recent versions.

tgrunzweig-cpacket commented 1 year ago

Is this the relevant environment setup file: https://github.com/nv-morpheus/Morpheus/blob/branch-23.07/docker/conda/environments/cuda11.8_examples.yml ? In there s3fs is rigidly set to 22.08.2, and this is in the latest 23.07 branch. Perhaps needs to be upgraded there?

dagardner-nv commented 1 year ago

@Tzahi-cpacket agreed the only question is if it introduces any incompatibilities with other code. I'll take a pass at it.

nv-morpheus / Morpheus