COGReader can't open s3 URI in requester pays bucket

[i21510]

Hi,

I have created a fresh python 3.8 virtual environment. I have only installed jupyterlab and aiocogeo with s3 support:

pip install aiocogeo[s3]

This is a sample of my pip freeze output:

aioboto3==9.3.1
aiobotocore==2.1.0
aiocache==0.11.1
aiocogeo==0.3.0
aiofiles==0.8.0
aiohttp==3.6.2
aioitertools==0.9.0
boto3==1.20.24
botocore==1.23.24

Configuration to read from requester-pays bucket seems to be correct:

print(os.environ['AWS_REQUEST_PAYER'])
requester

The following snippet attempts to open an s3 URI in a requester-pays bucket.

from aiocogeo import COGReader

async with COGReader("s3://usgs-landsat/collection02/level-2/standard/oli-tirs/2017/031/032/LC08_L2SP_031032_20171212_20200902_02_T1/LC08_L2SP_031032_20171212_20200902_02_T1_SR_B2.TIF") as cog:
    print(cog.profile)

It fails with the following error:

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Input In [6], in <module>
----> 1 async with COGReader("s3://usgs-landsat/collection02/level-2/standard/oli-tirs/2017/031/032/LC08_L2SP_031032_20171212_20200902_02_T1/LC08_L2SP_031032_20171212_20200902_02_T1_SR_B2.TIF") as cog:
      2     print(cog.profile)

File ~/.virtualenvs/cogeo/lib/python3.8/site-packages/aiocogeo/cog.py:60, in COGReader.__aenter__(self)
     58 async def __aenter__(self):
     59     """Open the image and read the header"""
---> 60     await self.open()
     61     return self

File ~/.virtualenvs/cogeo/lib/python3.8/site-packages/aiocogeo/cog.py:72, in COGReader.open(self)
     71 async def open(self):
---> 72     await self._open()

File ~/.virtualenvs/cogeo/lib/python3.8/site-packages/aiocogeo/cog.py:76, in COGReader._open(self)
     74 async def _open(self):
     75     """internal method to open the cog by reading the file header"""
---> 76     async with Filesystem.create_from_filepath(self.filepath, **self.kwargs) as file_reader:
     77         self._file_reader = file_reader
     78         # Do the first request

File ~/.virtualenvs/cogeo/lib/python3.8/site-packages/aiocogeo/filesystems.py:289, in S3Filesystem.__aenter__(self)
    287 async def __aenter__(self):
    288     splits = urlsplit(self.filepath)
--> 289     self.resource = await aioboto3.resource('s3').__aenter__()
    290     self.object = await self.resource.Object(splits.netloc, splits.path[1:])
    291     return self

AttributeError: module 'aioboto3' has no attribute 'resource'

I have no trouble downloading the file with aws-cli

Thanks

[geospatial-jeff]

Thanks for the issue! I think this is caused by aioboto3>=9.0.0, please try installing something like aioboto3==8.* and see if that fixes it.

[i21510]

That fixed it, thanks!

[i21510]

Hi, After updating another dependency in my project, I also had to install versions of botocore/boto3/aiobotocore which are not compatible with aioboto3 8.x. Thus, I'm stuck with a package conflict. Are there any plans of updating to aioboto3 9.x? Is it even possible to do so?

Thanks

[i21510]

Found a workaround, you can build a presigned URL even if the bucket does not belong to your account and has requester pays enabled: https://stackoverflow.com/questions/61199204/how-to-create-a-signed-s3-url-for-requester-pays-bucket-in-python