xcube serve published data not completly s3 compatible

[AliceBalfanz]

Describe the bug when using the newest versions of aiobotocore = 1.3.1 and fsspec = 2021.6.1, an error is thrown when accessing a cube which is published via xcube serve. OSError: [Errno 5] An error occurred () when calling the ListObjectsV2 operation:

To Reproduce Steps to reproduce the behavior:

1. Checkout newest xcube version and build a new enviromnet (make sure aiobotocore = 1.3.1 and fsspec = 2021.6.1)

2. open jupyter notebook and from xcube.core.dsio import open_cube and ds = open_cube('http://service.demo.dcs4cop.eu/xcube/api/latest/s3bucket/bc_olci_ns_tirr_v1', format_name='zarr', s3_kwargs=dict(anon=True))

3. view error:

4. when downgrading the packages this issue does not appear.

   $ mamba install aiobotocore=1.2.2 -c conda-forge
   $ mamba install fsspec=0.9.0 -c conda-forge

Expected behavior That no error occurs with newer versions of aiobotocore and fsspec.

[forman]

This is related to #412, where aiobotocore 1.3 is causing problems not seen with 1.2.2.

[forman]

This is strange. With aiobotocore = 1.3.1 and fsspec = 2021.6.1 I can start our server:

    xcube serve -v -c ./examples/serve/demo/config.yml

and then this works without problems:

from xcube.core.dsio import open_cube
ds = open_cube('http://localhost:8080/s3bucket/local', format_name='zarr')

Can you, @AliceBalfanz?

[forman]

Please note that

2. open jupyter notebook and from xcube.core.dsio import open_cube and ds = open_cube('http://service.demo.dcs4cop.eu/xcube/api/latest/s3bucket/bc_olci_ns_tirr_v1', format_name='zarr', s3_kwargs=dict(anon=True))

uses an outdated server version 0.6.1.

[AliceBalfanz]

Can you, @AliceBalfanz?

yes, works for me too (note, it should be ds = open_cube('http://localhost:8080/s3bucket/local', format_name='zarr', s3_kwargs=dict(anon=True)) because .aws might not be present ;) )

uses an outdated server version 0.6.1.

Yeah, so this is most likly to be the issue - all our servers are currently not up to date.... I will start the demo dcs4cop server with the newest version and test to be entirely sure.

[AliceBalfanz]

I updated dcs4cop demo server to 0.8.1. however, now I get a new error again, when calling

• open jupyter notebook and from xcube.core.dsio import open_cube and ds = open_cube('http://service.demo.dcs4cop.eu/xcube/api/latest/s3bucket/bc_olci_ns_tirr_v1', format_name='zarr', s3_kwargs=dict(anon=True))

ResponseParserError: Unable to parse response (not well-formed (invalid token): line 1, column 0), invalid XML received. Further retries may succeed:
b'{\n  "name": "xcube Server",\n  "description": "WMTS, catalogue, data access, tile, feature, time-series services for xarray-enabled data cubes",\n  "version": "0.8.1",\n  "configTime": "2021-07-09T08:31:18.441000Z",\n  "serverTime": "2021-07-09T08:33:09.381000Z",\n  "serverPID": 8649\n}'

[AliceBalfanz]

I will now test with latest master, to see if there is a difference between 0.8.1. and master

[AliceBalfanz]

Same Error Message with 0.8.1. and master as in comment above.

[forman]

Ok, found the reason. xcube fails to parse the correct endpoint URL from passed URL http://service.demo.dcs4cop.eu/xcube/api/latest/s3bucket/bc_olci_ns_tirr_v1 because it expects it to be formatted as an AWS S3 URL: https://<endpoint>/<bucket_name>/<path>. We must therefore specify the endpoint_url explicitely:

ds = open_cube(
    's3bucket/bc_olci_ns_tirr_v1', 
    format_name='zarr', 
    s3_kwargs={
        'anon': True
    },
    s3_client_kwargs={
        'endpoint_url': 'http://service.demo.dcs4cop.eu/xcube/api/latest'
    }
)

I'll therefore close this issue.