search

ome / ome2024-ngff-challenge

Project planning and material repository for the 2024 challenge to generate 1 PB of OME-Zarr data
https://pypi.org/project/ome2024-ngff-challenge/
BSD 3-Clause "New" or "Revised" License
11 stars 8 forks source link

TypeError: AioSession.__init__() got an unexpected keyword argument '//' #22

Open will-moore opened 1 month ago

will-moore commented 1 month ago

On my local Mac, I created a new conda env and installed just ome2024-ngff-challenge, then tried converting the remote sample from https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.4/idr0062A/6001240.zarr as described in the README...

(NB: I know this is working for Dom on our pilot-zarr3-dev server with env from https://github.com/dominikl/nf_ngff_conv/blob/main/bf2raw_env.yml but trying to use the env.yml on my local Mac took too long)

$ conda create -n ngff2024 python=3.12
$ conda activate ngff2024

$ pip install -e .

Successfully installed CacheControl-0.14.0 MarkupSafe-2.1.5 aiobotocore-2.13.1 aiohappyeyeballs-2.3.4 aiohttp-3.10.1 aioitertools-0.11.0 aiosignal-1.3.1 arcp-0.2.1 asciitree-0.3.3 attrs-24.1.0 bioblend-1.3.0 botocore-1.34.131 certifi-2024.7.4 charset-normalizer-3.3.2 click-8.1.7 crc32c-2.4.1 donfig-0.8.1.post1 fasteners-0.19 filelock-3.15.4 frozenlist-1.4.1 fsspec-2024.6.1 galaxy2cwl-0.1.4 gxformat2-0.19.0 idna-3.7 iniconfig-2.0.0 isodate-0.6.1 jinja2-3.1.4 jmespath-1.0.1 mistune-3.0.2 ml-dtypes-0.4.0 msgpack-1.0.8 multidict-6.0.5 mypy-extensions-1.0.0 numcodecs-0.13.0 numpy-2.0.1 ome2024-ngff-challenge-0.0.5.post5.dev0+1674057 packaging-24.1 pluggy-1.5.0 pyparsing-3.1.2 pytest-8.3.2 python-dateutil-2.9.0.post0 pyyaml-6.0.1 rdflib-7.0.0 requests-2.32.3 requests-toolbelt-1.0.0 rocrate-0.10.0 ruamel.yaml-0.18.6 ruamel.yaml.clib-0.2.8 s3fs-2024.6.1 schema-salad-8.7.20240718183047 six-1.16.0 tensorstore-0.1.64 tinydb-4.8.0 tqdm-4.66.5 tuspy-1.0.3 typing-extensions-4.12.2 urllib3-2.2.2 wrapt-1.16.0 yarl-1.9.4 zarr-3.0.0a0 zstandard-0.23.0

$ ome2024-ngff-challenge \
>         --input-bucket=idr \
>         --input-endpoint=https://uk1s3.embassy.ebi.ac.uk \
>         --input-anon \
>         zarr/v0.4/idr0062A/6001240.zarr \
>         /tmp/6001240.zarr
Traceback (most recent call last):
  File "/Users/wmoore/opt/anaconda3/envs/ngff2024/bin/ome2024-ngff-challenge", line 8, in <module>
    sys.exit(cli())
             ^^^^^
  File "/Users/wmoore/Desktop/NGFF/ome2024-ngff-challenge/src/ome2024_ngff_challenge/resave.py", line 724, in cli
    converted = main(ns, rocrate)
                ^^^^^^^^^^^^^^^^^
  File "/Users/wmoore/Desktop/NGFF/ome2024-ngff-challenge/src/ome2024_ngff_challenge/resave.py", line 545, in main
    input_config.open_group()
  File "/Users/wmoore/Desktop/NGFF/ome2024-ngff-challenge/src/ome2024_ngff_challenge/resave.py", line 251, in open_group
    self.zr_group = zarr.open_group(store=self.zr_store, zarr_format=2)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/wmoore/opt/anaconda3/envs/ngff2024/lib/python3.12/site-packages/zarr/api/synchronous.py", line 175, in open_group
    sync(
  File "/Users/wmoore/opt/anaconda3/envs/ngff2024/lib/python3.12/site-packages/zarr/sync.py", line 92, in sync
    raise return_result
  File "/Users/wmoore/opt/anaconda3/envs/ngff2024/lib/python3.12/site-packages/zarr/sync.py", line 51, in _runner
    return await coro
           ^^^^^^^^^^
  File "/Users/wmoore/opt/anaconda3/envs/ngff2024/lib/python3.12/site-packages/zarr/api/asynchronous.py", line 523, in open_group
    return await AsyncGroup.open(store_path, zarr_format=zarr_format)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/wmoore/opt/anaconda3/envs/ngff2024/lib/python3.12/site-packages/zarr/group.py", line 144, in open
    zgroup_bytes, zattrs_bytes = await asyncio.gather(
                                 ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/wmoore/opt/anaconda3/envs/ngff2024/lib/python3.12/site-packages/zarr/store/core.py", line 35, in get
    return await self.store.get(self.path, prototype=prototype, byte_range=byte_range)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/wmoore/opt/anaconda3/envs/ngff2024/lib/python3.12/site-packages/zarr/store/remote.py", line 98, in get
    await (
  File "/Users/wmoore/opt/anaconda3/envs/ngff2024/lib/python3.12/site-packages/s3fs/core.py", line 1128, in _cat_file
    return await _error_wrapper(_call_and_read, retries=self.retries)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/wmoore/opt/anaconda3/envs/ngff2024/lib/python3.12/site-packages/s3fs/core.py", line 145, in _error_wrapper
    raise err
  File "/Users/wmoore/opt/anaconda3/envs/ngff2024/lib/python3.12/site-packages/s3fs/core.py", line 113, in _error_wrapper
    return await func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/wmoore/opt/anaconda3/envs/ngff2024/lib/python3.12/site-packages/s3fs/core.py", line 1115, in _call_and_read
    resp = await self._call_s3(
           ^^^^^^^^^^^^^^^^^^^^
  File "/Users/wmoore/opt/anaconda3/envs/ngff2024/lib/python3.12/site-packages/s3fs/core.py", line 358, in _call_s3
    await self.set_session()
  File "/Users/wmoore/opt/anaconda3/envs/ngff2024/lib/python3.12/site-packages/s3fs/core.py", line 519, in set_session
    self.session = aiobotocore.session.AioSession(**self.kwargs)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: AioSession.__init__() got an unexpected keyword argument '//'
joshmoore commented 3 weeks ago

Hmmm.... I'm not sure @will-moore. With these versions:

aiobotocore==2.13.2
aiohappyeyeballs==2.4.0
aiohttp==3.10.5
aioitertools==0.11.0
aiosignal==1.3.1

things are working for me. (Fresh install just now in a new virtualenv) Even downgrading some of the libraries didn't show the error:

pip install -U aiobotocore==2.13.1 aiohttp==3.10.1
will-moore commented 3 weeks ago

On my python 3.10 env...

$ pip freeze | grep aio
aiobotocore==2.13.0
aiohttp==3.9.5
aioitertools==0.11.0
aiosignal==1.3.1

$ pip install -U "aiohttp==3.10.5"
$ pip install -U "aiobotocore==2.13.2"

$ pip freeze | grep aio   (now an exact match)
aiobotocore==2.13.2
aiohappyeyeballs==2.4.0
aiohttp==3.10.5
aioitertools==0.11.0
aiosignal==1.3.1

$ ome2024-ngff-challenge         --input-bucket=idr         --input-endpoint=https://uk1s3.embassy.ebi.ac.uk         --input-anon         zarr/v0.4/idr0062A/6001240.zarr         /tmp/6001240.zarr
Traceback (most recent call last):
  File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/bin/ome2024-ngff-challenge", line 8, in <module>
    sys.exit(cli())
  File "/Users/wmoore/Desktop/NGFF/ome2024-ngff-challenge/src/ome2024_ngff_challenge/resave.py", line 866, in cli
    converted = main(ns, rocrate)
  File "/Users/wmoore/Desktop/NGFF/ome2024-ngff-challenge/src/ome2024_ngff_challenge/resave.py", line 666, in main
    input_config.open_group()
  File "/Users/wmoore/Desktop/NGFF/ome2024-ngff-challenge/src/ome2024_ngff_challenge/resave.py", line 300, in open_group
    self.zr_group = zarr.open_group(store=self.zr_store, zarr_format=2)
  File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/zarr/api/synchronous.py", line 175, in open_group
    sync(
  File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/zarr/sync.py", line 92, in sync
    raise return_result
  File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/zarr/sync.py", line 51, in _runner
    return await coro
  File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/zarr/api/asynchronous.py", line 523, in open_group
    return await AsyncGroup.open(store_path, zarr_format=zarr_format)
  File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/zarr/group.py", line 144, in open
    zgroup_bytes, zattrs_bytes = await asyncio.gather(
  File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/zarr/store/core.py", line 35, in get
    return await self.store.get(self.path, prototype=prototype, byte_range=byte_range)
  File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/zarr/store/remote.py", line 98, in get
    await (
  File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/s3fs/core.py", line 1128, in _cat_file
    return await _error_wrapper(_call_and_read, retries=self.retries)
  File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/s3fs/core.py", line 145, in _error_wrapper
    raise err
  File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/s3fs/core.py", line 113, in _error_wrapper
    return await func(*args, **kwargs)
  File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/s3fs/core.py", line 1115, in _call_and_read
    resp = await self._call_s3(
  File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/s3fs/core.py", line 358, in _call_s3
    await self.set_session()
  File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/s3fs/core.py", line 519, in set_session
    self.session = aiobotocore.session.AioSession(**self.kwargs)
TypeError: AioSession.__init__() got an unexpected keyword argument '//'

Tried the same on a different python 3.12 env, as above... but got the same fail!

joshmoore commented 3 weeks ago

Do you have anything in an ~/.aws directory?

will-moore commented 3 weeks ago

Hmm yes...

$ cat ~/.aws/config
[default]
region =  
$ cat ~/.aws/credentials 
[default]
aws_access_key_id = ***************
aws_secret_access_key = ****************
joshmoore commented 3 weeks ago

I'd suggest moving that out of the way.

will-moore commented 3 weeks ago

$ mv .aws aws_configs made no difference I'm afraid. But thanks for the suggestion.