Python code not working due to conda error

[bobturneruk]

... trying to work on the java API and get the python upload/download code to work with this.

The python code is currently not working because Conda uses S3FS version 5.2 instead of the required 4.2. After I run "conda install s3fs=4.2.0" it all works fine.

[bobturneruk]

Finally got to looking at this @B0SKAMP! Current environment.yml pins s3fs=0.4.2:

https://github.com/ScottishCovidResponse/data_pipeline_api/blob/b4b2e51b02a09ef038aea41ff322ffc171325c80/environment.yml#L16

How are you creating your conda environment? Maybe recreating it will fix?

[bobturneruk]

I see now this is with installing from conda cloud, not from using a clone of the repo. I suspect it's the same issue as @johnnonweiler experienced.

[bobturneruk]

This runs without error on Windows 10:

conda create --force -n dpapi_test python=3.6
conda activate dpapi_test
conda install -c scottishcovidresponse data_pipeline_api

I'll check linux next. Raises some interesting questions on how to test conda packages!

[mjb3]

Thanks Bob. Tbh it's not so important now we have the Julia package. Thanks for following up though. Cheers, Martin

On Thu, 11 Feb 2021, 07:39 bobturneruk, notifications@github.com wrote:

Finally got to looking at this @mjb3 https://github.com/mjb3! Current environment.yml pins s3fs=0.4.2:

https://github.com/ScottishCovidResponse/data_pipeline_api/blob/b4b2e51b02a09ef038aea41ff322ffc171325c80/environment.yml#L16

How are you creating your conda environment? Maybe recreating it will fix?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ScottishCovidResponse/SCRCIssueTracking/issues/762#issuecomment-777254184, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABRRIYJVTXIEZYXX4TKBMFLS6OCSHANCNFSM4XLOTJ5A .

[johnnonweiler]

This is a different issue to the one I reported previously. (The issue that I reported appears to have been a problem with the conda server, which was fixed a few hours after I had reported it in our Zulip chat.)

I didn't see this issue at the time, but then I did see it later, after running conda update --all (on Windows 10). ~However, I tried again this morning, and the problem seems to have gone away. I don't know what has changed.~

[johnnonweiler]

I do still see the same problem. (I had forgotten that I need to run the download script to see the error message.)

[bobturneruk]

What command yields the error, please, @johnnonweiler?

[johnnonweiler]

This command yields the error: python -m data_pipeline_api.registry.download --config config.yaml

[bobturneruk]

Thanks. I'm getting (Windows 10, for my own reference):

Traceback (most recent call last):
  File "C:\Users\Bob Turner\.conda\envs\dpapi_test\lib\runpy.py", line 183, in _run_module_as_main
    mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
  File "C:\Users\Bob Turner\.conda\envs\dpapi_test\lib\runpy.py", line 109, in _get_module_details
    __import__(pkg_name)
  File "C:\Users\Bob Turner\Documents\data_pipeline_api\data_pipeline_api\registry\__init__.py", line 1, in <module>
    from data_pipeline_api.registry.downloader import Downloader
  File "C:\Users\Bob Turner\Documents\data_pipeline_api\data_pipeline_api\registry\downloader.py", line 16, in <module>
    from data_pipeline_api.registry.common import (
  File "C:\Users\Bob Turner\Documents\data_pipeline_api\data_pipeline_api\registry\common.py", line 25, in <module>
    from data_pipeline_api.file_api import FileAPI
  File "C:\Users\Bob Turner\Documents\data_pipeline_api\data_pipeline_api\file_api.py", line 6, in <module>
    from dataclasses import dataclass
ModuleNotFoundError: No module named 'dataclasses'

[bobturneruk]

Similar to Linux @johnnonweiler, @B0SKAMP?

[johnnonweiler]

I'm using Windows 10. I get:

Traceback (most recent call last):
  File "C:\Users\jnonweil\Miniconda3\envs\dpapi_test\lib\runpy.py", line 183, in _run_module_as_main
    mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
  File "C:\Users\jnonweil\Miniconda3\envs\dpapi_test\lib\runpy.py", line 109, in _get_module_details
    __import__(pkg_name)
  File "C:\Users\jnonweil\Miniconda3\envs\dpapi_test\lib\site-packages\data_pipeline_api\registry\__init__.py", line 1, in <module>
    from data_pipeline_api.registry.downloader import Downloader
  File "C:\Users\jnonweil\Miniconda3\envs\dpapi_test\lib\site-packages\data_pipeline_api\registry\downloader.py", line 16, in <module>
    from data_pipeline_api.registry.common import (
  File "C:\Users\jnonweil\Miniconda3\envs\dpapi_test\lib\site-packages\data_pipeline_api\registry\common.py", line 23, in <module>
    from s3fs import S3FileSystem
  File "C:\Users\jnonweil\Miniconda3\envs\dpapi_test\lib\site-packages\s3fs\__init__.py", line 1, in <module>
    from .core import S3FileSystem, S3File
  File "C:\Users\jnonweil\Miniconda3\envs\dpapi_test\lib\site-packages\s3fs\core.py", line 13, in <module>
    import aiobotocore
  File "C:\Users\jnonweil\Miniconda3\envs\dpapi_test\lib\site-packages\aiobotocore\__init__.py", line 1, in <module>
    from .session import get_session, AioSession
  File "C:\Users\jnonweil\Miniconda3\envs\dpapi_test\lib\site-packages\aiobotocore\session.py", line 6, in <module>
    from .client import AioClientCreator, AioBaseClient
  File "C:\Users\jnonweil\Miniconda3\envs\dpapi_test\lib\site-packages\aiobotocore\client.py", line 2, in <module>
    from botocore.client import logger, PaginatorDocstring, ClientCreator, \
ImportError: cannot import name 'S3ArnParamHandler' from 'botocore.client' (C:\Users\jnonweil\Miniconda3\envs\dpapi_test\lib\site-packages\botocore\client.py

(As Bram suggests, I believe this can be fixed with conda install s3fs=4.2.0.)

[B0SKAMP]

the main issue seems to be:

conda install -c scottishcovidresponse data_pipeline_api

On my windows 10 fresh updated conda, installs an environment with:  s3fs               pkgs/main/noarch::s3fs-0.5.2-pyhd3eb1b0_0

and the error is indeed the one in john's email below. Bram

On 11/02/2021 11:40, johnnonweiler wrote:

I'm using Windows 10. I get:

Traceback (most recent call last): File "C:\Users\jnonweil\Miniconda3\envs\dpapi_test\lib\runpy.py", line 183, in _run_module_as_main mod_name, mod_spec, code = _get_module_details(mod_name, _Error) File "C:\Users\jnonweil\Miniconda3\envs\dpapi_test\lib\runpy.py", line 109, in _get_module_details import(pkg_name) File "C:\Users\jnonweil\Miniconda3\envs\dpapi_test\lib\site-packages\data_pipeline_api\registry__init__.py", line 1, in from data_pipeline_api.registry.downloader import Downloader File "C:\Users\jnonweil\Miniconda3\envs\dpapi_test\lib\site-packages\data_pipeline_api\registry\downloader.py", line 16, in from data_pipeline_api.registry.common import ( File "C:\Users\jnonweil\Miniconda3\envs\dpapi_test\lib\site-packages\data_pipeline_api\registry\common.py", line 23, in from s3fs import S3FileSystem File "C:\Users\jnonweil\Miniconda3\envs\dpapi_test\lib\site-packages\s3fs__init.py", line 1, in from .core import S3FileSystem, S3File File "C:\Users\jnonweil\Miniconda3\envs\dpapi_test\lib\site-packages\s3fs\core.py", line 13, in import aiobotocore File "C:\Users\jnonweil\Miniconda3\envs\dpapi_test\lib\site-packages\aiobotocore\init__.py", line 1, in from .session import get_session, AioSession File "C:\Users\jnonweil\Miniconda3\envs\dpapi_test\lib\site-packages\aiobotocore\session.py", line 6, in from .client import AioClientCreator, AioBaseClient File "C:\Users\jnonweil\Miniconda3\envs\dpapi_test\lib\site-packages\aiobotocore\client.py", line 2, in from botocore.client import logger, PaginatorDocstring, ClientCreator, \ ImportError: cannot import name 'S3ArnParamHandler' from 'botocore.client' (C:\Users\jnonweil\Miniconda3\envs\dpapi_test\lib\site-packages\botocore\client.py

(As Bram suggests, I believe this can be fixed with |conda install s3fs=4.2.0|.)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ScottishCovidResponse/SCRCIssueTracking/issues/762#issuecomment-777388974, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFRKLVROO25PQXCQU5T7D23S6O633ANCNFSM4XLOTJ5A.

[bobturneruk]

Thanks, I'll try and reproduce on my machine. (I'm sure you're right about the fix, I get same s3fs version, I'm thinking a bit about future-proofing and want to make sure I understand what's going on on my box as well!).

[bobturneruk]

And I kind of want to know why it's using that version as we specify the correct one here https://github.com/ScottishCovidResponse/data_pipeline_api/blob/master/environment.yml

[bobturneruk]

Related to https://github.com/ScottishCovidResponse/SCRCIssueTracking/issues/776

[DennisReddyhoff]

and also #770

[bobturneruk]

Install from conda cloud is no longer supported, use:

pip install git+https://github.com/ScottishCovidResponse/data_pipeline_api.git 

[bobturneruk]

Currently the command yields a different error:

python -m data_pipeline_api.registry.download --config config.yaml

Traceback (most recent call last):
  File "C:\Users\bobturner\.conda\envs\data_pipeline_api\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "C:\Users\bobturner\.conda\envs\data_pipeline_api\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\bobturner\Documents\data_pipeline_api\data_pipeline_api\registry\download.py", line 154, in <module>
    download_cli()
  File "C:\Users\bobturner\.conda\envs\data_pipeline_api\lib\site-packages\click\core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "C:\Users\bobturner\.conda\envs\data_pipeline_api\lib\site-packages\click\core.py", line 782, in main
    rv = self.invoke(ctx)
  File "C:\Users\bobturner\.conda\envs\data_pipeline_api\lib\site-packages\click\core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "C:\Users\bobturner\.conda\envs\data_pipeline_api\lib\site-packages\click\core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "C:\Users\bobturner\Documents\data_pipeline_api\data_pipeline_api\registry\download.py", line 148, in download_cli
    download_from_config_file(config_filename=config, token=token)
  File "C:\Users\bobturner\Documents\data_pipeline_api\data_pipeline_api\registry\download.py", line 129, in download_from_config_file
    download_from_configs(run_metadata, read_configs, token, root)
  File "C:\Users\bobturner\Documents\data_pipeline_api\data_pipeline_api\registry\download.py", line 99, in download_from_configs
    read_config, run_metadata.get(RunMetadata.default_input_namespace)
  File "C:\Users\bobturner\Documents\data_pipeline_api\data_pipeline_api\registry\download.py", line 53, in _parse_read_config
    raise ValueError(f"No namespace specified for read_config {read_config}")
ValueError: No namespace specified for read_config {'where': {'data_product': 'human/mixing-matrix', 'component': 'mixing-matrix'}}

[bobturneruk]

Perhaps this is a problem with my config.yaml. Not sure if this is worth fixing given ongoing changes?