search

CS-SI / eodag

Earth Observation Data Access Gateway
https://eodag.readthedocs.io
Apache License 2.0
324 stars 44 forks source link

USGS Config Fails to Load on Initialization #685

Closed varisht-tathya closed 1 year ago

varisht-tathya commented 1 year ago

Describe the bug On initializing eodag without using any custom config, usgs doesn't get initialized. It doesn't show up in list of providers as well. Updating provider configuration using a yaml formatted string fails as well for usgs since it is not a part of the providers list.

Code To Reproduce CLI commands or Python code snippet to reproduce the bug. Please use maximum verbosity using:

from eodag import EODataAccessGateway
from eodag.utils.logging import setup_logging
setup_logging(verbose=3)

dag = EODataAccessGateway()
dag.available_providers()
# ['astraea_eod', 'cop_ads', 'cop_cds', 'cop_dataspace', 'creodias', 'earth_search', 'earth_search_cog', 'earth_search_gcs', 'ecmwf', 'mundi', 'onda', 'peps', 'sara', 'theia', 'usgs_satapi_aws']
dag.update_providers_config(open('/eodag_config.yaml').read())
dag.available_providers()
# ['astraea_eod', 'cop_ads', 'cop_cds', 'cop_dataspace', 'creodias', 'earth_search', 'earth_search_cog', 'earth_search_gcs', 'ecmwf', 'mundi', 'onda', 'peps', 'sara', 'theia', 'usgs_satapi_aws']

Output Compete output obtained with maximal verbosity. 2023-03-28 16:28:43,998 eodag.config [INFO ] (config ) Loading user configuration from: /home/varisht/.config/eodag/eodag.yml 2023-03-28 16:28:44,013 eodag.config [INFO ] (config ) sobloo: unknown provider found in user conf, trying to use provided configuration 2023-03-28 16:28:44,014 eodag.config [WARNING ] (config ) sobloo skipped: could not be loaded from user configuration 2023-03-28 16:28:44,014 eodag.config [DEBUG ] (config ) Traceback (most recent call last): File "/home/varisht/anaconda3/envs/arosics/lib/python3.7/site-packages/eodag/config.py", line 382, in override_config_from_mapping config[provider] = ProviderConfig.from_mapping(new_conf) File "/home/varisht/anaconda3/envs/arosics/lib/python3.7/site-packages/eodag/config.py", line 127, in from_mapping mapping[key] = PluginConfig.from_mapping(mapping[key]) File "/home/varisht/anaconda3/envs/arosics/lib/python3.7/site-packages/eodag/config.py", line 203, in from_mapping c.dict.update(mapping) TypeError: 'NoneType' object is not iterable

2023-03-28 16:28:44,161 eodag.core [INFO ] (core ) usgs: provider needing auth for search has been pruned because no crendentials could be found 2023-03-28 16:28:44,161 eodag.core [INFO ] (core ) aws_eos: provider needing auth for search has been pruned because no crendentials could be found 2023-03-28 16:28:44,161 eodag.core [INFO ] (core ) meteoblue: provider needing auth for search has been pruned because no crendentials could be found 2023-03-28 16:28:44,296 eodag.core [DEBUG ] (core ) Opening product types index in /home/varisht/.config/eodag/.index 2023-03-28 16:28:44,304 eodag.core [INFO ] (core ) Locations configuration loaded from /home/varisht/.config/eodag/locations.yml

Environment:

Additional context Add any other context about the bug here. Have encountered this bug on Python 3.9 as well. I don't see this bug when using a custom yaml config when initializing the EODataAccessGateway.

sbrunato commented 1 year ago

Hello @varisht-tathya , this is the actual expected behaviour:

see https://eodag.readthedocs.io/en/stable/notebooks/api_user_guide/2_providers_products_available.html

If a provider is configured to need authentication for search, and has no crendentials set, it will be pruned on EODAG initialization, and will not appear in available providers list.

Then, if usgs has already been pruned during initialization, it will not be restored.

Is there a particular reason to not use /eodag_config.yaml on EODataAccessGateway init ?

varisht-tathya commented 1 year ago

The reason for not using /eodag_config.yaml on init was that the file would have to be stored locally, which is not great for security considering it contains passwords.

sbrunato commented 1 year ago

ok thanks. We can plan to change this mechanism and restore pruned providers when configuration is updated: Do not delete providers but keep their conf in something like core._pruned_providers_config