search

CS-SI / eodag

Earth Observation Data Access Gateway
https://eodag.readthedocs.io
Apache License 2.0
315 stars 43 forks source link

Setting Custom provider as preferred doesn't work #567

Closed remi-braun closed 10 months ago

remi-braun commented 1 year ago

Describe the bug Setting custom provider as preferred doesn't seem to work.

Code To Reproduce

from eodag import setup_logging
from eodag.api.core import EODataAccessGateway

if __name__ == "__main__":
    # Create an EODAG custom STAC provider
    setup_logging(verbose=3)
    dag = EODataAccessGateway()

    # Add the custom STAC provider + output + login an password
    outputs_prefix = r"D:\capella_eodag"

    dag.update_providers_config("""
    capella_provider:
        search:
            type: StaticStacSearch
            api_endpoint: https://capella-open-data.s3.us-west-2.amazonaws.com/stac/capella-open-data-by-product-type/capella-open-data-slc/collection.json
        products:
            GENERIC_PRODUCT_TYPE:
                productType: '{productType}'
        download:
            type: AwsDownload
            flatten_top_dirs: True
            outputs_prefix: %s
    """ % (outputs_prefix))

    # Set the custom STAC provider as preferred
    dag.set_preferred_provider("capella_provider")
    print(dag.get_preferred_provider())

Output

2022-12-07 10:40:35,074 eodag.config                     [INFO    ] (config           ) Loading user configuration from: C:\Users\rbraun\.config\eodag\eodag.yml
2022-12-07 10:40:35,236 eodag.core                       [INFO    ] (core             ) usgs: provider needing auth for search has been pruned because no crendentials could be found
2022-12-07 10:40:35,236 eodag.core                       [INFO    ] (core             ) aws_eos: provider needing auth for search has been pruned because no crendentials could be found
2022-12-07 10:40:35,360 eodag.core                       [DEBUG   ] (core             ) Opening product types index in C:\Users\rbraun\.config\eodag\.index
2022-12-07 10:40:35,370 eodag.core                       [INFO    ] (core             ) Locations configuration loaded from C:\Users\rbraun\.config\eodag\locations.yml
2022-12-07 10:40:35,370 eodag.config                     [INFO    ] (config           ) capella_provider: unknown provider found in user conf, trying to use provided configuration
('peps', 1)

Environment:

Additional context

The custom provider comes from #519

sbrunato commented 1 year ago

Hello @remi-braun , I cannot reproduce the issue on my side... I get:

('capella_provider', 2)

Can you please try to run the following commands at the end of your example, and post the result?

print([(p, c.priority) for p,c in dag.providers_config.items()])
dag.set_preferred_provider("capella_provider")
print([(p, c.priority) for p,c in dag.providers_config.items()])
print(dag.get_preferred_provider())
remi-braun commented 1 year ago

I get: 

[('scihub', 0), ('theia', 0), ('peps', 1), ('sobloo', 0), ('creodias', 0), ('mundi', 0), ('onda', 0), ('astraea_eod', 0), ('usgs_satapi_aws', 0), ('earth_search', 0), ('earth_search_cog', 0), ('earth_search_gcs', 0), ('ecmwf', 0), ('cop_ads', 0), ('cop_cds', 0), ('capella_provider', 0)]
[('scihub', 0), ('theia', 0), ('peps', 1), ('sobloo', 0), ('creodias', 0), ('mundi', 0), ('onda', 0), ('astraea_eod', 0), ('usgs_satapi_aws', 0), ('earth_search', 0), ('earth_search_cog', 0), ('earth_search_gcs', 0), ('ecmwf', 0), ('cop_ads', 0), ('cop_cds', 0), ('capella_provider', 0)]
('peps', 1)

Capella stays at 0 😰

sbrunato commented 1 year ago

Can you try to install v2.7.0 in a new virtual environment, and to use a new empty configuration file (instead of your ~/.config/eodag/eodag.yml) ? This may help us to identify what causes this issue.

I also tried your example on windows, and got capella_provider as preferred provider as expected....

remi-braun commented 1 year ago

Same behaviour after deleting the eodag.yml (which was empty).

However, it works when recreating an empty conda env with only python and eodag:

name: eodownload_2
channels:
  - conda-forge
dependencies:
  - python=3.9
  - eodag

I don't understand why my env doesn't (I just reinstalled it). Conda list gives:

# packages in environment at C:\Users\rbraun\Anaconda3\envs\eodownload:
#
# Name                    Version                   Build  Channel
astroid                   2.12.13                  pypi_0    pypi
attrs                     22.1.0                   pypi_0    pypi
azure-core                1.26.1                   pypi_0    pypi
azure-storage-blob        12.14.1                  pypi_0    pypi
black                     22.10.0                  pypi_0    pypi
bleach                    5.0.1                    pypi_0    pypi
boto3                     1.26.25                  pypi_0    pypi
botocore                  1.29.25                  pypi_0    pypi
bzip2                     1.0.8                h8ffe710_4    conda-forge
ca-certificates           2022.12.7            h5b45459_0    conda-forge
cachetools                5.2.0                    pypi_0    pypi
cdsapi                    0.5.1                    pypi_0    pypi
certifi                   2022.12.7                pypi_0    pypi
cffi                      1.15.1                   pypi_0    pypi
cfgv                      3.3.1                    pypi_0    pypi
chardet                   5.1.0                    pypi_0    pypi
charset-normalizer        2.1.1                    pypi_0    pypi
click                     8.1.3                    pypi_0    pypi
click-plugins             1.1.1                    pypi_0    pypi
cligj                     0.7.2                    pypi_0    pypi
cloudpathlib              0.10.0                   pypi_0    pypi
colorama                  0.4.6                    pypi_0    pypi
colorlog                  6.7.0                    pypi_0    pypi
commonmark                0.9.1                    pypi_0    pypi
coverage                  6.5.0                    pypi_0    pypi
cryptography              38.0.4                   pypi_0    pypi
decorator                 5.1.1                    pypi_0    pypi
dill                      0.3.6                    pypi_0    pypi
distlib                   0.3.6                    pypi_0    pypi
docutils                  0.19                     pypi_0    pypi
ecmwf-api-client          1.6.3                    pypi_0    pypi
eodag                     2.7.0                    pypi_0    pypi
eodag-sentinelsat         0.4.1                    pypi_0    pypi
exceptiongroup            1.0.4                    pypi_0    pypi
filelock                  3.8.2                    pypi_0    pypi
fiona                     1.8.22                   pypi_0    pypi
flake8                    6.0.0                    pypi_0    pypi
flasgger                  0.9.5                    pypi_0    pypi
flask                     2.2.2                    pypi_0    pypi
geojson                   2.5.0                    pypi_0    pypi
geomet                    1.0.0                    pypi_0    pypi
geopandas                 0.12.1                   pypi_0    pypi
google-api-core           2.11.0                   pypi_0    pypi
google-auth               2.15.0                   pypi_0    pypi
google-cloud-core         2.3.2                    pypi_0    pypi
google-cloud-storage      2.7.0                    pypi_0    pypi
google-crc32c             1.5.0                    pypi_0    pypi
google-resumable-media    2.4.0                    pypi_0    pypi
googleapis-common-protos  1.57.0                   pypi_0    pypi
html2text                 2020.1.16                pypi_0    pypi
identify                  2.5.9                    pypi_0    pypi
idna                      3.4                      pypi_0    pypi
importlib-metadata        5.1.0                    pypi_0    pypi
iniconfig                 1.1.1                    pypi_0    pypi
isodate                   0.6.1                    pypi_0    pypi
isort                     5.10.1                   pypi_0    pypi
itsdangerous              2.1.2                    pypi_0    pypi
jaraco-classes            3.2.3                    pypi_0    pypi
jinja2                    3.1.2                    pypi_0    pypi
jmespath                  1.0.1                    pypi_0    pypi
jsonpath-ng               1.5.3                    pypi_0    pypi
jsonschema                4.17.3                   pypi_0    pypi
keyring                   23.11.0                  pypi_0    pypi
lazy-object-proxy         1.8.0                    pypi_0    pypi
libffi                    3.4.2                h8ffe710_5    conda-forge
libsqlite                 3.40.0               hcfcfb64_0    conda-forge
libzlib                   1.2.13               hcfcfb64_4    conda-forge
lxml                      4.9.1                    pypi_0    pypi
mako                      1.2.4                    pypi_0    pypi
markdown                  3.4.1                    pypi_0    pypi
markupsafe                2.1.1                    pypi_0    pypi
mccabe                    0.7.0                    pypi_0    pypi
mistune                   2.0.4                    pypi_0    pypi
more-itertools            9.0.0                    pypi_0    pypi
msrest                    0.7.1                    pypi_0    pypi
munch                     2.5.0                    pypi_0    pypi
mypy-extensions           0.4.3                    pypi_0    pypi
nodeenv                   1.7.0                    pypi_0    pypi
numpy                     1.23.5                   pypi_0    pypi
oauthlib                  3.2.2                    pypi_0    pypi
openssl                   3.0.7                hcfcfb64_1    conda-forge
owslib                    0.27.2                   pypi_0    pypi
packaging                 22.0                     pypi_0    pypi
pandas                    1.5.2                    pypi_0    pypi
pathspec                  0.10.2                   pypi_0    pypi
pdoc3                     0.10.0                   pypi_0    pypi
pip                       22.3.1             pyhd8ed1ab_0    conda-forge
pkginfo                   1.9.2                    pypi_0    pypi
platformdirs              2.6.0                    pypi_0    pypi
pluggy                    1.0.0                    pypi_0    pypi
ply                       3.11                     pypi_0    pypi
pre-commit                2.20.0                   pypi_0    pypi
protobuf                  4.21.11                  pypi_0    pypi
psutil                    5.9.4                    pypi_0    pypi
pyasn1                    0.4.8                    pypi_0    pypi
pyasn1-modules            0.2.8                    pypi_0    pypi
pycodestyle               2.10.0                   pypi_0    pypi
pycparser                 2.21                     pypi_0    pypi
pyflakes                  3.0.1                    pypi_0    pypi
pygments                  2.13.0                   pypi_0    pypi
pylint                    2.15.8                   pypi_0    pypi
pyproj                    3.4.0                    pypi_0    pypi
pyproject-api             1.2.1                    pypi_0    pypi
pyrsistent                0.19.2                   pypi_0    pypi
pyshp                     2.3.1                    pypi_0    pypi
pystac                    1.6.1                    pypi_0    pypi
pytest                    7.2.0                    pypi_0    pypi
pytest-cov                4.0.0                    pypi_0    pypi
python                    3.9.15          h4de0772_0_cpython    conda-forge
python-dateutil           2.8.2                    pypi_0    pypi
pytz                      2022.6                   pypi_0    pypi
pywin32-ctypes            0.2.0                    pypi_0    pypi
pyyaml                    6.0                      pypi_0    pypi
readme-renderer           37.3                     pypi_0    pypi
requests                  2.28.1                   pypi_0    pypi
requests-futures          1.0.0                    pypi_0    pypi
requests-oauthlib         1.3.1                    pypi_0    pypi
requests-toolbelt         0.10.1                   pypi_0    pypi
rfc3986                   2.0.0                    pypi_0    pypi
rich                      12.6.0                   pypi_0    pypi
rsa                       4.9                      pypi_0    pypi
s3transfer                0.6.0                    pypi_0    pypi
sentinelsat               1.1.1                    pypi_0    pypi
sertit                    1.20.3                   pypi_0    pypi
setuptools                65.5.1             pyhd8ed1ab_0    conda-forge
shapely                   1.8.5.post1              pypi_0    pypi
six                       1.16.0                   pypi_0    pypi
tk                        8.6.12               h8ffe710_0    conda-forge
toml                      0.10.2                   pypi_0    pypi
tomli                     2.0.1                    pypi_0    pypi
tomlkit                   0.11.6                   pypi_0    pypi
tox                       4.0.2                    pypi_0    pypi
tqdm                      4.64.1                   pypi_0    pypi
twine                     4.0.2                    pypi_0    pypi
typing-extensions         4.4.0                    pypi_0    pypi
tzdata                    2022g                h191b570_0    conda-forge
ucrt                      10.0.22621.0         h57928b3_0    conda-forge
urllib3                   1.26.13                  pypi_0    pypi
usgs                      0.3.4                    pypi_0    pypi
vc                        14.3                 h3d8a991_9    conda-forge
virtualenv                20.17.1                  pypi_0    pypi
vs2015_runtime            14.32.31332          h1d6e394_9    conda-forge
webencodings              0.5.1                    pypi_0    pypi
werkzeug                  2.2.2                    pypi_0    pypi
wheel                     0.38.4             pyhd8ed1ab_0    conda-forge
whoosh                    2.7.4                    pypi_0    pypi
wrapt                     1.14.1                   pypi_0    pypi
xz                        5.2.6                h8d14728_0    conda-forge
zipp                      3.11.0                   pypi_0    pypi
remi-braun commented 1 year ago

I tried to add one lib at a time, and it seems to fail when adding eodag-sentinelsat but I don't know why 😰

remi-braun commented 1 year ago

Hey !

Do you have news on this topic ? 😅 (with eodag-sentinelsat installed, set_preferred_provider seems not to work properly)

sbrunato commented 1 year ago

Hi @remi-braun

Do you have news on this topic ?

I tried again to reproduce the issue, but did not manage... Do you have this issue in a new and clean virtual environment with only eodag-sentinelsat (and its dependencies) installed?

remi-braun commented 1 year ago

This is really weird, it's happening to me on several environments both Linux (in Docker) and Windows.

I did the job previously to start from a new environment (see the message send on the 8/12): The environment with only eodag works, but breaks when adding eodag-sentinelsat

This works:

conda env create -f environment.yml

name: eodag_test
channels:
  - conda-forge
dependencies:
  - python=3.9
  - eodag
    print([(p, c.priority) for p, c in dag.providers_config.items()])
    dag.set_preferred_provider("capella")
    print([(p, c.priority) for p, c in dag.providers_config.items()])
    print(dag.get_preferred_provider())
[('theia', 0), ('peps', 1), ('creodias', 0), ('mundi', 0), ('onda', 0), ('astraea_eod', 0), ('usgs_satapi_aws', 0), ('earth_search', 0), ('earth_search_cog', 0), ('earth_search_gcs', 0), ('ecmwf', 0), ('cop_ads', 0), ('cop_cds', 0), ('sara', 0), ('cop_dataspace', 0), ('planetary_computer', 0), ('capella', 0)]
[('theia', 0), ('peps', 1), ('creodias', 0), ('mundi', 0), ('onda', 0), ('astraea_eod', 0), ('usgs_satapi_aws', 0), ('earth_search', 0), ('earth_search_cog', 0), ('earth_search_gcs', 0), ('ecmwf', 0), ('cop_ads', 0), ('cop_cds', 0), ('sara', 0), ('cop_dataspace', 0), ('planetary_computer', 0), ('capella', 2)]
('capella', 2)

This doesn't:

conda env update -f environment.yml

name: eodag_test
channels:
  - conda-forge
dependencies:
  - python=3.9
  - packaging
  - eodag
  - pip
  - pip:
      - eodag-sentinelsat
    print([(p, c.priority) for p, c in dag.providers_config.items()])
    dag.set_preferred_provider("capella")
    print([(p, c.priority) for p, c in dag.providers_config.items()])
    print(dag.get_preferred_provider())
[('scihub', 0), ('theia', 0), ('peps', 1), ('creodias', 0), ('mundi', 0), ('onda', 0), ('astraea_eod', 0), ('usgs_satapi_aws', 0), ('earth_search', 0), ('earth_search_cog', 0), ('earth_search_gcs', 0), ('ecmwf', 0), ('cop_ads', 0), ('cop_cds', 0), ('sara', 0), ('cop_dataspace', 0), ('planetary_computer', 0), ('capella', 0)]
[('scihub', 0), ('theia', 0), ('peps', 1), ('creodias', 0), ('mundi', 0), ('onda', 0), ('astraea_eod', 0), ('usgs_satapi_aws', 0), ('earth_search', 0), ('earth_search_cog', 0), ('earth_search_gcs', 0), ('ecmwf', 0), ('cop_ads', 0), ('cop_cds', 0), ('sara', 0), ('cop_dataspace', 0), ('planetary_computer', 0), ('capella', 0)]
('peps', 1)