xcube-dev / xcube

xcube is a Python package for generating and exploiting data cubes powered by xarray, dask, and zarr.
https://xcube.readthedocs.io/
MIT License
195 stars 18 forks source link

Updated xcube server's dataset configuration extraction methodology #1048

Closed konstntokas closed 2 months ago

konstntokas commented 2 months ago

The xcube server's dataset configuration extraction methodology has been updated. When the data resource ID is provided in the Path field, xcube will attempt to access the dataset using the given ID. If wildcard patterns are used, the server will crawl through the data store to find matching data IDs. This process may result in a long setup time if the data store contains numerous data IDs. A UserWarning will be issued for the "stac" data store.

Checklist:

konstntokas commented 2 months ago

I tested it with the stac configuration example (snippet shown below), I showed last week to @gunbra32 and @forman, and the setup of the server runs without any noticeable delay.

DataStores:
  - Identifier: stac
    StoreId: stac
    StoreParams:
      url: https://s3.eu-central-1.wasabisys.com/stac/odse/catalog.json
    Datasets:
      - Identifier: Example
        Path: lcv_blue_landsat.glad.ard/lcv_blue_landsat.glad.ard_1999.12.02..2000.03.20/lcv_blue_landsat.glad.ard_1999.12.02..2000.03.20.json
        Style: S2L2A