GO common data retrieval tool

meiertgrootes commented 3 years ago

as team atlas, depending on the outcome of #141 we want to develop a common data retrieval tool for EO data. this will facilitate our project work and be a general purpose tool of use to the broader community.

[ ] meet requirements in #139
[ ] meet requirements from Radar Veg and Eratosthenes

fnattino commented 3 years ago

Open-source tools for downloading assets and metadata are sat-search and sat-stac. They can be interfaced with geopandas for geospatial analysis. An example of how these pieces of software can be exploited to retrieve imagery and store metadata for subsequent queries is illustrated in this notebook.

fnattino commented 3 years ago

STAC-compliant catalogs that can be queried by sat-search should be available for:

Landsat 8
Sentinel-2 (processed at 1C and 2A levels) For 1C-level Sentinel-2 data, the assets are not freely available so one needs to redirect download to the Google cloud storage.

Catalog queries include fields related to the area-of-interest (bounding box, tile), periods, cloud covers. Subsequent queries on the locally-stored catalog are possible via geopandas.

The only requirement from #139 that would need to be verified is the availability of Sentinel-1 data.

fnattino commented 3 years ago

Another relevant aspect for the tool is to be able to talk to the SURF dCache storage system. pystac allows to implement specific reader/writer methods to create a STAC catalog, so that one could implement specific readers/writers using the dCache API.

fnattino commented 3 years ago

I have created a first version of a tool implementing the features described above, see here. A notebook illustrate the main functionalities.

rogerkuou commented 3 years ago

@fnattino Hi Francesco, this is really cool! I play it quite a well with different downloading queries. I was using our eecolidar webdav to do the test since I do not have access (yet?) to the Eratosthenes one. Everything works as instructed. Just one minor comment:

Two dependencies (only) used in the Notebook: satsearch and matplotlib are not contained in the module. Maybe we can mention in the Notebook that people should install them?

For the purpose of talking to dCache I think it is quite sufficient now. Let's see in the future what kinds new requirements can we get from the real application.

fnattino commented 3 years ago

Thanks @rogerkuou, added instructions on how to install these to the notebook!

NLeSC / team-atlas

GO common data retrieval tool #142