Closed Zeitsperre closed 2 years ago
This needs to be integrated with Mourad's plan to host everything on THREDDS. It also has to be compatible with future search interface and ESGF storage. There is also a CEDA project looking at this. This also needs to be discussed with CRIM. This is high priority, but that should not mean rushing it.
The PAVICS crawler essentially does exactly this. It reads all facets and throws them into a Apache Spark (NoSQL) database. I agree that there are a lot of approaches for doing this and I like CEDA's approach. In this instance, I'm thinking of something for small scales (e.g. a few hundred/thousand files).
As this is simply a generic project, having it rely on an existing THREDDS server or a MySQL/PostgreSQL infrastructure is a bit overkill. SQLite3 would be good given the limited scope.
But then we'll have to maintain two crawlers, two databases, two clients, ... no ? Unless I misunderstand where this project falls in our operationalization plan?
From a project management perspective, who's specifying the requirements for this ? What are the deliverables exactly? Where are the use cases ? Has anyone done a lit. review ? For example https://esgf.llnl.gov/esgf-media/2018-F2F/2018-12-06/ESGF-F2F-2018-PYESSV-Greenslade.pdf https://github.com/ES-DOC/pyessv
I had done some looking around, spent a lot of time going through PyOuranos, and figured a reimplementation was needed. The requirements are written down in my notes so nothing online yet. This project (from my perspective) loosely has the following goals (based on some of the design goals of PyOuranos):
This project is something I could see members of SSC using and possibly others contributing to. I had done a bit of looking around but I never came across PyESSV. This might remove a lot of the code needed for handling ESGF metadata.
We share the same goals, but this is meant to be a production service. It needs to be engineered, and this implies consultations with experts, a review of existing software, an understanding of the roadmap of other interconnected projects, and I don't think we have that.
Also, from my perspective, whatever we come up with has to be compatible with what the ESGF is cooking up at the moment since we want a unified catalog of local and remote files.
Following up on our conversation from a few weeks ago, there appear to be a few approaches to handling this issue in play right now:
Where I see Miranda fitting in is in performing the following:
The local needs (i.e.: Adding new data to our internal servers and providing a common, controlled-ish vocabulary for data outside the ESGF and ECMWF ecosystem) are what this library is focusing on for the moment. There's a very real scenario that these validation tools will be refactored elsewhere, possibly xscen. To be determined.
For now, the work towards this goal is present in #24, in the validators
, decoders
and cv
modules. Once I can wrap my head around https://github.com/Ouranosinc/pavics-vdb/pull/46, I'll have a better idea of what to focus on moving forward.
Thanks again for your analysis!
The machinery needed for performing this in Miranda in now in the main branch. Closed with #24
Using either the NetCDF4 or Xarray libraries, create a means of scraping metadata from files based on the following approaches:
Examining the filename and
Path
parts.Examining the NetCDF4 file Global Attributes.
Creating a facet dictionary for each file examined (
{Group: {FilePath: Facets}, {FilePath: Facets}, ... }
)Allow for identification and querying based on a given facet.
Possible approach: Creation and population of a SQLite3 DB file?