Closed rabernat closed 2 years ago
Thanks for the very detailed review of these packages (intake and pooch), and thank you for your interest in CliMetLab.
Regarding intake, we do plan to have interoperability between climetlab and intake
We have in mind a CliMetLab plugin integrating an intake dataset into climetlab.load_dataset()
. And also, having an intake data set integrating any climetlab dataset into an intake catalogue. intake.open_catalog("climetlab")
. The final API for the end-user could be somehow similar.
Discussion on how to write these plugins would be a good idea. Additionally, discussions on the differences between publishing a dataset with climetlab vs intake could clarify both approaches.
Regarding using pooch for caching
Replacing the caching mechanism of CliMetLab with another package (such as pooch) is indeed always possible. We currently use our own code for caching because we are still exploring different caching mechanisms (invalidation policies, and various ways to share cache between different users, or using different backends, such as S3, partial caching of the data, I am not sure how pooch handle partial download using HTTP PARTS, or using different formats). One starting point would be to integrate pooch into climetlab as a source plugin, allowing comparison and benchmarks on real use cases. This could be a nice way to assess if switching to pooch is the way to go and to let the users choose.
CliMetLab is domain-specific
Generally speaking, CliMetLab is a domain-specific package (climate and meteorology) relying heavily on more general-purpose packages such as those in the Pangeo stack. We envision CliMetLab as a thin layer around these, providing python functionalities for the scientist/engineer/data-scientist in these domains, and while we need to develop some code ourselves, we are always happy to mutualise the effort as much as possible.
Closing for now, anybody should feel free to reopen (or open another specific issue) if there are relevant packages that should be considered.
Greetings and thanks for taking the time and effort to create open source software for the meteorology / climate communities. I sincerely commend this effort! π I am particularly in inspired by the stated goal of the climetlab package:
I think it's safe to say that very many people in the software community share this goal. It has been a particular focus of the Pangeo project for many years. However, it is also a difficult goal, given the vast diversity of different data providers, catalogs, and data formats that we encounter in the wild. Therefore, I believe that collaboration is essential for achieving this goal.
In this spirit, I would like to invite the climetlab developers to collaborate with the Pangeo project and related python packages. There may be some ways we can combine efforts to deliver more effective software with a lower overall maintenance burden.
A primary possible area of collaboration would be to reduce duplication in functionality across the ecosystem. Reducing duplication is good because it:
In that spirit, here are some existing packages that offer functionality similar to climetlab.
Intake
https://github.com/intake/intake
Intake is the main tool we currently use in Pangeo to provide "convenient" data access in Pangeo (usually via the intake-xarray plugin). Intake has similar goals but a different architecture to climetlab's data source feature. With intake, one creates a catalog yaml file (example), which specifies the data sources and options for loading the data.
For example, to load the grib example file from the climetlab docs, I would write a yaml file like this
catalog.yaml
```yaml plugins: source: - module: intake_xarray sources: sample_grib_data: description: Sample GRIB file driver: netcdf args: urlpath: 'simplecache::https://github.com/ecmwf/climetlab/raw/main/docs/examples/test.grib' xarray_kwargs: engine: cfgrib ```and then open it as an xarray dataset
Some other intake features that may be useful to this project:
In terms of architecture:
My unsolicited opinion is that the climetlab approach--writing python code for each new dataset--is not scalable to the volume and diversity of meteorology / climate datasets that exist in the world. Going down that path means effectively writing code to describe the structure / layout of every dataset in the world. Leveraging established community standards for data catalogs, or allowing users to very easily create their own catalogs, seems to me as the only viable path forward.
So a specific suggestion would be to refactor climetlab to use intake interally, rather than duplicating much of intake's functionality for data downloading, caching, loading, templating, etc. This would allow you to delete lots of code π and lower your maintenance burden. New functionality in terms of data loaders could be pursued upstream as intake plugins.
Pooch
https://github.com/fatiando/pooch
Pooch has a much narrower scope than intake. It is extremely stable and solid if what you want to do is download remote files to a local computer. It supports many of the same protocols as climetlab, and some other ones, such as Zenodo-based DOI downloads.
Here is how pooch would be used to download the climetlab test grib data
Here the opportunity for climetlab is to leverage Pooch's downloading / caching capabilities, rather than duplicating similar capabilities internally.
It may be possible that you looked at these packages and decided that they had feature gaps or bugs that made them unusable for your project. If so, an alternate path could be to work with the upstream libraries to resolve these gaps and bugs, instead of duplicating their functionality. Part of my goal in opening this issue is to state clearly that the broader scientific python community welcomes your involvement in and contributions to upstream packages. We would benefit greatly from your expertise.
Thank you for taking the time to read my long issue. I reiterate my commendation of your efforts to provide open-source software to the community and my alignment with your vision regarding the goals of this package. I welcome a discussion on these topics or any other ways you think we could be collaborating. π