search

astropy / astropy-data

The source for the astropy data repository (although the primary server is not on github)
11 stars 55 forks source link

Use Intake #87

Open martindurant opened 4 years ago

martindurant commented 4 years ago

A light suggestion that people here may want to consider expressing this collection of data-sets using Intake, which is a cataloging standard allowing you to describe the metadata and way-to-load for data in a single browsable, searchable spec, e.g., a YAML file. This is, for instance, what the pangeo collaboration does. Most of their data is in zarr or other xarray-compatible formats, so you would also need intake-astro to enable loading FITS data from remote files, with local caching if desired, and lazy-loading with Dask.

pllim commented 3 years ago

Thank you for the suggestion! Unfortunately, the current maintainers are stretched pretty thin to investigate this option. If you are interested to implement a proof-of-concept for this new solution, we would be happy to review your PR and continue the discussion there.

pllim commented 1 year ago

Looks like scipy uses pooch now but even that seems a little heavyweight for our purpose here. 🤷