MetOffice / XBTs_classification

Project for the classification of eXpendable Bathy Thermographs
BSD 3-Clause "New" or "Revised" License
4 stars 2 forks source link

create intake catalog to abstract data locations #5

Open stevehadd opened 4 years ago

stevehadd commented 4 years ago

Currently we are hard coding the data file locations. It would be nice to create a basic intake catalog which can be used to access the data. Different catalogs with the correct paths can then be created for different processing platforms (primarily AWS, azure/pangeo and MO spice).

stevehadd commented 3 years ago

After chatting to Theo:

stevehadd commented 3 years ago

XBT data is already available through a public AWS bucket from NOA: https://noaa-wod-pds.s3.amazonaws.com/index.html

stevehadd commented 3 years ago

More useful intake information: https://intake.readthedocs.io/en/latest/catalog.html

stevehadd commented 3 years ago

We should probably include a way to generate the catalog files from a template, to accomodate future changes.