For a typical business, a data warehouse might help business analysts answer questions like "how many sales did we make last week?" via a query to an API or database. Our business is science data, so we may want to ask "How many MB of data did we ingest last week?" or "what files are available for dataset X between dates Y and Z?" or "Where was dataset X migrated to/from and when?" A data warehouse would be the source of truth for information about our dataset inventory.
We have a "data warehouse" system for ECS data in the form of CMR, but the rest of our data is managed solely as items on disk, and it's often not predictable where on disk that data will be or how to find the data you're interested in. Was it migrated to a new datapool recently? How do we determine the date a particular file corresponds with (we currently have to know or discover ourselves "where in the filename is the time?", "what format, e.g. YYYYMMDD, or YYYYDOY, or something else?", "Where are the gaps in coverage?")
It would be useful to provide a service that enables users to:
Create new records when adding new data files to a dataset
Update records when changing files in a dataset, e.g. migrating from one datapool to another.
Query for records in a dataset by date/time/other dimensions
Perform analysis and generate reports on stored datasets, e.g. dataset rate of growth, gaps in coverage, etc.
If we ran a tool like minio in front of all of our datapools, a data warehouse could return S3 URLs for datasets and access could be done over the s3 protocol instead of requiring disk mounts. This would make the transition to the cloud more transparent for our apps.
For a typical business, a data warehouse might help business analysts answer questions like "how many sales did we make last week?" via a query to an API or database. Our business is science data, so we may want to ask "How many MB of data did we ingest last week?" or "what files are available for dataset X between dates Y and Z?" or "Where was dataset X migrated to/from and when?" A data warehouse would be the source of truth for information about our dataset inventory.
We have a "data warehouse" system for ECS data in the form of CMR, but the rest of our data is managed solely as items on disk, and it's often not predictable where on disk that data will be or how to find the data you're interested in. Was it migrated to a new datapool recently? How do we determine the date a particular file corresponds with (we currently have to know or discover ourselves "where in the filename is the time?", "what format, e.g. YYYYMMDD, or YYYYDOY, or something else?", "Where are the gaps in coverage?")
It would be useful to provide a service that enables users to:
If we ran a tool like
minioin front of all of our datapools, a data warehouse could return S3 URLs for datasets and access could be done over the s3 protocol instead of requiring disk mounts. This would make the transition to the cloud more transparent for our apps.