search

ClimateIndicator / forcing-timeseries

MIT License
14 stars 2 forks source link

Consider applying for the AWS open data program #17

Open hellais opened 3 months ago

hellais commented 3 months ago

There are several large files which are currently hosted on a university site (eg. https://www.geo.vu.nl/~gwerf/GFED/GFED4/GFED4.1s_{year}.hdf5) which could benefit from having long term fast archival and retrieval.

I believe they could and should be eligible for the AWS open data program which would mean they would be stored forever in s3 and would be retrieved much faster than from the geo.vu.nl university server.

Since our data (where our means OONI) is part of this program (https://registry.opendata.aws/ooni/), I would be happy to make introductions if it's helpful and in any case make a case for why such important environmental data should be archived by them.

chrisroadmap commented 3 months ago

I could check with the dataset creator. As you see in #18, we are not in control of many of the datasets that we make use of, so it's possible that we would miss updates. Could AWS automatically mirror datasets that are hosted on vu.nl?