Closed peterdesmet closed 8 years ago
The BALTRAD infrastructure is built for real-time applications, not for long-term storage. So we need our own data repository. KNMI can host longer-term data, so the choice should be KNMI or our own data storage.
Default directory tree will be radar/yyyy/mm/dd/hh/radar_yyyymmddhhmm.h5
, where each file contains one profile for one radar at one time instant only. Baltrad provides data at 15 min interval, so 4 files per directory
@adokter: if the h5 file contains data from one radar for a 15 minute interval, the /dd
directory will contain 4 * 24 (96) files, no?
@peterdesmet: sorry typo, also hour is a subfolder
We'll upload everything to an Amazon S3 bucket.
@peterdesmet do you want me to include code to create the bucket as well? We'll be currently running this in our sandbox environment, meaning the bucket can be gone at the end of the day.
@bartaelterman code to create the bucket: that would be useful.
@peterdesmet @bartaelterman
A different service worth considering is this EU-funded Dropbox-like service designed specifically for the exchange and storage of research data:
They offer 20GB of storage. Given that one bird profile is 5kB (zipped), it can store 4M profiles. A network of 100 radars produces 4*24*365*100=3.5M
profiles per year. So a limitation there, but we could inform whether these limits can be enlarged
Since we can easily setup things on S3, I would choose that for now. I would revisit EUDAT or any of the other possible repository once we have a functional data flow. Closing this issue: we now that a data repository is needed.
Do we really need to build this? And if so, how do we keep it simple?
It depends: storing the data on our own/LifeWatch servers gives us more control. Another option is to depend on the BALTRAD infrastructure for hosting, backing up and giving access to our data. @leijnse, what is advisable? A: Data repository is required
How do we organize our data?
To limit the scope, we kinda decided to not have a database with the raw data, but rather have researchers download the data as files. That means that the data should ideally be organized in such a way that you can easily download logical packages. E.g. by having file format (e.g. hdf5) high up in the hierarchy, you avoid that users have to download "duplicate" data (in different formats) when selecting a directory to download.
@leijnse @hvangasteren, will users be more interested to download a time period for all radars or a radar for all time periods?
Suggested structure:
/fileformat/yyyy/mm/dd/radar_id.hdf5
where each file contains 24h of data.