search

enram / data-repository

Data quality assessment
https://enram.github.io/data-repository/
MIT License
3 stars 1 forks source link

Extract and store settings data #7

Closed peterdesmet closed 8 years ago

peterdesmet commented 8 years ago

Do we really need to build this? And if so, how do we keep it simple?

@leijnse from what I understood in the meeting, it would be nice to extract and record the radar + bird algorithm settings used to generate the data. That data can change over time and is recorded in the HDF5 format. That is potentially challenging, so: how necessary is this? Is this information you can easily retrieve in another way? How many users are interested in this information?

adokter commented 8 years ago

Scientists will need this information, because we are and will be for a considerable time in a development phase, where there is no such thing as the perfect processing tool yet. Therefore I think for the years to come we need the information on algorithm and radar settings. We will likely end up in a situation where the same radar data will be processed in different ways, or that data will be re-run with updated algorithm settings. Information is of interest to ENRAM scientists only, so not a huge user group. But I do think ENRAM scientist will be the core user group in the coming years.

peterdesmet commented 8 years ago

Ok, if you rerun, do we want to keep data from an old run or only the most recent version? And if you want to keep it, only the archived data (h5) or versioned in the database as well. It could increase the complexity a lot.

adokter commented 8 years ago

I would say: keep it simple for now and replace old data with reprocessed data.

peterdesmet commented 8 years ago

Great!