search

noi-techpark / bdp-commons

GNU Affero General Public License v3.0
2 stars 12 forks source link

meteorology-eurac: Measurements go way back to year 1950, should we integrate all of them in the ODH? #547

Open Piiit opened 2 years ago

Piiit commented 2 years ago

The measurements go way back to year 1950. Do I need to sync such old data or is there a threshold from which year to sync data?

Piiit commented 2 years ago

@rcavaliere @sseppi Do you have any suggestion here?

rcavaliere commented 2 years ago

@Piiit the value here is in the historical data for climatological reasons. Therefore we should integrate all the history. Can we manage this or is this too much?

Piiit commented 2 years ago

@rcavaliere We need to see, @SaimonasFOS could make a test to see how big for example one month or year of data is, and then we calculate if it is feasible or not...

SaimonasFOS commented 2 years ago

@Piiit @rcavaliere For one station there are 365 rows in a year. Each row has 4 measurements: (maximum, minimum and mean temperature and total precipitation). There are around 350 stations in Eurac dataset. However, not every station has data from way back to 1950. From my testing, if I filter one station but do not filter date, then the amount of rows in response varies between 2000 and 25000 and the response size can be up to 3 MB (querying all measurements from all stations at once is not feasible, there must be either date filter or station filter). So, assuming the average amount of rows is 13500 for each station: 13500 4 measurements per row 350 stations = 18 900 000 measurements.

Piiit commented 1 year ago

@rcavaliere @SaimonasFOS @sseppi

Simon (@dulvui) will take this task from now on...