archive option of baltrad images

[stijnvanhoey]

The vp data before 1 november 2017 are actually provided with a different version of the algorithm. The poposal is to archive those images in the S3 bucket, but not provide them in the online interface.

@peterdesmet, should we provide an archive subfolder? Or maybe create a different S3 bucket?

[peterdesmet]

@stijnvanhoey, by images, do you mean vp data (h5)? @adokter: are the vp files generated before november of any use? i.e. Is the data quality sufficient to keep them around?

[peterdesmet]

@adokter, sorry, only saw email now. For how long do you want to keep the old data around? @stijnvanhoey, an archive subfolder seems sufficient, would not create separate bucket.

[stijnvanhoey]

@peterdesmet indeed, vp data ;-)

[adokter]

@peterdesmet Some of the old data is ok, but most of it is not. My reasoning was to have the possibility to keep the old data that was ok, so we could make it available again later. But it's not worth spending much time on this anyhow.

I will check with Liesbeth to prepare a list of radars that can be deleted permanently

[plieper]

In may opinion: you can keep NL, SE, and FI, the vp's of the other countries don't make sense. Also, Günther upgraded vol2bird on 21/11, so everything from 22/11 on should be good data.

[peterdesmet]

Summary

To keep:

• all: 2017-11-22 onwards
• fi: all
• nl: all
• se: all
• be: 2016-09-19 2016-10-09 (flyway)
• bg: 2016-09-19 2016-10-09 (flyway)
• ct: 2016-09-19 2016-10-09 (flyway)
• cz: 2016-09-19 2016-10-09 (flyway)
• de: 2016-09-09 2016-10-09 (flyway)
• fi: 2016-09-19 2016-10-09 (flyway)
• fr: 2016-09-19 2016-10-09 (flyway)
• nl: 2016-09-19 2016-10-09 (flyway)
• pl: 2016-09-19 2016-10-09 (flyway)
• pt: 2016-08-01 2016-11-30 (flyway)
• se: 2016-09-19 2016-10-09 (flyway)
• si: 2016-09-19 2016-10-09 (flyway)

To remove:

• [x] be: 2016-11 - 2017-11-21
• [x] bg: 2016-11 - 2017-11-21 (= nothing to remove)
• [x] ch: 2016-11 - 2017-11-21 (= remove all)
• [x] ct: 2016-11 - 2017-11-21 (= nothing to remove)
• [x] cz: 2016-11 - 2017-11-21
• [x] de: 2016-11 - 2017-11-21
• [x] dk: 2016-11 - 2017-11-21 (= no flyway data)
• [x] ee: 2016-11 - 2017-11-21 (= no flyway data)
• [x] es: 2016-11 - 2017-11-21 (= no flyway data)
• [ ] fi: NOTHING TO REMOVE (all considered ok by @plieper)
• [x] fr: 2016-11 - 2017-11-21
• [x] hr: 2016-11 - 2017-11-21 (= no flyway)
• [ ] nl: NOTHING TO REMOVE (all considered ok by @plieper)
• [x] no: 2016-11 - 2017-11-21 (= no flyway)
• [x] pl: 2016-11 - 2017-11-21
• [ ] pt: NOTHING TO REMOVE (= flyway data only, but extends beyond 2016-11!)
• [ ] se: NOTHING TO REMOVE (all considered ok by @plieper)
• [x] si: 2016-11 - 2017-11-21
• [x] sk: 2016-11 - 2017-11-21 (= no flyway)

[stijnvanhoey]

Removed the useless data sets according to specification above. Coverage is adapted as well, currently executing a rerun of all zip-files to have these up to date as well.

[peterdesmet]

Nice, can you ping us if the zip files are OK? I know @plieper was interested in those.

[stijnvanhoey]

@plieper @peterdesmet the 'historical' zip files are updated as well and when new data comes in, these will be updated automatically.