Get some sample h5 flyway data

[peterdesmet]

Because of bug #4 and the low quality of the data in the data repository, we should test the pipeline further with some sample h5 flyway data. Doesn't have to be much (a couple of hours for 2-3 radars). That should also allow to test if the format of those data are the same as what is deposited in the repository.

[CeciliaNilsson709]

I sent you a few hours of data from the Swedish radars, let me know if you need something else!

[peterdesmet]

Thanks! I couple of questions:

1. I notice a difference in how the data are organized:

data you've send me:               year/month/day/hour/minute/  files for multiple radars
data repository:     radar/country/year/month/day/hour/         minute files for a single radar

bioRad contains some new code that can load data nicely if it is organized according to the data repository. Is all the flyway data you have organized in the first way and is there a specific reason for that?

2. The minute directories also contain an empty /merged directory. Should I just ignore that?

3. Is it OK if I put the data sample you gave me in this repository (which is public)?

[CeciliaNilsson709]

1. Yes, all data I have (from Liesbeth) is organized in the way I sent it to you. @plieper, is there are reason for the difference? Edit: It works for me to load it into bioRad with this organization...
2. Yes, ignore it
3. I am not sure about that. We got it before the ENRAM-OPERA license, and I think it was a bit of a special case. So to be safe probably not? (but if you really want to, I can look into it of course).

[peterdesmet]

@CeciliaNilsson709 Regarding 3: OK, there is not really a strong need to include that sample data in the repository right now, so I'm keeping it private. At some point we should get that flyway data in the ENRAM data repository as open data though. Not sure who we should ask for permission: let's figure that out later. :relieved:

@plieper: I'm surprised that the function bioRad::retrieve_vp_paths() seems to work with the way the flyway data are organized (i.e. with year first and radar last): that seems like magic :crystal_ball: :smile: but it's awesome!

[plieper]

credits for that go to @stijnvanhoey! :-)

[peterdesmet]

In that case: 👏 🎉 @stijnvanhoey 🙌 👍

[stijnvanhoey]

Thanks! No magic involved :smile:

As long the name convention of the file names itself (ccrrr_vp_yyyymmddhhmmss.h5 with cc=country, rrr = radar) is not changed, the download and read code won't break. File names are collected recursively in all subfolders and check is done on the resulting list of file names.

[peterdesmet]

A magician should never reveal its tricks 🔮 ... but I guessed something like that. Really useful.

[plieper]

@stijnvanhoey During fixing a recent merging bug we had in our merging script, we changed the file name to ccrrr_vp_yyyymmddhhmmss_0x. with . a number, eg. nldbl_vp_20160919T2000Z_0x7.h5. Will this have any influnce on bioRad::retrieve_vp_paths()?

This is for locally processed data, so not (yet) the vp's coming from bioRad.

[stijnvanhoey]

@peterdesmet some :cake: as reward will suffice...

@plieper With respect to the retrieve_vp_paths function, adding additional information at the end shouldn't break the name matching. All of the alternatives will be selected using the name matching: nldbl_vp_20160919T2000Z_0x7.h5, nldbl_vp_20160919T2000Z_0x8.h5,... Actually, the matching in done till day-level: the relevant section of the file name is nldbl_vp_20160919, whatever comes after it in the file name does not really matter.

[plieper]

@stijnvanhoey Ok, very good :-).

[peterdesmet]

🍰🍰 When the remaining issue is resolved 😌

[stijnvanhoey]

Is that a feature request or a bug? :wink: I'll fix that next week!