added yaml2netcdf.py

[d70-t]

This should solve #5.

[gdeboer2]

Saw the review request, but I'm not quite sure what to offer here, since I don't use YAML files or Python. I'd say that if the translation that this provides works for you it is probably ok and I'll look for the segment information in the NetCDF files.

[d70-t]

@gdeboer2 I don't plan to use the netCDF-files, but my Impression has been that you are interested in using these and I might be the one who should write the translation script. The question would be if the resulting files provide the information you need and are formatted in a sensible way.

[gdeboer2]

Understood, but I don’t know what the resulting files will produce, since I don’t program in Python. Do you have an example NetCDF file that I can evaluate?

On Aug 12, 2020, at 9:37 AM, Tobias Kölling notifications@github.com wrote:

@gdeboer2 https://github.com/gdeboer2 I don't plan to use the netCDF-files, but my Impression has been that you are interested in using these and I might be the one who should write the translation script. The question would be if the resulting files provide the information you need and are formatted in a sensible way.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/eurec4a/flight-phase-separation/pull/11#issuecomment-672949322, or unsubscribe https://github.com/notifications/unsubscribe-auth/AO3B45Q7CI7LO2EJIMFGHM3SAKZMTANCNFSM4P4S7JFQ.

[d70-t]

Hmm, I could create one as an example, but the script needs to get a timeline from somewhere. In the current implementation, that works by passing another netCDF-file to the script such that the output data is created for exactly the required time steps. As the timeline depends on how each user wants to analyse the data, I'd argue that running the script is part of the deal and if that doesn't work, the tools is not very useful. My assumption has been that while many people will not program in python, it should at least be possible for everyone to run scripts.

For reference, this is the ncdump output for my test:

$ ncdump -h test.nc 
netcdf test {
dimensions:
    time = 33219 ;
    kind = 5 ;
    segment = 4 ;
variables:
    double time(time) ;
        time:_FillValue = NaN ;
        time:units = "seconds since 2020-01-19 09:34:25.000004352" ;
        time:calendar = "proleptic_gregorian" ;
    string kind(kind) ;
    string segment(segment) ;
    byte part_of_kind(time, kind) ;
        part_of_kind:dtype = "bool" ;
    byte part_of_segment(time, segment) ;
        part_of_segment:dtype = "bool" ;
}

[gdeboer2]

Ok — I think that I’m understanding where this is going. Having said that, what I had envisioned is that the timeline would be provided with any data files produced for each platform. I may be in the minority, but my mode of operation is still download a data file and then process on my end. I was assuming that that file would have a flight state (or whatever it is to be called) variable in it at the same resolution as the rest of the variables for that file.

On Aug 12, 2020, at 9:53 AM, Tobias Kölling notifications@github.com wrote:

Hmm, I could create one as an example, but the script needs to get a timeline from somewhere. In the current implementation, that works by passing another netCDF-file to the script such that the output data is created for exactly the required time steps. As the timeline depends on how each user wants to analyse the data, I'd argue that running the script is part of the deal and if that doesn't work, the tools is not very useful. My assumption has been that while many people will not program in python, it should at least be possible for everyone to run scripts.

For reference, this is the ncdump output for my test:

$ ncdump -h test.nc netcdf test { dimensions: time = 33219 ; kind = 5 ; segment = 4 ; variables: double time(time) ; time:_FillValue = NaN ; time:units = "seconds since 2020-01-19 09:34:25.000004352" ; time:calendar = "proleptic_gregorian" ; string kind(kind) ; string segment(segment) ; byte part_of_kind(time, kind) ; part_of_kind:dtype = "bool" ; byte part_of_segment(time, segment) ; part_of_segment:dtype = "bool" ; } — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/eurec4a/flight-phase-separation/pull/11#issuecomment-672958638, or unsubscribe https://github.com/notifications/unsubscribe-auth/AO3B45SJNK5YBD3DWVVZP6LSAK3HVANCNFSM4P4S7JFQ.

[RobertPincus]

@d70-t Perhaps the netCDF files could be generated using the script as part of each release? This would lower the barrier to people like @gdeboer2 who don't want to set up a Python environment.

[d70-t]

I can only speak for HALO, but there are at least 3 or 4 different files for navigation data with different resolutions. Then there are instrument specific timelines which are sometimes different as well and then there might be some coarsened / regridded timestepping as currently discussed in the Workshop. So yes, we could generate these netCDF files automatically as part of a release or by other means, but before that, we'll have to come up with a sensible way of defining a common timeline.

[gdeboer2]

I guess that I’m not so concerned about the resolution issues, assuming that the version of the segment information provided is relatively high resolution. Re-gridding the flight state to anything finer than +/- 5 seconds should not be a big challenge, so offering this at one resolution that aligns with the navigation data should be enough. I was planning to provide this information at 5 or 10 Hz for RAAVEN. Anyone wanting finer than that should probably be ok with losing <0.1 s of data at the transitions!

On Aug 12, 2020, at 10:33 AM, Tobias Kölling notifications@github.com wrote:

I can only speak for HALO, but there are at least 3 or 4 different files for navigation data with different resolutions. Then there are instrument specific timelines which are sometimes different as well and then there might be some coarsened / regridded timestepping as currently discussed in the Workshop. So yes, we could generate these netCDF files automatically as part of a release or by other means, but before that, we'll have to come up with a sensible way of defining a common timeline.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/eurec4a/flight-phase-separation/pull/11#issuecomment-672981694, or unsubscribe https://github.com/notifications/unsubscribe-auth/AO3B45RUZJBP7YAJO4K27STSAK75VANCNFSM4P4S7JFQ.

[d70-t]

Maybe we could use the timeline data which is provided by get_navdata as a default. The resulting netcdf files could be zipped and attached to a release like with #9 or alternatively uploaded to Aeris automatically on release. Would that work for you?