Jira-744 for beta release

[nfb2021]

progress so far:

• [x] restructure output netcdf files for intra-annual metrics
• [x] implement compression for output netcdf files
• [x] adapt plotting accordingly

new files:

• validator.validation.intra_annual_temp_windows.py: contains code to create intra-annual slices a.k.a. temporal sub-windows
• validator.validation.netcdf_transcription.py: contains logic to translate the original pytesmo output netCDF file into the new qa4sm one
• custom_intra_annual_windows_example.json: not required for default cases, those are defined in the globals.py. kept to demonstrate injecting custom sub-temporal windows for now (as this functionality is implemented in validator.validation.intra_annual_slicer.IntraAnnualSlicesDefault to add flexibility for users)

I tried to highlight all new code with #$$

Note:

Triple Collocation Metrics are deactivated for the beta release

This is due to two problems:

1. in the default case, the qa4sm_reader throws an error when plotting tcol-related graphics, but tcol metrics are calculated and contained in the output nc file
2. when using tsws, tcol metrics are not even calculated Thus, the test test_validation_tcol fails (for both the default case and when using tsws)

Tests

• except for test_validation_tcol , all test pass when using the default case
• when using tsws, all tests except __test_c3s_validation_upscaling, test_validation_default and test_validation_tcol__ fail
• there are no tests specially developed for tsws

Pytesmo Bug

Not all dataset combinations may be used to run a validation. In some cases, pytesmo throws a bug along the line: ValueError("Expected a DatetimeIndex, but got <class 'pandas.core.indexes.base.Index'>."). This issue is reproducible when running the test __test_c3s_validation_upscaling__ in default case

Plotting

As the output netcdf files were restructured to better accommodate the tsws by introducing a new dimension, the qa4sm-reader had to be adapted accordingly. Plotting will not work with the currently publicly available qa4sm-reader, not even when using the default case

Code Formatting

It may be, that some code is formatted differently than in the awst/qa4sm master branch. Sorry for that!

To Do:

• [ ] solve TCOL issues
• [ ] adapt existing tests
• [ ] create new tsws specific tests

[nfb2021]

@wpreimes The required qa4sm-reader code can be found in this draft PR

[sheenaze]

Also, there is a migration conflict. If you're not sure how to solve it, we can have a call and I can help you :)

[nfb2021]

@wpreimes The required qa4sm-reader code can be found in this draft PR

I closed the draft PR to the qa4sm-reader master branch and instead did a proper PR to the beta-release branch