Validate 40 day 5km global resolution run

[m214089]

Problem (short description, max 150 words) We have a run made with 5km global resolution and want to validate the quality with ERA5 data.

Proposed solution (short description, max 200 words)

Check the first 10 days in their forecast skills. Look for properties in the remaining 30 days.

Data (what data do you need? what format?) ERA5 full resolution, best grib2.

Team

• Kornblueh Luis - role: data scientist
• Müller Ralf - role: tools engineer
• Schulzweida Uwe - role: tools engineer)

Is your team incomplete and you need extra help? If so, please list below the roles needed:

[m214089]

For working with a large data set the computer setup was not optimal as we couldn't transfer data between the local HPC and large disks and the local systems, but had to transfer the data from DKRZ/MPIM to ECMWF and vice versa as software tools have not been available.

The major problem was the blocking from inside ECMWF to the OpenDataHack LAN(!).

We have had a lot of copy wait time.

[m214089]

First step was the retrieval of 10m U and v from the climate data store. Two problems did arise

1. the example API python code and the data retrieval code where different and the later did not work. Has been update in the course of the afternoon.
2. The defaut grid for ERA5 is awkward: 1440x721 and breaks a lot of postprocessing tools. Usually the dimensions are a multiple of 2 and lon = 2*lat.

[m214089]

ICON data processing:

• Parallel processes: starting point: 8 TB
• Parallel processes: select 10m U and v: reduces to 50 GB (available every 15 min)
• Parallel and multi-threaded processes: apply lic operator: 240 GB
• Multi-threaded processing:create movie: 1.8 GB

[OpenDataHack2018]

This challenge was selected by team: ICON for real

[m214089]

Software work:

• install required libraries required by cdo (https://code.mpimet.mpg.de/projects/cdo). The INTEL compiler is pretty bad as it requires LD_LIBRARY_PATH which doesn't work properly with the Crays module system when set to Cray environment. Changed to gcc: works perfect, but 10% slower.
• Added an line integral convolution operator (http://www.zhanpingliu.org/research/flowvis/LIC/LIC_Source.htm) to cdo. Cleaned up the code, started optimizing and writing out as png (as the result is an image). Blended with wind velocity magnitude. Input is 10m wind vector (u, v).
• Adjusted color scale and blending with streamline equivalent generated image.
• Missing: Verification date and time in movie.
• Add existing coastlines (generated by magics++) with the right null meridian to allow for feature detection in the movie.
• Buy a bunch of 4k (8k) displays to be able to view the movies in the real resolution (20000000 gridpoints) The icon movie consists of 8000000 and the era5 movie of 1000000 pixels.

[m214089]

Viewing the first draft result:

[Try2Code]

4K movie available at https://youtu.be/XFKzoZstQiM