About

The goal of the GribMagic project is to unify the download process of public GRIB1/GRIB2 and netCDF data from numerical weather prediction models originating from different organizations and data providers.

This is a work in progress, as such GribMagic is currently considered to be beta software. As this is an early-stage project, contributions are highly appreciated.

Details

The software package includes different code bases. Both provide their features by means of a gribmagic subcommand.

• GribMagic Unity program: gribmagic unity --help
• DWD GRIB Downloader program: gribmagic dwd --help

You can find corresponding invocation examples later in this document.

Setup

Install the ecCodes package by ECMWF.

# Debian Linux
apt-get install --yes libeccodes0 cdo

# macOS/Homebrew
brew install eccodes cdo

Install GribMagic Python package.

pip install gribmagic --upgrade

Run GribMagic Unity program

Ad hoc usage

# List labels of available models.
gribmagic unity list

# Acquire data.
gribmagic unity acquire --model=dwd-icon-eu --timestamp=2021-10-03T00:00:00Z --target=.gribmagic-data

Configuration

To optionally adjust configuration settings, you can define the following environment variables:

GM_DATA_PATH={PATH_TO_PROJECT}/data

GM_MODEL_CONFIG={PATH_TO_PROJECT}/config/model_config.yml"
GM_MODEL_VARIABLES_MAPPING={PATH_TO_PROJECT}/config/model_variables_mapping.yml"
GM_MODEL_VARIABLES_LEVELS_MAPPING={PATH_TO_PROJECT}/config/model_variables_levels_mapping.yml"

ECCODES_DEFINITION_PATH=/usr/share/eccodes/definitions:/usr/local/opt/eccodes/share/eccodes/definitions

Run DWD GRIB Downloader program

# Install DWD GRIB Downloader program.
gribmagic install dwd-grib-downloader

# Acquire wind-specific parameters from ICON-D2.
wget https://raw.githubusercontent.com/earthobservations/gribmagic/98da3fd4f/examples/dwd/recipe_d2_wind.py
gribmagic dwd acquire --recipe=recipe_d2_wind.py --timestamp="2021101800" --output=.gribmagic-data/raw

# Acquire assorted parameters from ICON-GLOBAL.
wget https://raw.githubusercontent.com/earthobservations/gribmagic/98da3fd4f/examples/dwd/recipe_global_assorted.py
gribmagic dwd acquire --recipe=recipe_global_assorted.py --timestamp="2021101800" --output=.gribmagic-data/raw

When omitting the --timestamp parameter, the most recent modelrun is automatically selected. When omitting the --output parameter, it can be supplied using the GM_DATA_PATH environment variable.

In this manner, the most compact form to invoke gribmagic dwd would be something like:

export GM_DATA_PATH=.gribmagic-data/raw
gribmagic dwd acquire --recipe=recipe_d2_wind.py

Run bbox tool

Extract area of interest from GRIB files using a bounding box.

Extract subset by coordinates, a space-separated list of lat_min lat_max lon_min lon_max.

gribmagic smith bbox \
  ".gribmagic-data/raw/icon-d2/**/*regular-lat-lon*.grib2" \
  --output=.gribmagic-data/subgrid/icon-d2 \
  --bbox=46.0 47.5 14.5 16.8

Extract subset by 2-letter country name.

gribmagic smith bbox \
  ".gribmagic-data/raw/icon-d2/**/*regular-lat-lon*.grib2" \
  --output=.gribmagic-data/subgrid/icon-d2 \
  --country=AT

For plotting the result on a map, install Magics:

# Linux
apt-get install --yes libmagplus3v5

# macOS
make magics-install
make magics-info
export MAGPLUS_HOME=/usr/local/opt/magics-4.9.3

# Install GribMagic with Magics bindings
pip install gribmagic[plotting]

and use the --plot option:

gribmagic smith bbox \
  ".gribmagic-data/raw/icon-d2/**/*regular-lat-lon*.grib2" \
  --output=.gribmagic-data/subgrid \
  --country=AT \
  --plot

Run regridding tool

Acquire grid information files.

gribmagic install dwd-grid-information

Process regridding on a single file. The default is to assume ICON GLOBAL files and the regridding will take place with 0.250° resolution.

gribmagic smith regrid \
  ".gribmagic-data/raw/icon/**/*icon_global_icosahedral*.grib2" \
  --output=.gribmagic-data/regridded/icon-global

When aiming to regrid with 0.125° resolution, please specify the --resolution=0.125 argument, like:

gribmagic smith regrid \
  ".gribmagic-data/raw/icon/**/*icon_global_icosahedral*.grib2" \
  --output=.gribmagic-data/regridded/icon-global \
  --resolution=0.125

Run program in Docker

To use GribMagic in a Docker container, you have to build the Docker image like

docker build --tag gribmagic .

and then invoke it like

docker run -it --volume=$PWD/.gribmagic-data:/var/spool/gribmagic gribmagic:latest gribmagic unity acquire --model=dwd-icon-eu --timestamp=2021-10-03T00:00:00Z

Development

Acquire source code

git clone https://github.com/earthobservations/gribmagic
cd gribmagic

Run software tests

# Run all tests.
make test

# All tests, with code coverage report.
make test-coverage

# Run specific tests, with coverage report.
.venv/bin/pytest -vvv --cov-report=term-missing --cov=gribmagic.smith.regrid -k test_regrid

# Run tests in parallel.
make test-parallel
make test-coverage-parallel

Content attributions

The copyright of data sets, images and pictograms are held by their respective owners, unless otherwise noted.

Banner image

• 2 metre temperature from ECMWF, rendered using Magics.
  • Source: MARS labelling or ensemble forecast data (2013-03-25)
  • URL: http://download.ecmwf.int/test-data/magics/2m_temperature.grib