Processing and visualization of scientific data (netCDF, GRIB…)
Jupyter notebooks
Challenge description
The Copernicus Atmosphere Monitoring Service (CAMS) produces a wide range of forecast charts for atmospheric composition and air quality at the global (1) and European scale (2) every day. Forecast charts are published on the CAMS website but can be difficult to reproduce by users and the media.
All the forecasts are freely and openly available from the Atmosphere Data Store (ADS) (3). Given the open access nature of CAMS, and more generally Copernicus, data there is a strong requirement for users to be able to more effectively access and plot the forecast products. Jupyter notebooks have been developed for ECMWF web charts but have not yet been extended to the CAMS charts.
In addition to the web charts, CAMS also regularly shares information on its forecasts with users through social media, media, and training activities. The development of Jupyter notebooks for CAMS charts will also feed directly into these user outreach and communication activities to enhance the uptake of CAMS products with expert and non-expert users.
The project will be covering the following aspects:
Develop a standard Jupyter notebook template, or templates, which can be applied in a flexible manner to the web charts currently published on the CAMS website for global and regional forecasts.
Utilise and test ECMWF earthkit python library for data processing and visualization.
Test portability for using the Jupyter notebooks in training and communication applications.
Optional development to map earthkit libraries to other commonly available open source libraries.
Implementation proposal
All the data required to reproduce the plots is hosted on the ADS. We envisage that each Notebook for a web product will be generated programmatically based on a template and a configuration file. The type of information contained in a configuration file:
mapping between a web product name and a list of specific keys that should be used in an ADS API request (dataset name, variable name, level type ...)
unit or physical quantity conversion formulae
graphical attributes required to reproduce a chart like contour colour map and contour intervals (the CAMS website plots are currently generated using the ECMWF Magics plotting library. Your task will be to convert the current configuration to matplotlib code
We welcome proposals using different approaches that would solve the same problems.
We expect two distinct deliverables. One is a Python script that creates a Jupyter Notebook for a given CAMS web chart based on a configuration file. The other is a collection of Notebooks for every chart that we publish daily.
Challenge 14 - Jupyter notebooks for CAMS web charts
Goal
Develop Jupyter notebooks for reproducing CAMS global and European scale forecast charts.
Mentors and skills
Challenge description
The Copernicus Atmosphere Monitoring Service (CAMS) produces a wide range of forecast charts for atmospheric composition and air quality at the global (1) and European scale (2) every day. Forecast charts are published on the CAMS website but can be difficult to reproduce by users and the media.
All the forecasts are freely and openly available from the Atmosphere Data Store (ADS) (3). Given the open access nature of CAMS, and more generally Copernicus, data there is a strong requirement for users to be able to more effectively access and plot the forecast products. Jupyter notebooks have been developed for ECMWF web charts but have not yet been extended to the CAMS charts.
In addition to the web charts, CAMS also regularly shares information on its forecasts with users through social media, media, and training activities. The development of Jupyter notebooks for CAMS charts will also feed directly into these user outreach and communication activities to enhance the uptake of CAMS products with expert and non-expert users.
The project will be covering the following aspects:
Implementation proposal
All the data required to reproduce the plots is hosted on the ADS. We envisage that each Notebook for a web product will be generated programmatically based on a template and a configuration file. The type of information contained in a configuration file:
We welcome proposals using different approaches that would solve the same problems.
A practical example
Let’s try to reproduce a forecast chart which is available at (https://atmosphere.copernicus.eu/charts/packages/cams/products/nitrogen-dioxide-forecasts?layer_name=composition_no2_surface&projection=classical_global):
1) Data request
c.retrieve( 'cams-global-atmospheric-composition-forecasts', { 'date': '2024-03-13/2024-03-13', 'type': 'forecast', 'format': 'grib', 'variable': 'nitrogen_dioxide', 'model_level': '137', 'time': '00:00', 'leadtime_hour': '12', }, 'download.grib')
2) Unit conversion
To convert the values from mass mixing ratio to volume mixing ratio in ppbv we need to multiply then by:
Factor = 28.97 / 46.0055 * 1e9
3) Translating plot attributes
We will need to convert Magics plot style settings to Python. This is an extract from the current configuration file:
...
"contour_level_list": "0.01/0.02/0.05/0.1/0.2/0.5/1/2/5/10/15/20/50/100/300",
"contour_shade_colour_list": "rgb(0.607,0.780,0.917)/rgb(0.000,0.619,0.878)/rgb(0.000,0.482,0.749)/rgb(0.149,0.356,0.627)/rgb(0.149,0.568,0.439)/rgb(0.419,0.698,0.356)/rgb(0.658,0.788,0.419)/rgb(1.000,0.956,0.607)/rgb(0.956,0.819,0.219)/rgb(0.917,0.709,0.219)/rgb(0.866,0.529,0.250)/rgb(0.800,0.207,0.200)/rgb(0.607,0.086,0.137)/rgb(0.337,0.098,0.098)", ...
Deliverables
We expect two distinct deliverables. One is a Python script that creates a Jupyter Notebook for a given CAMS web chart based on a configuration file. The other is a collection of Notebooks for every chart that we publish daily.
References