search

tsherwen / AC_tools

Module for working with global/regional Chemical Transport Model (CTM) output and observations
MIT License
13 stars 10 forks source link

Improve scalability of reads of many NetCDF files via xarray/dask #119

Open tsherwen opened 2 years ago

tsherwen commented 2 years ago

A core bottleneck is the scalability of reads of many NetCDF files via xarray/dask.

This has been improved by many updates. However, this remains a core bottleneck for analysis.

Pre-xarray direct reading and handling of NetCDF files was faster and more scaleable. However, the benefits brought by using xarray/dask are very notable.