search

bopen / c3s-eqc-toolbox-template

CADS Toolbox template application
Apache License 2.0
6 stars 4 forks source link

Cmip6 precipitation variability assessment #163

Open almarbo opened 3 weeks ago

almarbo commented 3 weeks ago

Notebook description

This notebook downloads daily precipitation data from CMIP6 and calculates rolling variances for 10- and 30-year periods over the historical period from 1940 to 2014 and time aggregation of JJA. It displays the spatial patterns of the average 10- and 30-year variances globally (or over India if processing is too intensive). Finally, it presents a time series of the 10- and 30-year variances for spatially averaged values over India. The notebook’s structure closely follows that of this assessment, though simplified, as it only includes CMIP6 precipitation data without index calculations—just rolling variances over 10- and 30-year windows.

I have prepared an example to enable us to iterate and refine it before posting the issue on the ECMWF GitHub. This example covers the period from 2004 to 2014, with maps displayed for the region of India (ideally, the final version will be global if feasible). The rolling variance windows are set to 2 and 3 years in this example, given the considered period for this example is only 10 years. Please let me know if anything is unclear, or feel free to arrange a meeting if needed.

Notebook link or upload

example_cmip6_pr_variability.zip

Anything else we need to know?

No response

Environment

name: DEVELOP_am35323 channels: - conda-forge - defaults dependencies: - _libgcc_mutex=0.1=conda_forge - _openmp_mutex=4.5=2_gnu - accessible-pygments=0.0.5=pyhd8ed1ab_0 - aiohttp=3.9.5=py311h459d7ec_0 - aiosignal=1.3.1=pyhd8ed1ab_0 - alabaster=0.7.16=pyhd8ed1ab_0 - alsa-lib=1.2.12=h4ab18f5_0 - annotated-types=0.7.0=pyhd8ed1ab_0 - ansicolors=1.1.8=pyhd8ed1ab_0 - antlr-python-runtime=4.11.1=pyhd8ed1ab_0 - anyascii=0.3.2=pyhd8ed1ab_0 - anyio=4.4.0=pyhd8ed1ab_0 - appdirs=1.4.4=pyh9f0ad1d_0 - argon2-cffi=23.1.0=pyhd8ed1ab_0 - argon2-cffi-bindings=21.2.0=py311h459d7ec_4 - arrow=1.3.0=pyhd8ed1ab_0 - asciitree=0.3.3=py_2 - astroid=3.2.3=py311h38be061_0 - asttokens=2.4.1=pyhd8ed1ab_0 - async-lru=2.0.4=pyhd8ed1ab_0 - attr=2.5.1=h166bdaf_1 - attrs=23.2.0=pyh71513ae_0 - aws-c-auth=0.7.22=heee8711_7 - aws-c-cal=0.7.0=h816f305_0 - aws-c-common=0.9.23=h4ab18f5_0 - aws-c-compression=0.2.18=he027950_7 - aws-c-event-stream=0.4.2=hb72ac1a_14 - aws-c-http=0.8.2=had8cc17_4 - aws-c-io=0.14.9=h37d6bf3_5 - aws-c-mqtt=0.10.4=hb0abfc5_7 - aws-c-s3=0.6.0=h1f67ec3_0 - aws-c-sdkutils=0.1.16=he027950_3 - aws-checksums=0.1.18=he027950_7 - aws-crt-cpp=0.27.2=heffe44f_0 - aws-sdk-cpp=1.11.329=habc23cd_8 - azure-core-cpp=1.12.0=h830ed8b_0 - azure-identity-cpp=1.8.0=hdb0d106_1 - azure-storage-blobs-cpp=12.11.0=ha67cba7_1 - azure-storage-common-cpp=12.6.0=he3f277c_1 - azure-storage-files-datalake-cpp=12.10.0=h29b5301_1 - babel=2.14.0=pyhd8ed1ab_0 - beautifulsoup4=4.12.3=pyha770c72_0 - bleach=6.1.0=pyhd8ed1ab_0 - blosc=1.21.6=hef167b5_0 - bokeh=3.5.0=pyhd8ed1ab_0 - boltons=24.0.0=pyhd8ed1ab_0 - bottleneck=1.4.0=py311h18e1886_1 - branca=0.7.2=pyhd8ed1ab_0 - brotli=1.1.0=hd590300_1 - brotli-bin=1.1.0=hd590300_1 - brotli-python=1.1.0=py311hb755f60_1 - bzip2=1.0.8=hd590300_5 - c-ares=1.32.0=h4ab18f5_0 - ca-certificates=2024.7.4=hbcca054_0 - cached-property=1.5.2=hd8ed1ab_1 - cached_property=1.5.2=pyha770c72_1 - cairo=1.18.0=hbb29018_2 - cartopy=0.23.0=py311h14de704_1 - certifi=2024.7.4=pyhd8ed1ab_0 - cf-units=3.2.0=py311h18e1886_5 - cf_xarray=0.9.3=pyhd8ed1ab_0 - cffi=1.16.0=py311hb3a22ac_0 - cfgrib=0.9.13.0=pyhd8ed1ab_0 - cfgv=3.3.1=pyhd8ed1ab_0 - cfitsio=4.4.1=hf8ad068_0 - cftime=1.6.4=py311h18e1886_0 - charset-normalizer=3.3.2=pyhd8ed1ab_0 - click=8.1.7=unix_pyh707e725_0 - cloudpickle=3.0.0=pyhd8ed1ab_0 - cmocean=4.0.3=pyhd8ed1ab_0 - colorama=0.4.6=pyhd8ed1ab_0 - colorspacious=1.1.2=pyh24bf2e0_0 - comm=0.2.2=pyhd8ed1ab_0 - conda-merge=0.2.0=pyhd8ed1ab_0 - contourpy=1.2.1=py311h9547e67_0 - coverage=7.6.0=py311h61187de_0 - cycler=0.12.1=pyhd8ed1ab_0 - cytoolz=0.12.3=py311h459d7ec_0 - dask=2024.7.0=pyhd8ed1ab_0 - dask-core=2024.7.0=pyhd8ed1ab_0 - dask-expr=1.1.7=pyhd8ed1ab_0 - dataclasses=0.8=pyhc8e2a94_3 - dateparser=1.2.0=pyhd8ed1ab_0 - dbus=1.13.6=h5008d03_3 - debugpy=1.8.2=py311h4332511_0 - decorator=5.1.1=pyhd8ed1ab_0 - defusedxml=0.7.1=pyhd8ed1ab_0 - distlib=0.3.8=pyhd8ed1ab_0 - distributed=2024.7.0=pyhd8ed1ab_0 - docutils=0.20.1=py311h38be061_3 - eccodes=2.34.0=he84ddb8_0 - entrypoints=0.4=pyhd8ed1ab_0 - esmf=8.6.1=nompi_h0a5817f_2 - esmpy=8.6.1=pyhc1e730c_0 - exceptiongroup=1.2.0=pyhd8ed1ab_2 - executing=2.0.1=pyhd8ed1ab_0 - expat=2.6.2=h59595ed_0 - fasteners=0.17.3=pyhd8ed1ab_0 - filelock=3.15.4=pyhd8ed1ab_0 - findlibs=0.0.5=pyhd8ed1ab_0 - flexcache=0.3=pyhd8ed1ab_0 - flexparser=0.3.1=pyhd8ed1ab_0 - fmt=10.2.1=h00ab1b0_0 - folium=0.17.0=pyhd8ed1ab_0 - font-ttf-dejavu-sans-mono=2.37=hab24e00_0 - font-ttf-inconsolata=3.000=h77eed37_0 - font-ttf-source-code-pro=2.038=h77eed37_0 - font-ttf-ubuntu=0.83=h77eed37_2 - fontconfig=2.14.2=h14ed4e7_0 - fonts-conda-ecosystem=1=0 - fonts-conda-forge=1=0 - fonttools=4.53.1=py311h61187de_0 - fqdn=1.5.1=pyhd8ed1ab_0 - freeglut=3.2.2=ha6d2627_3 - freetype=2.12.1=h267a509_2 - freexl=2.0.0=h743c826_0 - frozenlist=1.4.1=py311h459d7ec_0 - fsspec=2024.6.1=pyhff2d567_0 - gdal=3.9.1=py311h8fb44bb_6 - geopandas=1.0.1=pyhd8ed1ab_0 - geopandas-base=1.0.1=pyha770c72_0 - geos=3.12.2=hac33072_0 - geotiff=1.7.3=hf7fa9e8_1 - gettext=0.22.5=h59595ed_2 - gettext-tools=0.22.5=h59595ed_2 - gflags=2.2.2=he1b5a44_1004 - giflib=5.2.2=hd590300_0 - glib=2.80.3=h8a4344b_1 - glib-tools=2.80.3=h73ef956_1 - glog=0.7.1=hbabe93e_0 - graphite2=1.3.13=h59595ed_1003 - greenlet=3.0.3=py311hb755f60_0 - gst-plugins-base=1.24.5=hbaaba92_0 - gstreamer=1.24.5=haf2f30d_0 - h11=0.14.0=pyhd8ed1ab_0 - h2=4.1.0=pyhd8ed1ab_0 - harfbuzz=8.5.0=hfac3d4d_0 - hdf4=4.2.15=h2a13503_7 - hdf5=1.14.3=nompi_hdf9ad27_105 - hpack=4.0.0=pyh9f0ad1d_0 - httpcore=1.0.5=pyhd8ed1ab_0 - httpx=0.27.0=pyhd8ed1ab_0 - hyperframe=6.0.1=pyhd8ed1ab_0 - icclim=6.5.0=pyhd8ed1ab_0 - icu=73.2=h59595ed_0 - identify=2.6.0=pyhd8ed1ab_0 - idna=3.7=pyhd8ed1ab_0 - imagesize=1.4.1=pyhd8ed1ab_0 - importlib-metadata=8.0.0=pyha770c72_0 - importlib_metadata=8.0.0=hd8ed1ab_0 - importlib_resources=6.4.0=pyhd8ed1ab_0 - iniconfig=2.0.0=pyhd8ed1ab_0 - ipykernel=6.29.5=pyh3099207_0 - ipython=8.26.0=pyh707e725_0 - isoduration=20.11.0=pyhd8ed1ab_0 - jasper=4.2.4=h536e39c_0 - jedi=0.19.1=pyhd8ed1ab_0 - jinja2=3.1.4=pyhd8ed1ab_0 - joblib=1.4.2=pyhd8ed1ab_0 - json-c=0.17=h7ab15ed_0 - json5=0.9.25=pyhd8ed1ab_0 - jsonpickle=3.2.2=pyhd8ed1ab_0 - jsonpointer=3.0.0=py311h38be061_0 - jsonschema=4.23.0=pyhd8ed1ab_0 - jsonschema-specifications=2023.12.1=pyhd8ed1ab_0 - jsonschema-with-format-nongpl=4.23.0=hd8ed1ab_0 - jupyter-book=1.0.2=pyhd8ed1ab_0 - jupyter-cache=1.0.0=pyhd8ed1ab_0 - jupyter-lsp=2.2.5=pyhd8ed1ab_0 - jupyter-server-proxy=4.3.0=pyhd8ed1ab_0 - jupyter_client=8.6.2=pyhd8ed1ab_0 - jupyter_core=5.7.2=py311h38be061_0 - jupyter_events=0.10.0=pyhd8ed1ab_0 - jupyter_server=2.14.1=pyhd8ed1ab_1 - jupyter_server_terminals=0.5.3=pyhd8ed1ab_0 - jupyterlab=4.2.3=pyhd8ed1ab_0 - jupyterlab-myst=2.4.2=pyhd8ed1ab_0 - jupyterlab_pygments=0.3.0=pyhd8ed1ab_1 - jupyterlab_server=2.27.2=pyhd8ed1ab_0 - kealib=1.5.3=hee9dde6_1 - keyutils=1.6.1=h166bdaf_0 - kiwisolver=1.4.5=py311h9547e67_1 - krb5=1.21.3=h659f571_0 - lame=3.100=h166bdaf_1003 - latexcodec=2.0.1=pyh9f0ad1d_0 - lcms2=2.16=hb7c19ff_0 - ld_impl_linux-64=2.40=hf3520f5_7 - lerc=4.0.0=h27087fc_0 - libabseil=20240116.2=cxx17_h59595ed_0 - libaec=1.1.3=h59595ed_0 - libarchive=3.7.4=hfca40fe_0 - libarrow=16.1.0=h56e7afd_13_cpu - libarrow-acero=16.1.0=he02047a_13_cpu - libarrow-dataset=16.1.0=he02047a_13_cpu - libarrow-substrait=16.1.0=hc9a23c6_13_cpu - libasprintf=0.22.5=h661eb56_2 - libasprintf-devel=0.22.5=h661eb56_2 - libblas=3.9.0=22_linux64_openblas - libbrotlicommon=1.1.0=hd590300_1 - libbrotlidec=1.1.0=hd590300_1 - libbrotlienc=1.1.0=hd590300_1 - libcap=2.69=h0f662aa_0 - libcblas=3.9.0=22_linux64_openblas - libclang-cpp15=15.0.7=default_h127d8a8_5 - libclang13=18.1.8=default_h6ae225f_0 - libcrc32c=1.1.2=h9c3ff4c_0 - libcups=2.3.3=h4637d8d_4 - libcurl=8.8.0=hca28451_1 - libdeflate=1.20=hd590300_0 - libedit=3.1.20191231=he28a2e2_2 - libev=4.33=hd590300_2 - libevent=2.1.12=hf998b51_1 - libexpat=2.6.2=h59595ed_0 - libffi=3.4.2=h7f98852_5 - libflac=1.4.3=h59595ed_0 - libgcc-ng=14.1.0=h77fa898_0 - libgcrypt=1.11.0=h4ab18f5_0 - libgdal=3.9.1=h8617584_6 - libgettextpo=0.22.5=h59595ed_2 - libgettextpo-devel=0.22.5=h59595ed_2 - libgfortran-ng=14.1.0=h69a702a_0 - libgfortran5=14.1.0=hc5f4f2c_0 - libglib=2.80.3=h8a4344b_1 - libglu=9.0.0=ha6d2627_1004 - libgomp=14.1.0=h77fa898_0 - libgoogle-cloud=2.26.0=h26d7fe4_0 - libgoogle-cloud-storage=2.26.0=ha262f82_0 - libgpg-error=1.50=h4f305b6_0 - libgrpc=1.62.2=h15f2491_0 - libiconv=1.17=hd590300_2 - libjpeg-turbo=3.0.0=hd590300_1 - libkml=1.3.0=hbbc8833_1020 - liblapack=3.9.0=22_linux64_openblas - libllvm14=14.0.6=hcd5def8_4 - libllvm15=15.0.7=hb3ce162_4 - libllvm18=18.1.8=hc9dba70_0 - libnetcdf=4.9.2=nompi_h135f659_114 - libnghttp2=1.58.0=h47da74e_1 - libnsl=2.0.1=hd590300_0 - libogg=1.3.5=h4ab18f5_0 - libopenblas=0.3.27=pthreads_hac2b453_1 - libopus=1.3.1=h7f98852_1 - libparquet=16.1.0=h9e5060d_13_cpu - libpng=1.6.43=h2797004_0 - libpq=16.3=ha72fbe1_0 - libprotobuf=4.25.3=h08a7969_0 - libre2-11=2023.09.01=h5a48ba9_2 - librttopo=1.1.0=hc670b87_16 - libsndfile=1.2.2=hc60ed4a_1 - libsodium=1.0.18=h36c2ea0_1 - libspatialite=5.1.0=h15fa968_8 - libsqlite=3.46.0=hde9e2c9_0 - libssh2=1.11.0=h0841786_0 - libstdcxx-ng=14.1.0=hc0a3c3a_0 - libsystemd0=255=h3516f8a_1 - libthrift=0.19.0=hb90f79a_1 - libtiff=4.6.0=h1dd3fc0_3 - libudunits2=2.2.28=h40f5838_3 - libutf8proc=2.8.0=h166bdaf_0 - libuuid=2.38.1=h0b41bf4_0 - libvorbis=1.3.7=h9c3ff4c_0 - libwebp-base=1.4.0=hd590300_0 - libxcb=1.16=hd590300_0 - libxcrypt=4.4.36=hd590300_1 - libxkbcommon=1.7.0=h2c5496b_1 - libxml2=2.12.7=h4c95cb1_3 - libzip=1.10.1=h2629f0a_3 - libzlib=1.3.1=h4ab18f5_1 - linkify-it-py=2.0.3=pyhd8ed1ab_0 - llvmlite=0.43.0=py311hbde99c3_0 - locket=1.0.0=pyhd8ed1ab_0 - lz4=4.3.3=py311h38e4bf4_0 - lz4-c=1.9.4=hcb278e6_0 - lzo=2.10=hd590300_1001 - make=4.3=hd18ef5c_1 - mapclassify=2.6.1=pyhd8ed1ab_0 - markdown-it-py=3.0.0=pyhd8ed1ab_0 - markupsafe=2.1.5=py311h459d7ec_0 - matplotlib=3.9.1=py311h38be061_0 - matplotlib-base=3.9.1=py311hffb96ce_0 - matplotlib-inline=0.1.7=pyhd8ed1ab_0 - mdit-py-plugins=0.4.1=pyhd8ed1ab_0 - mdurl=0.1.2=pyhd8ed1ab_0 - minizip=4.0.7=h401b404_0 - mistune=3.0.2=pyhd8ed1ab_0 - mpg123=1.32.6=h59595ed_0 - msgpack-python=1.0.8=py311h52f7536_0 - multidict=6.0.5=py311h459d7ec_0 - multiurl=0.3.1=pyhd8ed1ab_0 - munkres=1.1.4=pyh9f0ad1d_0 - mypy=1.10.1=py311h331c9d8_0 - mypy_extensions=1.0.0=pyha770c72_0 - mysql-common=8.3.0=hf1915f5_4 - mysql-libs=8.3.0=hca2cd23_4 - myst-nb=1.1.1=pyhd8ed1ab_0 - myst-parser=2.0.0=pyhd8ed1ab_0 - nbclient=0.10.0=pyhd8ed1ab_0 - nbconvert=7.16.4=hd8ed1ab_1 - nbconvert-core=7.16.4=pyhd8ed1ab_1 - nbconvert-pandoc=7.16.4=hd8ed1ab_1 - nbformat=5.10.4=pyhd8ed1ab_0 - nc-time-axis=1.4.1=pyhd8ed1ab_0 - ncurses=6.5=h59595ed_0 - nest-asyncio=1.6.0=pyhd8ed1ab_0 - netcdf-fortran=4.6.1=nompi_h228c76a_104 - netcdf4=1.7.1=nompi_py311h25b3b55_101 - networkx=3.3=pyhd8ed1ab_1 - nodeenv=1.9.1=pyhd8ed1ab_0 - notebook-shim=0.2.4=pyhd8ed1ab_0 - nspr=4.35=h27087fc_0 - nss=3.102=h593d115_0 - numba=0.60.0=py311h4bc866e_0 - numcodecs=0.12.1=py311h4332511_1 - numpy=1.26.4=py311h64a7726_0 - openjpeg=2.5.2=h488ebb8_0 - openssl=3.3.1=h4ab18f5_1 - orc=2.0.1=h17fec99_1 - overrides=7.7.0=pyhd8ed1ab_0 - packaging=24.1=pyhd8ed1ab_0 - pandoc=3.2.1=ha770c72_0 - pandocfilters=1.5.0=pyhd8ed1ab_0 - papermill=2.6.0=pyhd8ed1ab_0 - parso=0.8.4=pyhd8ed1ab_0 - partd=1.4.2=pyhd8ed1ab_0 - patsy=0.5.6=pyhd8ed1ab_0 - pcre2=10.44=h0f59acf_0 - pexpect=4.9.0=pyhd8ed1ab_0 - pickleshare=0.7.5=py_1003 - pillow=10.4.0=py311h82a398c_0 - pint=0.24.1=pyhd8ed1ab_1 - pip=24.0=pyhd8ed1ab_0 - pixman=0.43.2=h59595ed_0 - pkgutil-resolve-name=1.3.10=pyhd8ed1ab_1 - platformdirs=4.2.2=pyhd8ed1ab_0 - plotly=5.22.0=pyhd8ed1ab_0 - pluggy=1.5.0=pyhd8ed1ab_0 - ply=3.11=pyhd8ed1ab_2 - poppler=24.07.0=hb0d391f_0 - poppler-data=0.4.12=hd8ed1ab_0 - postgresql=16.3=h8e811e2_0 - pre-commit=3.7.1=pyha770c72_0 - proj=9.4.1=hb784bbd_0 - prometheus_client=0.20.0=pyhd8ed1ab_0 - prompt-toolkit=3.0.47=pyha770c72_0 - properscoring=0.1=py_0 - psutil=6.0.0=py311h331c9d8_0 - pthread-stubs=0.4=h36c2ea0_1001 - ptyprocess=0.7.0=pyhd3deb0d_0 - pulseaudio-client=17.0=hb77b528_0 - pure_eval=0.2.2=pyhd8ed1ab_0 - pyarrow=16.1.0=py311hbd00459_4 - pyarrow-core=16.1.0=py311h8c3dac4_4_cpu - pyarrow-hotfix=0.6=pyhd8ed1ab_0 - pybtex=0.24.0=pyhd8ed1ab_2 - pybtex-docutils=1.0.3=py311h38be061_1 - pycparser=2.22=pyhd8ed1ab_0 - pydantic=2.8.2=pyhd8ed1ab_0 - pydantic-core=2.20.1=py311hb3a8bbb_0 - pydantic-settings=2.3.4=pyhd8ed1ab_0 - pydata-sphinx-theme=0.15.4=pyhd8ed1ab_0 - pygments=2.18.0=pyhd8ed1ab_0 - pyogrio=0.9.0=py311hfc743a8_0 - pyparsing=3.1.2=pyhd8ed1ab_0 - pyproj=3.6.1=py311hc21b84f_7 - pyqt=5.15.9=py311hf0fb5b6_5 - pyqt5-sip=12.12.2=py311hb755f60_5 - pyshp=2.3.1=pyhd8ed1ab_0 - pysocks=1.7.1=pyha2e5f31_6 - pytest=8.2.2=pyhd8ed1ab_0 - pytest-cov=5.0.0=pyhd8ed1ab_0 - python=3.11.9=hb806964_0_cpython - python-dateutil=2.9.0=pyhd8ed1ab_0 - python-dotenv=1.0.1=pyhd8ed1ab_0 - python-eccodes=1.7.1=py311h18e1886_0 - python-fastjsonschema=2.20.0=pyhd8ed1ab_0 - python-json-logger=2.0.7=pyhd8ed1ab_0 - python-tzdata=2024.1=pyhd8ed1ab_0 - python_abi=3.11=4_cp311 - pytz=2024.1=pyhd8ed1ab_0 - pyyaml=6.0.1=py311h459d7ec_1 - pyzmq=26.0.3=py311h08a0b41_0 - qhull=2020.2=h434a139_3 - qt-main=5.15.8=ha2b5568_22 - re2=2023.09.01=h7f4b329_2 - readline=8.2=h8228510_1 - rechunker=0.5.2=pyhd8ed1ab_1 - referencing=0.35.1=pyhd8ed1ab_0 - regex=2024.5.15=py311h331c9d8_0 - requests=2.32.3=pyhd8ed1ab_0 - rfc3339-validator=0.1.4=pyhd8ed1ab_0 - rfc3986-validator=0.1.1=pyh9f0ad1d_0 - rich=13.7.1=pyhd8ed1ab_0 - rpds-py=0.19.0=py311hb3a8bbb_0 - s2n=1.4.17=he19d79f_0 - scikit-learn=1.5.1=py311hd632256_0 - scipy=1.14.0=py311h517d4fd_1 - seaborn=0.13.2=hd8ed1ab_2 - seaborn-base=0.13.2=pyhd8ed1ab_2 - send2trash=1.8.3=pyh0d859eb_0 - setuptools=70.2.0=pyhd8ed1ab_0 - shapely=2.0.4=py311h5925939_2 - shellingham=1.5.4=pyhd8ed1ab_0 - simpervisor=1.0.0=pyhd8ed1ab_0 - sip=6.7.12=py311hb755f60_0 - six=1.16.0=pyh6c4a22f_0 - snappy=1.2.1=ha2e4443_0 - sniffio=1.3.1=pyhd8ed1ab_0 - snowballstemmer=2.2.0=pyhd8ed1ab_0 - sortedcontainers=2.4.0=pyhd8ed1ab_0 - soupsieve=2.5=pyhd8ed1ab_1 - sparse=0.15.4=pyhd8ed1ab_0 - spdlog=1.13.0=hd2e6256_0 - sphinx=7.3.7=pyhd8ed1ab_0 - sphinx-autoapi=3.0.0=pyhd8ed1ab_0 - sphinx-book-theme=1.1.3=pyhd8ed1ab_0 - sphinx-comments=0.0.3=pyh9f0ad1d_0 - sphinx-copybutton=0.5.2=pyhd8ed1ab_0 - sphinx-design=0.6.0=pyhd8ed1ab_0 - sphinx-external-toc=1.0.1=pyhd8ed1ab_0 - sphinx-jupyterbook-latex=1.0.0=pyhd8ed1ab_0 - sphinx-multitoc-numbering=0.1.3=pyhd8ed1ab_0 - sphinx-thebe=0.3.1=pyhd8ed1ab_0 - sphinx-togglebutton=0.3.2=pyhd8ed1ab_0 - sphinxcontrib-applehelp=1.0.8=pyhd8ed1ab_0 - sphinxcontrib-bibtex=2.6.2=pyhd8ed1ab_0 - sphinxcontrib-devhelp=1.0.6=pyhd8ed1ab_0 - sphinxcontrib-htmlhelp=2.0.5=pyhd8ed1ab_0 - sphinxcontrib-jsmath=1.0.1=pyhd8ed1ab_0 - sphinxcontrib-qthelp=1.0.7=pyhd8ed1ab_0 - sphinxcontrib-serializinghtml=1.1.10=pyhd8ed1ab_0 - sqlalchemy=2.0.31=py311h331c9d8_0 - sqlite=3.46.0=h6d4b2fc_0 - stack_data=0.6.2=pyhd8ed1ab_0 - statsmodels=0.14.2=py311h18e1886_0 - structlog=24.2.0=pyhd8ed1ab_0 - tabulate=0.9.0=pyhd8ed1ab_1 - tblib=3.0.0=pyhd8ed1ab_0 - tenacity=8.5.0=pyhd8ed1ab_0 - terminado=0.18.1=pyh0d859eb_0 - threadpoolctl=3.5.0=pyhc1e730c_0 - tiledb=2.24.2=h9260d03_1 - tinycss2=1.3.0=pyhd8ed1ab_0 - tk=8.6.13=noxft_h4845f30_101 - toml=0.10.2=pyhd8ed1ab_0 - tomli=2.0.1=pyhd8ed1ab_0 - toolz=0.12.1=pyhd8ed1ab_0 - tornado=6.4.1=py311h331c9d8_0 - tqdm=4.66.4=pyhd8ed1ab_0 - traitlets=5.14.3=pyhd8ed1ab_0 - typer=0.12.3=pyhd8ed1ab_0 - typer-slim=0.12.3=pyhd8ed1ab_0 - typer-slim-standard=0.12.3=hd8ed1ab_0 - types-python-dateutil=2.9.0.20240316=pyhd8ed1ab_0 - typing-extensions=4.12.2=hd8ed1ab_0 - typing_extensions=4.12.2=pyha770c72_0 - typing_utils=0.1.0=pyhd8ed1ab_0 - tzcode=2024a=h3f72095_0 - tzdata=2024a=h0c530f3_0 - tzlocal=5.2=py311h38be061_0 - uc-micro-py=1.0.3=pyhd8ed1ab_0 - udunits2=2.2.28=h40f5838_3 - ukkonen=1.0.1=py311h9547e67_4 - uri-template=1.3.0=pyhd8ed1ab_0 - uriparser=0.9.8=hac33072_0 - urllib3=2.2.2=pyhd8ed1ab_1 - virtualenv=20.26.3=pyhd8ed1ab_0 - wcwidth=0.2.13=pyhd8ed1ab_0 - webcolors=24.6.0=pyhd8ed1ab_0 - webencodings=0.5.1=pyhd8ed1ab_2 - websocket-client=1.8.0=pyhd8ed1ab_0 - wheel=0.43.0=pyhd8ed1ab_1 - xarraymannkendall=1.4.5=pyhd8ed1ab_0 - xcb-util=0.4.1=hb711507_2 - xcb-util-image=0.4.0=hb711507_2 - xcb-util-keysyms=0.4.1=hb711507_0 - xcb-util-renderutil=0.3.10=hb711507_0 - xcb-util-wm=0.4.2=hb711507_0 - xerces-c=3.2.5=hac6953d_0 - xesmf=0.8.6=pyhd8ed1ab_0 - xhistogram=0.3.2=pyhd8ed1ab_0 - xkeyboard-config=2.42=h4ab18f5_0 - xorg-fixesproto=5.0=h7f98852_1002 - xorg-inputproto=2.3.2=h7f98852_1002 - xorg-kbproto=1.0.7=h7f98852_1002 - xorg-libice=1.1.1=hd590300_0 - xorg-libsm=1.2.4=h7391055_0 - xorg-libx11=1.8.9=hb711507_1 - xorg-libxau=1.0.11=hd590300_0 - xorg-libxdmcp=1.1.3=h7f98852_0 - xorg-libxext=1.3.4=h0b41bf4_2 - xorg-libxfixes=5.0.3=h7f98852_1004 - xorg-libxi=1.7.10=h7f98852_0 - xorg-libxrender=0.9.11=hd590300_0 - xorg-renderproto=0.11.1=h7f98852_1002 - xorg-xextproto=7.3.0=h0b41bf4_1003 - xorg-xf86vidmodeproto=2.3.1=h7f98852_1002 - xorg-xproto=7.0.31=h7f98852_1007 - xskillscore=0.0.26=pyhd8ed1ab_0 - xyzservices=2024.6.0=pyhd8ed1ab_0 - xz=5.2.6=h166bdaf_0 - yamale=5.2.1=pyhca7485f_0 - yaml=0.2.5=h7f98852_2 - yarl=1.9.4=py311h459d7ec_0 - zarr=2.18.2=pyhd8ed1ab_0 - zeromq=4.3.5=h75354e8_4 - zict=3.0.0=pyhd8ed1ab_0 - zipp=3.19.2=pyhd8ed1ab_0 - zlib=1.3.1=h4ab18f5_1 - zstandard=0.22.0=py311hb6f056b_1 - zstd=1.5.6=ha6fb4c9_0 - pip: - c3s-eqc-automatic-quality-control==0.1.2.dev112+g680c19a - cacholote==1.1.0 - cads-api-client==1.4.5 - cads-toolbox==0.0.2b0 - cdsapi==0.7.3 - cgul==0.0.4 - coucal==0.0.1b3 - dill==0.3.8 - emohawk==0.0.4b0 - kaleido==0.2.1 - lmoments3==1.0.6 - pandas==2.1.4 - pymannkendall==1.4.3 - skillmetrics==1.2.4 - xarray==2023.10.1 - xclim==0.47.0 - xlsxwriter==3.2.0 prefix: /users_home/cmcc/am35323/.conda/envs/DEVELOP_am35323
malmans2 commented 3 weeks ago

Hi @almarbo,

It looks like you haven't been using the VM. More importantly, you're running an outdated version of the EQC software. The latest version now relies on earthkit-data, the official ECMWF library, whereas you're still using emohawk, which has been deprecated by ECMWF.

When we migrated to the new VM a few months ago, we ensured that all existing notebooks were cached to use earthkit. I recommend either updating your environment and clearing the cache or switching to the VM.

I glanced through the notebook, and the overall structure looks good as it follows the other templates. Is there anything specific you’d like me to review? Do you want me to cache the notebook on the VM?

almarbo commented 3 weeks ago

Hi @malmans2 ,

Thank you very much for the quick response. I have been working on our machine because we may want to experiment with extended regions or periods in the future, which may require additional resources. I appreciate the tips and will update my environment accordingly. Another reason I haven't been using the VM is that I have been ill, and I believe I can only access the VM through my office computer (the one where I shared the public SSH key).

It would be great if you could cache the notebook in the VM. Additionally, could you review the caching functions I am using for calculating the rolling variances (.rolling)? I wouldd also appreciate any insights on the "time" handling within those functions. I have a feeling I might not be working as efficiently as possible.

almarbo commented 3 weeks ago

It would also be nice if you could also review the timeseries (last section), since it is pretty new.

malmans2 commented 3 weeks ago

OK, I started downloading data on the VM. I'll look into this tomorrow if we have enough data.

malmans2 commented 3 weeks ago

I wouldd also appreciate any insights on the "time" handling within those functions.

Let's start with this, is your goal to use 360 days calendars? If yes, we can just use this function: https://docs.xarray.dev/en/stable/generated/xarray.DataArray.convert_calendar.html

Let me know the parameters you'd like to use (i.e., calendar and align_on)

almarbo commented 3 weeks ago

My goal is to calculate the window width for each model, as each has a different length. Once I calculate the rolling variances, I select the last day of each year (season), since I only need one value per year (in other words, the step between one rolling variance value and the next is one year; however, I couldn’t find a step parameter to control this directly). For some models, which follow a 360-day calendar, this day falls on August 30, while for others, it’s August 31. This creates an issue when concatenating, as it leads to double of times in comparison to what I would like. That's why I change at the end from 31st of August to 30th, to account for this

malmans2 commented 3 weeks ago

OK, let's chat about it later. I don't fully understand why you can change the calendar in various places (e.g., remove Feb 29th or convert to proleptic_gregorian), but you can't change it at the very beginning so everything is aligned.

malmans2 commented 3 weeks ago

@almarbo, this is the snippet I showed you during the meeting:

import xarray as xr

ds = xr.tutorial.open_dataset("air_temperature")
ds_var = (
    ds.rolling(time=365 * 4)
    .construct("window_dim")
    .groupby("time.year")
    .last()
    .var("window_dim")
)

Is that the operation you need to perform?

malmans2 commented 3 weeks ago

And here is another test:

import xarray as xr

ds = xr.tutorial.open_dataset("air_temperature")
ds_var_mattia = (
    ds.rolling(time=365 * 4)
    .construct("window_dim")
    .groupby("time.year")
    .last()
    .var("window_dim", keep_attrs=True)
)
ds_var_albert = (
    ds.rolling(time=365 * 4)
    .var()
    .groupby("time.year")
    .last()
)

xr.testing.assert_allclose(ds_var_mattia, ds_var_albert)
almarbo commented 3 weeks ago

Hi @malmans2,

I haven’t had time to test this yet but will do so ASAP.

For now, I’ve improved and updated everything and incorporated the adjustments we discussed earlier (time issues, unit conversions, reducing lines using loops, etc.).

I’ve also included the coefficient of variation calculation: the square root of the variance divided by the mean (always using moving windows):

 ## Apply rolling variance over the calculated window size and calculate the coefficient of variation
    rolling_var = ds.rolling(time=days_in_window, min_periods=days_in_window, center=False).var()
    rolling_mean= ds.rolling(time=days_in_window, min_periods=days_in_window, center=False).mean()
    rolling_CV=(rolling_var**(1/2))/rolling_mean

Thanks again, and enjoy the long weekend!

malmans2 commented 2 weeks ago

For now, I’ve improved and updated everything and incorporated the adjustments we discussed earlier (time issues, unit conversions, reducing lines using loops, etc.).

Did you send me the new version?

almarbo commented 2 weeks ago

oh...sorry I didn't realise that I did not update the notebook here. I have now updated it, thanks for noticing

malmans2 commented 2 weeks ago

Hi @almarbo,

After revising your functions, I was able to run the notebook on the VM successfully.

Your version significantly increases memory usage and the kernel dies, so I can't really test it. For example, I noticed that the last plot in your notebook starts from 2005 rather than 2004.

Here are the functions I changed:

def compute_rolling_variance(ds, window_years, timeseries, resample_reduction):
    (da,) = ds.data_vars.values()
    days_in_window = (360 if timeseries == "annual" else 90) * window_years
    time = ds["time"].groupby("time.year").last()
    grouped = (
        da.rolling(time=days_in_window)
        .construct("window_dim")
        .groupby("time.year")
        .last()
        .assign_coords(time=time)
        .swap_dims(year="time")
        .drop_vars("year")
    )
    var = grouped.var("window_dim", skipna=False)
    coeff = (var ** (1 / 2)) / grouped.mean("window_dim")
    var *= (1000 if resample_reduction else 86400) ** 2
    ds = xr.merge([var, coeff.rename("coefficient_of_variation")])
    return ds.expand_dims(variance=[window_years])

def compute_variance(
    ds,
    timeseries,
    year_start,
    year_stop,
    variance_values,
    resample_reduction=None,
    request_grid_out=None,
    **regrid_kwargs,
):
    # Apply resample reduction if specified
    if resample_reduction:
        resampled = ds.resample(time="1D")
        ds = getattr(resampled, resample_reduction)(keep_attrs=True)

    # Select the time series
    ds = select_timeseries(ds, timeseries, year_start, year_stop)
    ds = ds.convert_calendar("360_day", align_on="year", use_cftime=True)

    # Compute variances
    datasets = [
        compute_rolling_variance(
            ds,
            window_years=variance,
            timeseries=timeseries,
            resample_reduction=resample_reduction,
        )
        for variance in variance_values
    ]
    ds = xr.concat(datasets, "variance")

    # Handle grid output if requested
    if request_grid_out:
        grid_out = get_grid_out(request_grid_out, method=regrid_kwargs["method"])
        ds = diagnostics.regrid(ds, grid_out=grid_out, **regrid_kwargs)

    return ds.convert_calendar("proleptic_gregorian", align_on="date", use_cftime=False)

Here is the notebook executed: https://gist.github.com/malmans2/22f2c270b226053e2281270f14b7581e

There are a couple of things I don't fully understand, but I only focused on optimising the new code:

  1. Why do you parametrise the interpolation method but then hardcode it to bilinear in the code?
  2. Why do you convert the variance units without also adjusting the coefficients accordingly?

Could you please test it? If everything works, let me know once you've finalised the notebook if you need me to cache the whole timeseries on the VM.

almarbo commented 2 weeks ago

hi @malmans2,

Since I’m using 2- and 3-year windows, variance calculation requires at least 2 years for the 2-year window and 3 years for the 3-year window. In 2004, there’s only one year of data available, so the 2-year variance can’t be computed. Similarly, in 2005, there are only two years, which is insufficient for the 3-year variance. Consequently, the first plot (Fig. 7a) starts in 2005, while Fig. 7b begins in 2006.

For example, I noticed that the last plot in your notebook starts from 2005 rather than 2004.

The initial idea was to allow either conservative or bilinear interpolation, so I tried parametrising the interpolation method. However, when using the parameterised approach, I encountered issues with some models in model_datasets returning NaNs. Strangely, without parametrising, this problem disappeared. I haven’t identified why this happens yet, as I intended to test it further, but I forgot to mention this issue to you.

Why do you parametrise the interpolation method but then hardcode it to bilinear in the code?

The variance has units, while the coefficient of variation does not, as it’s a dimensionless ratio (the square root of the variance divided by the mean). When calculating the coefficient of variation, units cancel out naturally, so no further conversion is needed. This is why only the units for variance are adjusted, not for the coefficient of variation.

Why do you convert the variance units without also adjusting the coefficients accordingly?

malmans2 commented 2 weeks ago

For example, I noticed that the last plot in your notebook starts from 2005 rather than 2004.

OK, I see. Because we use construct now, you might have to explicitly remove those years. Can you implement it or do you need me to look into it?

Why do you parametrise the interpolation method but then hardcode it to bilinear in the code?

I see, it's just std/mean.

Why do you parametrise the interpolation method but then hardcode it to bilinear in the code?

I tried the parameterised interpolation method with bilinear yesterday and didn’t encounter any issues. So, the problem might only occur with the conservative interpolation. What you’re doing now isn’t ideal: you’re caching a function with the conservative interpolation method, but then using bilinear. Please try again, and let me know if you experience any issues.

malmans2 commented 2 weeks ago

Actually, the first point might be quite easy. There's no NaNs in the original dataset right? If that's the cose, we can just do this:

var = grouped.var("window_dim", skipna=False)
malmans2 commented 2 weeks ago

Indeed, that works as expected. Here is the latest version: https://gist.github.com/malmans2/22f2c270b226053e2281270f14b7581e

I also restored the parametrisation of the interpolation, it works fine with bilinear.

malmans2 commented 2 weeks ago

I just tried conservative and I don't see any issue, but maybe I'm overlooking something. I think you have everything you need to finalise your notebook.

almarbo commented 2 weeks ago

Hi @malmans2 ,

I've been testing, and some aspects of the output seem strange. I’m seeing NaNs in certain interpolated datasets (but not for all models or for every latitude and longitude). The number of models displaying NaNs also depends on the interpolation method.

Here’s an example of what I mean:

With bilinear interpolation:

ds_interpolated.isel(variance=0,latitude=3,longitude=3).pr.values
array([[           nan,            nan,            nan,            nan,
                   nan,            nan,            nan,            nan,
                   nan,            nan,            nan],
       [           nan, 1.48577905e+00, 1.36083412e+00, 1.08807281e-01,
        3.28417659e-01, 3.76656085e-01, 3.55924129e+00, 3.81952238e+00,
        9.70415711e-01, 1.00730753e+00, 4.21003699e-01],
       [           nan,            nan,            nan,            nan,
                   nan,            nan,            nan,            nan,
                   nan,            nan,            nan],
       [           nan, 2.50745702e+00, 3.43969371e-03, 1.58406961e+00,
        1.78741062e+00, 8.41390550e-01, 8.22588563e-01, 1.99909782e+00,
        3.04484844e-01, 2.79019605e-02, 3.49222729e-03],
       [           nan,            nan,            nan,            nan,
                   nan,            nan,            nan,            nan,
                   nan,            nan,            nan],
       [           nan, 5.02978325e-01, 5.02960443e-01, 6.16614008e-04,
        6.65174040e-04, 1.43549478e-04, 2.03579810e-04, 6.53892348e-05,
        2.39252537e-01, 2.39732191e-01, 8.81395754e-05],
       [           nan, 2.79195714e+00, 1.11923683e+00, 1.87317148e-01,
        2.73526609e-01, 1.34938931e+00, 5.44377714e-02, 2.19866753e-01,
        1.14071131e+00, 1.13717651e+00, 5.12892865e-02],
       [           nan,            nan,            nan,            nan,
                   nan,            nan,            nan,            nan,
                   nan,            nan,            nan],
       [           nan,            nan,            nan,            nan,
                   nan,            nan,            nan,            nan,
                   nan,            nan,            nan],
       [           nan,            nan,            nan,            nan,
                   nan,            nan,            nan,            nan,
                   nan,            nan,            nan],
       [           nan,            nan,            nan,            nan,
                   nan,            nan,            nan,            nan,
                   nan,            nan,            nan],
       [           nan,            nan,            nan,            nan,
                   nan,            nan,            nan,            nan,
                   nan,            nan,            nan],
       [           nan, 6.49854660e-01, 5.87625384e-01, 2.75233746e-01,
        7.17305779e-01, 1.70369816e+00, 1.79918849e+00, 1.23120618e+00,
        1.07741749e+00, 2.40537190e+00, 5.01404762e-01],
       [           nan,            nan,            nan,            nan,
                   nan,            nan,            nan,            nan,
                   nan,            nan,            nan],
       [           nan,            nan,            nan,            nan,
                   nan,            nan,            nan,            nan,
                   nan,            nan,            nan],
       [           nan,            nan,            nan,            nan,
                   nan,            nan,            nan,            nan,
                   nan,            nan,            nan]])

With conservative interpolation:

ds_interpolated.isel(variance=0,latitude=3,longitude=3).pr.values
array([[           nan, 1.30008769e+00, 2.57578313e-01, 3.82155299e-01,
        2.74047434e-01, 9.88518912e-03, 2.64352083e+00, 2.63979268e+00,
        3.79363485e-02, 1.05725668e-01, 2.44640529e-01],
       [           nan, 5.43833733e-01, 5.25143802e-01, 1.04788408e-01,
        1.47605419e-01, 1.53546199e-01, 4.65563631e+00, 4.78511286e+00,
        2.69868761e-01, 5.15363932e-01, 4.58529919e-01],
       [           nan, 1.87336933e-02, 7.21187368e-02, 6.20284975e-01,
        6.13317847e-01, 1.31843731e-01, 3.85855651e-03, 6.07993156e-02,
        7.67587423e-02, 2.60598026e-02, 1.93415617e-03],
       [           nan, 2.31074333e+00, 2.25445884e-03, 1.53524625e+00,
        1.72679913e+00, 8.12479436e-01, 8.24728251e-01, 1.91649163e+00,
        2.16389090e-01, 1.92284733e-02, 2.63395743e-03],
       [           nan, 1.28151095e-02, 1.27671622e-02, 3.14679360e+00,
        3.28719354e+00, 1.87367558e-01, 3.00795659e-02, 3.14509292e-04,
        1.11408126e+00, 6.10319555e-01, 1.68544865e+01],
       [           nan, 4.62359756e-01, 4.62344140e-01, 1.95181783e-04,
        2.19962385e-04, 5.83723631e-05, 1.93975793e-04, 8.65517577e-05,
        1.62397370e-01, 1.62802055e-01, 8.52702215e-05],
       [           nan, 2.34785438e+00, 8.20793211e-01, 1.86102048e-01,
        2.55484730e-01, 1.25649166e+00, 5.41776232e-02, 1.69797778e-01,
        8.91150653e-01, 8.61929059e-01, 2.73648407e-02],
       [           nan, 1.86399981e-01, 2.32090592e-01, 3.38725716e-01,
        2.93454528e-01, 1.95196182e-01, 1.71690449e-01, 8.24425891e-02,
        2.87002828e-02, 3.17514271e-01, 3.88155967e-01],
       [           nan, 4.28708017e-01, 1.84207693e-01, 2.20691875e-01,
        2.77562469e-01, 3.49615157e-01, 3.00025254e-01, 1.49142861e-01,
        5.55958934e-02, 3.77535187e-02, 5.44080853e-01],
       [           nan, 6.51404258e-05, 1.06267067e-05, 1.08483185e-01,
        1.31084071e-01, 6.12673530e-03, 2.86407169e-05, 3.47949835e-05,
        1.08073877e-04, 3.51361283e-04, 5.34706576e-04],
       [           nan, 2.60324210e-01, 7.60723501e-02, 1.89131409e-01,
        9.84028876e-02, 1.49680808e-01, 1.69381708e-01, 5.09373099e-02,
        2.79927440e-02, 1.22439079e-01, 1.56387925e-01],
       [           nan,            nan,            nan,            nan,
                   nan,            nan,            nan,            nan,
                   nan,            nan,            nan],
       [           nan, 5.58600128e-01, 5.56065738e-01, 3.46008599e-01,
        3.79052073e-01, 7.08569229e-01, 3.79988253e-01, 3.75964731e-01,
        2.58975639e-03, 1.50425577e+00, 3.48897845e-01],
       [           nan, 3.52193737e+00, 2.40626359e+00, 1.34040967e-01,
        1.32518172e-01, 1.39978873e-02, 1.44275429e-03, 7.04050530e-04,
        4.87617683e-04, 1.32263839e+00, 1.32290649e+00],
       [           nan, 2.02681409e-03, 2.03517085e-03, 2.01257476e-04,
        2.83534233e-04, 6.94739702e-01, 1.70598502e+00, 2.76057728e-02,
        1.83888156e-02, 3.33575320e-02, 8.18527854e-02],
       [           nan, 2.64513004e-03, 1.00604992e-03, 2.63605452e-05,
        5.96900424e-03, 5.96777303e-03, 2.99748429e-03, 8.16623196e-02,
        7.91179016e-02, 8.49343371e-03, 9.30850208e-03]])

I have not found nans for the non interpolated data, so that makes me think that it may have to do with the interpolation?

malmans2 commented 2 weeks ago

Hi @almarbo,

I believe we explored this issue when working with the previous templates. The NaNs likely occur because some of the cutouts are smaller than the grid you’re interpolating to. To avoid this, you can add a buffer around the cutout area or adjust the interpolation settings (e.g., extend the cutout area by 1 degree in each direction, or remove 1 degree after interpolating)

Try this, which I think makes it clear what's going on:

ds_interpolated["pr"].isel(variance=0).isnull().sum("time").plot(col="model", col_wrap=4)

image

This also explains why different interpolations result in different NaNs, as each interpolation method has its own requirements regarding neighbouring points.

malmans2 commented 2 weeks ago

Also, in case it's not clear, the sum of NaNs is not zero because we did not drop those years where we don't have enough data for the rolling reduction. If you need to drop them, this is the syntax: .dropna("time", how="all")

almarbo commented 2 weeks ago

Thanks @malmans2 ,

totally forgot about this. In fact when we plot we set the extent so that we don't show the edges.

I will continue working on our local machine until we receive the second round of feedback. The methods outlined here should, in principle, be final, but as we discussed in our meeting, feedback can sometimes depend on the results and may require slight adjustments to the methodology, period, area, or variance windows to better address the the user question. I will let you know in case we need further help. Thanks a lot