ClimateImpactLab / downscaleCMIP6

Downscaling & bias correction of CMIP6 tasmin, tasmax, and pr for the R/CIL GDPCIR project
MIT License
136 stars 33 forks source link

KACE-1-0-G tasmin fails at cleaning step because of nans and values out of range #588

Closed emileten closed 2 years ago

emileten commented 2 years ago

Workflow : https://argo.cildc6.org/workflows/default/e2e-kace-1-0-g-tasmin-zqtkc?tab=workflow&nodePanelView=inputs-outputs

ssp126 and ssp245 fail :

Validating gs://clean-b1dbca25/cmip6/ScenarioMIP/NIMS-KMA/KACE-1-0-G/ssp245/r1i1p1f1/day/tasmin/gr/v20200317.zarr
Traceback (most recent call last):
  File "/argo/staging/script", line 47, in <module>
    tasks = dask.compute(*tasks)
  File "/opt/conda/lib/python3.9/site-packages/dask/base.py", line 570, in compute
    results = schedule(dsk, keys, **kwargs)
  File "/opt/conda/lib/python3.9/site-packages/dask/threaded.py", line 79, in get
    results = get_async(
  File "/opt/conda/lib/python3.9/site-packages/dask/local.py", line 507, in get_async
    raise_exception(exc, tb)
  File "/opt/conda/lib/python3.9/site-packages/dask/local.py", line 315, in reraise
    raise exc
  File "/opt/conda/lib/python3.9/site-packages/dask/local.py", line 220, in execute_task
    result = _execute_task(task, data)
  File "/opt/conda/lib/python3.9/site-packages/dask/core.py", line 119, in _execute_task
    return func(*(_execute_task(a, cache) for a in args))
  File "/argo/staging/script", line 32, in clear_memory_intensive_tests
    _test_temp_range(d, v)
  File "/opt/dodola/dodola/core.py", line 779, in _test_temp_range
    assert (ds[var].min() > 130) and (
AssertionError: tasmin values are invalid

ssp370 also fails :

Validating gs://clean-b1dbca25/cmip6/ScenarioMIP/NIMS-KMA/KACE-1-0-G/ssp370/r1i1p1f1/day/tasmin/gr/v20200317.zarr
Traceback (most recent call last):
  File "/argo/staging/script", line 47, in <module>
    tasks = dask.compute(*tasks)
  File "/opt/conda/lib/python3.9/site-packages/dask/base.py", line 570, in compute
    results = schedule(dsk, keys, **kwargs)
  File "/opt/conda/lib/python3.9/site-packages/dask/threaded.py", line 79, in get
    results = get_async(
  File "/opt/conda/lib/python3.9/site-packages/dask/local.py", line 507, in get_async
    raise_exception(exc, tb)
  File "/opt/conda/lib/python3.9/site-packages/dask/local.py", line 315, in reraise
    raise exc
  File "/opt/conda/lib/python3.9/site-packages/dask/local.py", line 220, in execute_task
    result = _execute_task(task, data)
  File "/opt/conda/lib/python3.9/site-packages/dask/core.py", line 119, in _execute_task
    return func(*(_execute_task(a, cache) for a in args))
  File "/argo/staging/script", line 29, in clear_memory_intensive_tests
    _test_for_nans(d, v)
  File "/opt/dodola/dodola/core.py", line 701, in _test_for_nans
    assert ds[var].isnull().sum() == 0, "there are nans!"
AssertionError: there are nans!

Blocks progress on #573

emileten commented 2 years ago

Given that we don't know what to do with nans, we decided to abandon this model.