`mask_landsea` fails when using 2D fx data on 4D cube data

[valeriupredoi]

Minimal recipe

documentation:
  description: |
    Various example preprocessors.
  authors:
    - righi_mattia
  title: Test recipe for ESMValCore preprocessors.
  maintainer:
    - righi_mattia
  projects:
    - c3s-magic
datasets:
  - {dataset: bcc-csm1-1, project: CMIP5, exp: historical, ensemble: r1i1p1}
preprocessors:
  preprocessor_1:
    extract_levels:
      levels: bcc-csm1-1
      scheme: linear
    mask_landsea:
      mask_out: land
diagnostics:
  diagnostic_1: &diag
    description: Preprocessor test diagnostic.
    variables:
      ta:
        preprocessor: preprocessor_1
        mip: Amon
        start_year: 2000
        end_year: 2002
      orog:
        mip: fx
      sftlf:
        mip: fx
    scripts: null

Trace (full)

2021-10-08 12:44:56,533 UTC [25166] ERROR   Failed to run mask_landsea(air_temperature / (K)               (time: 36; air_pressure: 17; latitude: 64; longitude: 128)
    Dimension coordinates:
        time                             x                 -             -              -
        air_pressure                     -                 x             -              -
        latitude                         -                 -             x              -
        longitude                        -                 -             -              x
    Ancillary variables:
        land_area_fraction               -                 -             x              x
    Cell methods:
        mean                        time (20 mintues)
    Attributes:
        Conventions                 CF-1.4
        associated_files            baseURL: http://cmip-pcmdi.llnl.gov/CMIP5/dataLocation gridspecFile: gridspec_atmos_fx_bcc-csm1-1_historical_r0i0p0.nc...
        branch_time                 470.0
        cmor_version                2.5.6
        contact                     Dr. Tongwen Wu (twwu@cma.gov.cn)
        experiment                  historical
        experiment_id               historical
        forcing                     Nat Ant GHG SD Oz Sl Vl SS Ds BC OC
        frequency                   mon
        initialization_method       1
        institute_id                BCC
        institution                 Beijing Climate Center(BCC),China Meteorological Administration,China
        model_id                    bcc-csm1-1
        modeling_realm              atmos
        original_name               T
        parent_experiment           pre-industrial control
        parent_experiment_id        piControl
        parent_experiment_rip       r1i1p1
        physics_version             1
        product                     output
        project_id                  CMIP5
        realization                 1
        source                      bcc-csm1-1:atmosphere:  BCC_AGCM2.1 (T42L26); land: BCC_AVIM1.0;ocean:...
        table_id                    Table Amon (11 April 2011) 1cfdc7322cf2f4a32614826fab42c1ab
        title                       bcc-csm1-1 model output prepared for CMIP5 historical, {'mask_out': 'land'})
2021-10-08 12:44:56,682 UTC [25155] INFO    Maximum memory used (estimate): 0.7 GB
2021-10-08 12:44:56,684 UTC [25155] INFO    Sampled every second. It may be inaccurate if short but high spikes in memory consumption occur.
2021-10-08 12:44:56,685 UTC [25155] ERROR   Program terminated abnormally, see stack trace below for more information:
multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/home/valeriu/miniconda3-fresh/envs/esmvaltool-stock/lib/python3.9/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/home/valeriu/ESMValCore/esmvalcore/_task.py", line 780, in _run_task
    output_files = task.run()
  File "/home/valeriu/ESMValCore/esmvalcore/_task.py", line 253, in run
    self.output_files = self._run(input_files)
  File "/home/valeriu/ESMValCore/esmvalcore/preprocessor/__init__.py", line 489, in _run
    product.apply(step, self.debug)
  File "/home/valeriu/ESMValCore/esmvalcore/preprocessor/__init__.py", line 356, in apply
    self.cubes = preprocess(self.cubes, step, **self.settings[step])
  File "/home/valeriu/ESMValCore/esmvalcore/preprocessor/__init__.py", line 302, in preprocess
    result.append(_run_preproc_function(function, item, settings))
  File "/home/valeriu/ESMValCore/esmvalcore/preprocessor/__init__.py", line 285, in _run_preproc_function
    return function(items, **kwargs)
  File "/home/valeriu/ESMValCore/esmvalcore/preprocessor/_mask.py", line 118, in mask_landsea
    fx_cube.data = da.broadcast_to(fx_cube.core_data(), cube.shape)
  File "/home/valeriu/miniconda3-fresh/envs/esmvaltool-stock/lib/python3.9/site-packages/iris/coords.py", line 740, in data
    self._values = data
  File "/home/valeriu/miniconda3-fresh/envs/esmvaltool-stock/lib/python3.9/site-packages/iris/coords.py", line 241, in _values
    self._values_dm.data = values
  File "/home/valeriu/miniconda3-fresh/envs/esmvaltool-stock/lib/python3.9/site-packages/iris/_data_manager.py", line 259, in data
    raise ValueError(emsg.format(self.shape, data.shape))
ValueError: Require data with shape (64, 128), got (36, 17, 64, 128).
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/valeriu/ESMValCore/esmvalcore/_main.py", line 479, in run
    fire.Fire(ESMValTool())
  File "/home/valeriu/miniconda3-fresh/envs/esmvaltool-stock/lib/python3.9/site-packages/fire/core.py", line 141, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/home/valeriu/miniconda3-fresh/envs/esmvaltool-stock/lib/python3.9/site-packages/fire/core.py", line 466, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/home/valeriu/miniconda3-fresh/envs/esmvaltool-stock/lib/python3.9/site-packages/fire/core.py", line 681, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/home/valeriu/ESMValCore/esmvalcore/_main.py", line 423, in run
    process_recipe(recipe_file=recipe, config_user=cfg)
  File "/home/valeriu/ESMValCore/esmvalcore/_main.py", line 125, in process_recipe
    recipe.run()
  File "/home/valeriu/ESMValCore/esmvalcore/_recipe.py", line 1557, in run
    self.tasks.run(max_parallel_tasks=self._cfg['max_parallel_tasks'])
  File "/home/valeriu/ESMValCore/esmvalcore/_task.py", line 709, in run
    self._run_parallel(max_parallel_tasks)
  File "/home/valeriu/ESMValCore/esmvalcore/_task.py", line 752, in _run_parallel
    _copy_results(task, running[task])
  File "/home/valeriu/ESMValCore/esmvalcore/_task.py", line 775, in _copy_results
    task.output_files, task.products = future.get()
  File "/home/valeriu/miniconda3-fresh/envs/esmvaltool-stock/lib/python3.9/multiprocessing/pool.py", line 771, in get
    raise self._value
  File "/home/valeriu/miniconda3-fresh/envs/esmvaltool-stock/lib/python3.9/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/home/valeriu/ESMValCore/esmvalcore/_task.py", line 780, in _run_task
    output_files = task.run()
  File "/home/valeriu/ESMValCore/esmvalcore/_task.py", line 253, in run
    self.output_files = self._run(input_files)
  File "/home/valeriu/ESMValCore/esmvalcore/preprocessor/__init__.py", line 489, in _run
    product.apply(step, self.debug)
  File "/home/valeriu/ESMValCore/esmvalcore/preprocessor/__init__.py", line 356, in apply
    self.cubes = preprocess(self.cubes, step, **self.settings[step])
  File "/home/valeriu/ESMValCore/esmvalcore/preprocessor/__init__.py", line 302, in preprocess
    result.append(_run_preproc_function(function, item, settings))
  File "/home/valeriu/ESMValCore/esmvalcore/preprocessor/__init__.py", line 285, in _run_preproc_function
    return function(items, **kwargs)
  File "/home/valeriu/ESMValCore/esmvalcore/preprocessor/_mask.py", line 118, in mask_landsea
    fx_cube.data = da.broadcast_to(fx_cube.core_data(), cube.shape)
  File "/home/valeriu/miniconda3-fresh/envs/esmvaltool-stock/lib/python3.9/site-packages/iris/coords.py", line 740, in data
    self._values = data
  File "/home/valeriu/miniconda3-fresh/envs/esmvaltool-stock/lib/python3.9/site-packages/iris/coords.py", line 241, in _values
    self._values_dm.data = values
  File "/home/valeriu/miniconda3-fresh/envs/esmvaltool-stock/lib/python3.9/site-packages/iris/_data_manager.py", line 259, in data
    raise ValueError(emsg.format(self.shape, data.shape))
ValueError: Require data with shape (64, 128), got (36, 17, 64, 128).
2021-10-08 12:44:56,690 UTC [25155] INFO    
If you have a question or need help, please start a new discussion on https://github.com/ESMValGroup/ESMValTool/discussions
If you suspect this is a bug, please open an issue on https://github.com/ESMValGroup/ESMValTool/issues
To make it easier to find out what the problem is, please consider attaching the files run/recipe_*.yml and run/main_log_debug.txt from the output directory.

Issue in a nutshell

fx data has shape: (64, 128) and data cube has shape: (36, 17, 64, 128); in _mask.py l.117 (not 118 as in the trace, I added a line at pos 117 inspect the two cubes)

fx_cube.data = da.broadcast_to(fx_cube.core_data(), cube.shape)

fails to broadcast fx data onto cube data.

[valeriupredoi]

iris version: iris 3.1.0 py39hf3d152e_1 conda-forge

[valeriupredoi]

looks like iris is not permitting reshaping: in lib/python3.9/site-packages/iris/_data_manager.py l.259

        if init_done and self.shape != data.shape:
            # The _ONLY_ data reshape permitted is converting a 0-dimensional
            # array i.e. self.shape == () into a 1-dimensional array of length
            # one i.e. data.shape == (1,)
            if self.shape or data.shape != (1,):
                emsg = "Require data with shape {!r}, got {!r}."
                raise ValueError(emsg.format(self.shape, data.shape))

[zklaus]

Well, yes, the shape of a cube can not be changed like this.

[valeriupredoi]

yip, but broadcasting the fx data onto the shape of the data cube should be allowed no?

[zklaus]

Yes. I think a straight-forward fix is as follows: Since fx_cube.data effectively is only a temporary variable here, just use it as such, i.e. s/fx_cube.data/fx_cube_data/ in L117-118. Beware that the same problem happens a little later in the file, so perhaps best to search for broadcast_to. Also, we should add a test with different size cubes.

[valeriupredoi]

right on, Klaus! work started in https://github.com/ESMValGroup/ESMValCore/pull/1350