Unable to merge cubes with cell measures when using a time constraint

[ehogan]

I am unable to merge cubes with cell measures when using a time constraint (discovered in #2048). The problem can be reproduced by executing the code below (I can send the netCDF files directly to the person investigating the issue, if required).

If I try to load the netCDF files using iris.load_cube and I don't use a time constraint, iris.load_cube fails with a ConstraintMismatchError: failed to merge into a single cube. Coordinates in cube.dim_coords differ: time., which is what I expect (I can instead use iris.load then concatenate the resulting cubes).

However, when I use a time constraint, the time coordinate in each cube is demoted to a scalar coordinate, which means it should be possible to merge the cubes. But iris.load_cube fails with a Unequal lengths. Cube dimension 0 => 3; metadata u'area of T grid cells' dimension 0 => 1020.. Dimension 0 corresponds to the time coordinate (the constraint selects 3 months of monthly data) and the area of T grid cells is the long name of the variable specified as the value of the cell_measures attribute of the data variable in the netCDF file I am trying to load. However, the cell measures is described only by the latitude and longitude auxiliary coordinates, so I don't understand why it's trying to compare the time dimension with the cell measures?

% ipython
Python 2.7.6 (default, Jan  3 2014, 16:42:21) 
Type "copyright", "credits" or "license" for more information.

IPython 2.2.0 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: import iris

In [2]: from iris.time import PartialDateTime

In [3]: print iris.__version__
1.10.0-DEV

In [4]: filenames = <filenames>

In [5]: variable_name = 'sst'

In [6]: def time_constraint(cell):
   ...:     return PartialDateTime(2023, 7, 21) <= cell.point <= PartialDateTime(2023, 9, 21)
   ...: 

In [7]: def callback(cube, field, filename):
   ...:     for attribute in cube.attributes.keys():
   ...:         if attribute.startswith('comment') or attribute == 'history':
   ...:             del cube.attributes[attribute]
   ...:     return cube
   ...: 

In [8]: with iris.FUTURE.context(cell_datetime_objects=True):
   ...:     constraint = iris.Constraint(cube_func=lambda cube: cube.var_name == variable_name) & iris.Constraint(time=time_constraint)
   ...:     merged_cube = iris.load_cube(filenames, constraint, callback)
   ...:     
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-17-e62f45e558b9> in <module>()
      1 with iris.FUTURE.context(cell_datetime_objects=True):
      2         constraint = iris.Constraint(cube_func=lambda cube: cube.var_name == variable_name) & iris.Constraint(time=time_constraint)
----> 3         merged_cube = iris.load_cube(filenames, constraint, callback)
      4 

/project/avd/iris/live/testing/iris/__init__.pyc in load_cube(uris, constraint, callback)
    345         raise ValueError('only a single constraint is allowed')
    346 
--> 347     cubes = _load_collection(uris, constraints, callback).merged().cubes()
    348 
    349     try:

/project/avd/iris/live/testing/iris/cube.pyc in merged(self, unique)
    182         """
    183         return _CubeFilterCollection([pair.merged(unique) for pair in
--> 184                                       self.pairs])
    185 
    186 

/project/avd/iris/live/testing/iris/cube.pyc in merged(self, unique)
    124 
    125         """
--> 126         return _CubeFilter(self.constraint, self.cubes.merge(unique))
    127 
    128 

/project/avd/iris/live/testing/iris/cube.pyc in merge(self, unique)
    466         for name in sorted(proto_cubes_by_name, key=_none_sort):
    467             for proto_cube in proto_cubes_by_name[name]:
--> 468                 merged_cubes.extend(proto_cube.merge(unique=unique))
    469 
    470         return merged_cubes

/project/avd/iris/live/testing/iris/_merge.pyc in merge(self, unique)
   1236                         ma.count_masked(merged_data) == 0):
   1237                     merged_data = merged_data.data
-> 1238             merged_cube = self._get_cube(merged_data)
   1239             merged_cubes.append(merged_cube)
   1240 

/project/avd/iris/live/testing/iris/_merge.pyc in _get_cube(self, data)
   1483                               aux_coords_and_dims=aux_coords_and_dims,
   1484                               cell_measures_and_dims=cms_and_dims,
-> 1485                               **kwargs)
   1486 
   1487         # Add on any aux coord factories.

/project/avd/iris/live/testing/iris/cube.pyc in __init__(self, data, standard_name, long_name, var_name, units, attributes, cell_methods, dim_coords_and_dims, aux_coords_and_dims, aux_factories, cell_measures_and_dims)
    771         if cell_measures_and_dims:
    772             for cell_measure, dims in cell_measures_and_dims:
--> 773                 self.add_cell_measure(cell_measure, dims)
    774 
    775     @property

/project/avd/iris/live/testing/iris/cube.pyc in add_cell_measure(self, cell_measure, data_dims)
    979         if self.cell_measures(cell_measure):
    980             raise ValueError('Duplicate cell_measures are not permitted.')
--> 981         data_dims = self._check_multi_dim_metadata(cell_measure, data_dims)
    982         self._cell_measures_and_dims.append([cell_measure, data_dims])
    983         self._cell_measures_and_dims.sort(key=lambda cm_dims:

/project/avd/iris/live/testing/iris/cube.pyc in _check_multi_dim_metadata(self, metadata, data_dims)
    928                     raise ValueError(msg.format(dim, self.shape[dim],
    929                                                 metadata.name(), i,
--> 930                                                 metadata.shape[i]))
    931         elif metadata.shape != (1,):
    932             msg = 'Missing data dimensions for multi-valued {} {!r}'

ValueError: Unequal lengths. Cube dimension 0 => 3; metadata u'area of T grid cells' dimension 0 => 1020.

[trexfeathers]

Have been reviewing some historic records elsewhere - this problem has also been experienced by @rcomer, so this is not an isolated issue.

[github-actions[bot]]

In order to maintain a backlog of relevant issues, we automatically label them as stale after 500 days of inactivity.

If this issue is still important to you, then please comment on this issue and the stale label will be removed.

Otherwise this issue will be automatically closed in 28 days time.

[github-actions[bot]]

This stale issue has been automatically closed due to a lack of community activity.

If you still care about this issue, then please either:

• Re-open this issue, if you have sufficient permissions, or
• Add a comment pinging @SciTools/iris-devs who will re-open on your behalf.

[BenjaminHarrisonMO]

I have the same issue. The responses acknowledge the issue has been highlighted before but does provide any solutions or alternatives.

[HGWright]

@SciTools/peloton We have not forgotten about this, there are many merge/concatenate problems and we hope to be able to get to them in a future sprint.

[berndbecker]

cubelist or cube? return cube, not cubelist! 0: sea_water_potential_temperature / (degree_C) (-- : 3606; -- : 4322) 1: sea_water_potential_temperature / (degree_C) (-- : 3606; -- : 4322) 2: sea_water_potential_temperature / (degree_C) (-- : 3606; -- : 4322) 3: sea_water_potential_temperature / (degree_C) (-- : 3606; -- : 4322) 4: sea_water_potential_temperature / (degree_C) (-- : 3606; -- : 4322) Unequal lengths. Cube dimension 0 => 5; metadata 'cell_area' dimension 0 => 3606. Traceback (most recent call last): File "/net/home/h03/orca12/repositories/BB-FrontFinding/code_base/front_finder.py", line 335, in make_extended_mask_cube d3mask_cube = slice_list_mask.merge_cube() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/h03/orca12/.conda/envs/geovista-env/lib/python3.11/site-packages/iris/cube.py", line 442, in merge_cube (merged_cube,) = proto_cube.merge() ^^^^^^^^^^^^^^^^^^ File "/home/h03/orca12/.conda/envs/geovista-env/lib/python3.11/site-packages/iris/_merge.py", line 1325, in merge merged_cube = self._get_cube(merged_data) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/h03/orca12/.conda/envs/geovista-env/lib/python3.11/site-packages/iris/_merge.py", line 1606, in _get_cube cube = iris.cube.Cube( ^^^^^^^^^^^^^^^ File "/home/h03/orca12/.conda/envs/geovista-env/lib/python3.11/site-packages/iris/cube.py", line 1031, in init self.add_cell_measure(cell_measure, dims) File "/home/h03/orca12/.conda/envs/geovista-env/lib/python3.11/site-packages/iris/cube.py", line 1336, in add_cell_measure data_dims = self._check_multi_dim_metadata(cell_measure, data_dims) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/h03/orca12/.conda/envs/geovista-env/lib/python3.11/site-packages/iris/cube.py", line 1218, in _check_multi_dim_metadata raise iris.exceptions.CannotAddError( iris.exceptions.CannotAddError: Unequal lengths. Cube dimension 0 => 5; metadata 'cell_area' dimension 0 => 3606.

The same issue still presents itself when trying to merge horizontal slices of orca12 ocean data. Is there a work around available anywhere? Cheers!