search

xCDAT / xcdat

An extension of xarray for climate data analysis on structured grids.
https://xcdat.readthedocs.io/en/latest/
Apache License 2.0
101 stars 11 forks source link

[Bug]: Regrid2 breaks in `_preserve_bounds` because `_regrid` incorrectly transposes `output_data` shape, which doesn't align with original dim order used by bounds #625

Closed tomvothecoder closed 1 month ago

tomvothecoder commented 3 months ago

What happened?

There is an issue where transposing the shape of data causes misalignment with the original dimension order. When we attempt to preserve bounds, the original dimension order is still being used which does not align with the transposed dimension order.

Example

Let's say we have ds_a with an original dimension of shape of (lat: 1, lev: 2, lon: 3).

After transposing the data, output_data become shape=(2, 1, 3).

https://github.com/xCDAT/xcdat/blob/d0e33aaea04ffeca0301cd7773e4f981c7957653/xcdat/regridder/regrid2.py#L164-L172

We recreate the xr.DataArray (output_da) using output_data and dims. As mentioned above, the shape of output_data is now transposed, but it does not align with the original dims. This results in output_da having a new dimension shape of (lat: 2, lev: 1, lon: 3) https://github.com/xCDAT/xcdat/blob/d0e33aaea04ffeca0301cd7773e4f981c7957653/xcdat/regridder/regrid2.py#L184-L197

When we attempt to preserve lat_bnds in _preserve_bounds(), the original lat_bnds still has has a shape of (lat: 1, lev: 2, lon: 3), which does not align with (lat: 2, lev: 1, lon: 3).

https://github.com/xCDAT/xcdat/blob/d0e33aaea04ffeca0301cd7773e4f981c7957653/xcdat/regridder/base.py#L48-L58

This results in ValueError: cannot reindex or align along dimension 'lat' because of conflicting dimension sizes: {1, 2} (note: an index is found along that dimension with size=1).

What did you expect to happen? Are there are possible answers you came across?

Maybe output_data shape should align with the original dim order or we should also transpose the shape of the bounds?

Minimal Complete Verifiable Example (MVCE)

import numpy as np
import xarray as xr
import xcdat as xc

# 1. Create two datasets with dimensions (lat=1, lev=2, lon=3)
ds_a = xr.Dataset(
    {
        "lat": xr.DataArray(
            name="lat", dims="lat", data=[0], attrs={"axis": "Y", "bounds": "lat_bnds"}
        ),
        "lev": xr.DataArray(name="lev", dims="lev", data=[0, 1], attrs={"axis": "Z"}),
        "lon": xr.DataArray(
            name="lon", dims="lon", data=[0, 1, 2], attrs={"axis": "X"}
        ),
    }
)
ds_a["T"] = xr.DataArray(name="T", dims=["lat", "lev", "lon"], data=np.zeros((1, 2, 3)))
ds_a = ds_a.bounds.add_missing_bounds(axes=["X", "Y", "Z"])

# Manually add lat_bnds because xCDAT does not support adding bounds for
# singleton coords (yet)
ds_a["lat_bnds"] = xr.DataArray(name="lat_bnds", dims=["lat", "bnds"], data=[[-1, 1]])

ds_b = ds_a.copy()

# 2. Regrid dataset b to a using regrid2, bilinear
output_grid = ds_a.regridder.grid
ds_b_regrid = ds_b.regridder.horizontal(
    "T", output_grid, tool="regrid2", method="bilinear"
)
# ValueError: cannot reindex or align along dimension 'lat' because of conflicting dimension sizes: {1, 2} (note: an index is found along that dimension with size=1)

# 3. Breaks because `_regrid` incorrectly reorders dimensions
# Resulting in the `output_data` shape to become (lat=2, lev=1, lon=3).
˘

Relevant log output

ValueError                                Traceback (most recent call last)
File /global/u2/v/vo13/E3SM-Project/e3sm_diags/auxiliary_tools/cdat_regression_testing/657-meridional-mean-2d/657_meridional_mean_2d_run_script.py:19
     16 ds_b = ds_a.copy()
     18 output_grid = ds_a.regridder.grid
---> 19 ds_b_regrid = ds_b.regridder.horizontal(
     20     "T", output_grid, tool="regrid2", method="bilinear"
     21 )

File /global/u2/v/vo13/mambaforge/envs/e3sm_diags_dev_nompi_659/lib/python3.10/site-packages/xcdat/regridder/accessor.py:324, in RegridderAccessor.horizontal(self, data_var, output_grid, tool, **options)
    322 input_grid = _get_input_grid(self._ds, data_var, ["X", "Y"])
    323 regridder = regrid_tool(input_grid, output_grid, **options)
--> 324 output_ds = regridder.horizontal(data_var, self._ds)
    326 return output_ds

File /global/u2/v/vo13/mambaforge/envs/e3sm_diags_dev_nompi_659/lib/python3.10/site-packages/xcdat/regridder/regrid2.py:85, in Regrid2Regridder.horizontal(self, data_var, ds)
     79     input_data_var = input_data_var.where(src_mask != 0.0, masked_value)
     81 output_data = _regrid(
     82     input_data_var, src_lat_bnds, src_lon_bnds, dst_lat_bnds, dst_lon_bnds
     83 )
---> 85 output_ds = _build_dataset(
     86     ds,
     87     data_var,
     88     output_data,
     89     dst_lat_bnds,
     90     dst_lon_bnds,
     91     self._input_grid,
     92     self._output_grid,
     93 )
     95 return output_ds

File /global/u2/v/vo13/mambaforge/envs/e3sm_diags_dev_nompi_659/lib/python3.10/site-packages/xcdat/regridder/regrid2.py:206, in _build_dataset(ds, data_var, output_data, dst_lat_bnds, dst_lon_bnds, input_grid, output_grid)
    199 output_data_vars[data_var] = output_da
    201 output_ds = xr.Dataset(
    202     output_data_vars,
    203     attrs=input_grid.attrs.copy(),
    204 )
--> 206 output_ds = _preserve_bounds(ds, output_grid, output_ds, ["X", "Y"])
    208 return output_ds

File /global/u2/v/vo13/mambaforge/envs/e3sm_diags_dev_nompi_659/lib/python3.10/site-packages/xcdat/regridder/base.py:56, in _preserve_bounds(input_ds, output_grid, output_ds, drop_axis)
     54         else:
     55             if bnds.name not in output_ds:
---> 56                 output_ds[bnds.name] = bnds.copy()
     58 return output_ds

File /global/u2/v/vo13/mambaforge/envs/e3sm_diags_dev_nompi_659/lib/python3.10/site-packages/xarray/core/dataset.py:1603, in Dataset.__setitem__(self, key, value)
   1598     if isinstance(value, Dataset):
   1599         raise TypeError(
   1600             "Cannot assign a Dataset to a single key - only a DataArray or Variable "
   1601             "object can be stored under a single key."
   1602         )
-> 1603     self.update({key: value})
   1605 elif utils.iterable_of_hashable(key):
   1606     keylist = list(key)

File /global/u2/v/vo13/mambaforge/envs/e3sm_diags_dev_nompi_659/lib/python3.10/site-packages/xarray/core/dataset.py:5617, in Dataset.update(self, other)
   5581 def update(self, other: CoercibleMapping) -> Self:
   5582     """Update this dataset's variables with those from another dataset.
   5583 
   5584     Just like :py:meth:`dict.update` this is a in-place operation.
   (...)
   5615     Dataset.merge
   5616     """
-> 5617     merge_result = dataset_update_method(self, other)
   5618     return self._replace(inplace=True, **merge_result._asdict())

File /global/u2/v/vo13/mambaforge/envs/e3sm_diags_dev_nompi_659/lib/python3.10/site-packages/xarray/core/merge.py:1075, in dataset_update_method(dataset, other)
   1072             if coord_names:
   1073                 other[key] = value.drop_vars(coord_names)
-> 1075 return merge_core(
   1076     [dataset, other],
   1077     priority_arg=1,
   1078     indexes=dataset.xindexes,
   1079     combine_attrs="override",
   1080 )

File /global/u2/v/vo13/mambaforge/envs/e3sm_diags_dev_nompi_659/lib/python3.10/site-packages/xarray/core/merge.py:711, in merge_core(objects, compat, join, combine_attrs, priority_arg, explicit_coords, indexes, fill_value, skip_align_args)
    708 skip_align_objs = [(pos, objects.pop(pos)) for pos in skip_align_args]
    710 coerced = coerce_pandas_values(objects)
--> 711 aligned = deep_align(
    712     coerced, join=join, copy=False, indexes=indexes, fill_value=fill_value
    713 )
    715 for pos, obj in skip_align_objs:
    716     aligned.insert(pos, obj)

File /global/u2/v/vo13/mambaforge/envs/e3sm_diags_dev_nompi_659/lib/python3.10/site-packages/xarray/core/alignment.py:946, in deep_align(objects, join, copy, indexes, exclude, raise_on_invalid, fill_value)
    943     else:
    944         out.append(variables)
--> 946 aligned = align(
    947     *targets,
    948     join=join,
    949     copy=copy,
    950     indexes=indexes,
    951     exclude=exclude,
...
    477         f"cannot reindex or align along dimension {dim!r} "
    478         f"because of conflicting dimension sizes: {sizes!r}" + add_err_msg
    479     )

ValueError: cannot reindex or align along dimension 'lat' because of conflicting dimension sizes: {1, 2} (note: an index is found along that dimension with size=1)

Anything else we need to know?

No response

Environment

Latest main xCDAT

tomvothecoder commented 3 months ago

Have you come across this issue @lee1043?

lee1043 commented 3 months ago

@tomvothecoder this is interesting. No, I haven't. Maybe because my data mostly have [time, lat, lon] or [lat, lon] dimensions.

tomvothecoder commented 3 months ago

@tomvothecoder this is interesting. No, I haven't. Maybe because my data mostly have [time, lat, lon] or [lat, lon] dimensions.

It could be that the logic in _regrid expects a certain order of dimensions, or it doesn't consider an order of [lat, lev, lon].

As a path forward around this issue, I decided to use xESMF with conservative_normed instead. I got very close results when comparing to CDAT's regrid2 for all of the datasets I tested (PR). 

Comparing:
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/657-meridional-mean-2d/meridional_mean_2d/ERA5/ERA5-OMEGA-ANN-global_ref.nc
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/main/meridional_mean_2d/ERA5/ERA5-OMEGA-ANN-global_ref.nc
    * var_key: OMEGA

Not equal to tolerance rtol=1e-05, atol=0

Mismatched elements: 6 / 24480 (0.0245%)
Max absolute difference: 4.56288177e-07
Max relative difference: 5.34042702e-05
 x: array([[ 0.784679,  0.796702,  0.792179, ...,  0.804814,  0.773629,
         0.774743],
       [ 1.379984,  1.417249,  1.4173  , ...,  1.432473,  1.391159,...
 y: array([[ 0.784679,  0.796702,  0.792179, ...,  0.804814,  0.773629,
         0.774743],
       [ 1.379984,  1.417249,  1.4173  , ...,  1.432473,  1.391159,...
Comparing:
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/657-meridional-mean-2d/meridional_mean_2d/ERA5/ERA5-OMEGA-ANN-global_test.nc
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/main/meridional_mean_2d/ERA5/ERA5-OMEGA-ANN-global_test.nc
    * var_key: OMEGA
    * All close and within relative tolerance (1e-05)
Comparing:
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/657-meridional-mean-2d/meridional_mean_2d/ERA5/ERA5-OMEGA-JJA-global_ref.nc
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/main/meridional_mean_2d/ERA5/ERA5-OMEGA-JJA-global_ref.nc
    * var_key: OMEGA

Not equal to tolerance rtol=1e-05, atol=0

Mismatched elements: 3 / 24480 (0.0123%)
Max absolute difference: 5.97006945e-07
Max relative difference: 1.49039703e-05
 x: array([[1.08258 , 1.069207, 1.07484 , ..., 1.090696, 1.087342, 1.09467 ],
       [1.563664, 1.593989, 1.628474, ..., 1.553196, 1.556833, 1.570584],
       [2.220292, 2.307285, 2.364905, ..., 2.215879, 2.220069, 2.228939],...
 y: array([[1.08258 , 1.069207, 1.07484 , ..., 1.090696, 1.087342, 1.09467 ],
       [1.563664, 1.593989, 1.628474, ..., 1.553196, 1.556833, 1.570584],
       [2.220292, 2.307285, 2.364905, ..., 2.215879, 2.220069, 2.228939],...
Comparing:
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/657-meridional-mean-2d/meridional_mean_2d/ERA5/ERA5-OMEGA-JJA-global_test.nc
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/main/meridional_mean_2d/ERA5/ERA5-OMEGA-JJA-global_test.nc
    * var_key: OMEGA

Not equal to tolerance rtol=1e-05, atol=0

Mismatched elements: 2 / 6120 (0.0327%)
Max absolute difference: 3.39774408e-07
Max relative difference: 0.00015039
 x: array([[ 1.150119,  1.168407,  1.16202 , ...,  1.153911,  1.138019,
         1.152851],
       [ 1.567948,  1.599667,  1.601973, ...,  1.42647 ,  1.435826,...
 y: array([[ 1.150119,  1.168407,  1.16202 , ...,  1.153911,  1.138019,
         1.152851],
       [ 1.567948,  1.599667,  1.601973, ...,  1.42647 ,  1.435826,...
Comparing:
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/657-meridional-mean-2d/meridional_mean_2d/ERA5/ERA5-RELHUM-ANN-global_ref.nc
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/main/meridional_mean_2d/ERA5/ERA5-RELHUM-ANN-global_ref.nc
    * var_key: RELHUM
    * All close and within relative tolerance (1e-05)
Comparing:
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/657-meridional-mean-2d/meridional_mean_2d/ERA5/ERA5-RELHUM-ANN-global_test.nc
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/main/meridional_mean_2d/ERA5/ERA5-RELHUM-ANN-global_test.nc
    * var_key: RELHUM
    * All close and within relative tolerance (1e-05)
Comparing:
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/657-meridional-mean-2d/meridional_mean_2d/ERA5/ERA5-RELHUM-JJA-global_ref.nc
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/main/meridional_mean_2d/ERA5/ERA5-RELHUM-JJA-global_ref.nc
    * var_key: RELHUM
    * All close and within relative tolerance (1e-05)
Comparing:
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/657-meridional-mean-2d/meridional_mean_2d/ERA5/ERA5-RELHUM-JJA-global_test.nc
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/main/meridional_mean_2d/ERA5/ERA5-RELHUM-JJA-global_test.nc
    * var_key: RELHUM
    * All close and within relative tolerance (1e-05)
Comparing:
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/657-meridional-mean-2d/meridional_mean_2d/ERA5/ERA5-T-ANN-global_ref.nc
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/main/meridional_mean_2d/ERA5/ERA5-T-ANN-global_ref.nc
    * var_key: T
    * All close and within relative tolerance (1e-05)
Comparing:
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/657-meridional-mean-2d/meridional_mean_2d/ERA5/ERA5-T-ANN-global_test.nc
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/main/meridional_mean_2d/ERA5/ERA5-T-ANN-global_test.nc
    * var_key: T
    * All close and within relative tolerance (1e-05)
Comparing:
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/657-meridional-mean-2d/meridional_mean_2d/ERA5/ERA5-T-JJA-global_ref.nc
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/main/meridional_mean_2d/ERA5/ERA5-T-JJA-global_ref.nc
    * var_key: T
    * All close and within relative tolerance (1e-05)
Comparing:
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/657-meridional-mean-2d/meridional_mean_2d/ERA5/ERA5-T-JJA-global_test.nc
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/main/meridional_mean_2d/ERA5/ERA5-T-JJA-global_test.nc
    * var_key: T
    * All close and within relative tolerance (1e-05)
Comparing:
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/657-meridional-mean-2d/meridional_mean_2d/ERA5/ERA5-U-ANN-global_ref.nc
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/main/meridional_mean_2d/ERA5/ERA5-U-ANN-global_ref.nc
    * var_key: U

Not equal to tolerance rtol=1e-05, atol=0

Mismatched elements: 1 / 24480 (0.00408%)
Max absolute difference: 8.52794937e-08
Max relative difference: 1.73621834e-05
 x: array([[10.269112, 10.260021, 10.25072 , ..., 10.296745, 10.28736 ,
        10.278145],
       [12.365973, 12.353776, 12.341225, ..., 12.402582, 12.390261,...
 y: array([[10.269112, 10.260021, 10.25072 , ..., 10.296745, 10.28736 ,
        10.278145],
       [12.365973, 12.353776, 12.341225, ..., 12.402582, 12.390261,...
Comparing:
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/657-meridional-mean-2d/meridional_mean_2d/ERA5/ERA5-U-ANN-global_test.nc
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/main/meridional_mean_2d/ERA5/ERA5-U-ANN-global_test.nc
    * var_key: U

Not equal to tolerance rtol=1e-05, atol=0

Mismatched elements: 1 / 6120 (0.0163%)
Max absolute difference: 1.45889873e-07
Max relative difference: 1.08735566e-05
 x: array([[ 9.712424,  9.692257,  9.675111, ...,  9.79681 ,  9.779955,
         9.765068],
       [12.482069, 12.457917, 12.434379, ..., 12.581674, 12.560391,...
 y: array([[ 9.712424,  9.692257,  9.675112, ...,  9.79681 ,  9.779955,
         9.765068],
       [12.482069, 12.457917, 12.434379, ..., 12.581674, 12.560391,...
Comparing:
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/657-meridional-mean-2d/meridional_mean_2d/ERA5/ERA5-U-JJA-global_ref.nc
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/main/meridional_mean_2d/ERA5/ERA5-U-JJA-global_ref.nc
    * var_key: U

Not equal to tolerance rtol=1e-05, atol=0

Mismatched elements: 1 / 24480 (0.00408%)
Max absolute difference: 8.38617442e-08
Max relative difference: 1.96239722e-05
 x: array([[ 8.311289,  8.302058,  8.292533, ...,  8.338692,  8.329687,
         8.320503],
       [10.096993, 10.085039, 10.072245, ..., 10.131809, 10.120227,...
 y: array([[ 8.311289,  8.302058,  8.292533, ...,  8.338692,  8.329687,
         8.320503],
       [10.096994, 10.085039, 10.072245, ..., 10.131809, 10.120227,...
Comparing:
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/657-meridional-mean-2d/meridional_mean_2d/ERA5/ERA5-U-JJA-global_test.nc
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/main/meridional_mean_2d/ERA5/ERA5-U-JJA-global_test.nc
    * var_key: U
    * All close and within relative tolerance (1e-05)
Comparing:
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/657-meridional-mean-2d/meridional_mean_2d/MERRA2/MERRA2-OMEGA-ANN-global_ref.nc
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/main/meridional_mean_2d/MERRA2/MERRA2-OMEGA-ANN-global_ref.nc
    * var_key: OMEGA

Not equal to tolerance rtol=1e-05, atol=0

Mismatched elements: 2 / 9792 (0.0204%)
Max absolute difference: 3.15701861e-07
Max relative difference: 2.42041494e-05
 x: array([[0.914233, 0.906507, 0.900358, ..., 0.954717, 0.945675, 0.928962],
       [1.617818, 1.601587, 1.580476, ..., 1.631384, 1.64521 , 1.638424],
       [2.321404, 2.296666, 2.260593, ..., 2.308052, 2.344744, 2.347885],...
 y: array([[0.914233, 0.906507, 0.900358, ..., 0.954717, 0.945675, 0.928962],
       [1.617818, 1.601587, 1.580476, ..., 1.631384, 1.64521 , 1.638424],
       [2.321404, 2.296666, 2.260593, ..., 2.308052, 2.344744, 2.347885],...
Comparing:
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/657-meridional-mean-2d/meridional_mean_2d/MERRA2/MERRA2-OMEGA-ANN-global_test.nc
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/main/meridional_mean_2d/MERRA2/MERRA2-OMEGA-ANN-global_test.nc
    * var_key: OMEGA
    * All close and within relative tolerance (1e-05)
Comparing:
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/657-meridional-mean-2d/meridional_mean_2d/MERRA2/MERRA2-OMEGA-JJA-global_ref.nc
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/main/meridional_mean_2d/MERRA2/MERRA2-OMEGA-JJA-global_ref.nc
    * var_key: OMEGA

Not equal to tolerance rtol=1e-05, atol=0

Mismatched elements: 2 / 9792 (0.0204%)
Max absolute difference: 6.93198768e-07
Max relative difference: 5.40329309e-05
 x: array([[ 1.283442,  1.27651 ,  1.25776 , ...,  1.292833,  1.29537 ,
         1.287421],
       [ 1.93963 ,  1.950937,  1.957762, ...,  1.901146,  1.91872 ,...
 y: array([[ 1.283442,  1.27651 ,  1.25776 , ...,  1.292833,  1.29537 ,
         1.287421],
       [ 1.93963 ,  1.950937,  1.957762, ...,  1.901146,  1.91872 ,...
Comparing:
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/657-meridional-mean-2d/meridional_mean_2d/MERRA2/MERRA2-OMEGA-JJA-global_test.nc
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/main/meridional_mean_2d/MERRA2/MERRA2-OMEGA-JJA-global_test.nc
    * var_key: OMEGA

Not equal to tolerance rtol=1e-05, atol=0

Mismatched elements: 2 / 6120 (0.0327%)
Max absolute difference: 3.39774408e-07
Max relative difference: 0.00015039
 x: array([[ 1.150119,  1.168407,  1.16202 , ...,  1.153911,  1.138019,
         1.152851],
       [ 1.567948,  1.599667,  1.601973, ...,  1.42647 ,  1.435826,...
 y: array([[ 1.150119,  1.168407,  1.16202 , ...,  1.153911,  1.138019,
         1.152851],
       [ 1.567948,  1.599667,  1.601973, ...,  1.42647 ,  1.435826,...
Comparing:
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/657-meridional-mean-2d/meridional_mean_2d/MERRA2/MERRA2-RELHUM-ANN-global_ref.nc
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/main/meridional_mean_2d/MERRA2/MERRA2-RELHUM-ANN-global_ref.nc
    * var_key: RELHUM
    * All close and within relative tolerance (1e-05)
Comparing:
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/657-meridional-mean-2d/meridional_mean_2d/MERRA2/MERRA2-RELHUM-ANN-global_test.nc
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/main/meridional_mean_2d/MERRA2/MERRA2-RELHUM-ANN-global_test.nc
    * var_key: RELHUM
    * All close and within relative tolerance (1e-05)
Comparing:
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/657-meridional-mean-2d/meridional_mean_2d/MERRA2/MERRA2-RELHUM-JJA-global_ref.nc
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/main/meridional_mean_2d/MERRA2/MERRA2-RELHUM-JJA-global_ref.nc
    * var_key: RELHUM
    * All close and within relative tolerance (1e-05)
Comparing:
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/657-meridional-mean-2d/meridional_mean_2d/MERRA2/MERRA2-RELHUM-JJA-global_test.nc
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/main/meridional_mean_2d/MERRA2/MERRA2-RELHUM-JJA-global_test.nc
    * var_key: RELHUM
    * All close and within relative tolerance (1e-05)
Comparing:
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/657-meridional-mean-2d/meridional_mean_2d/MERRA2/MERRA2-T-ANN-global_ref.nc
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/main/meridional_mean_2d/MERRA2/MERRA2-T-ANN-global_ref.nc
    * var_key: T
    * All close and within relative tolerance (1e-05)
Comparing:
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/657-meridional-mean-2d/meridional_mean_2d/MERRA2/MERRA2-T-ANN-global_test.nc
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/main/meridional_mean_2d/MERRA2/MERRA2-T-ANN-global_test.nc
    * var_key: T
    * All close and within relative tolerance (1e-05)
Comparing:
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/657-meridional-mean-2d/meridional_mean_2d/MERRA2/MERRA2-T-JJA-global_ref.nc
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/main/meridional_mean_2d/MERRA2/MERRA2-T-JJA-global_ref.nc
    * var_key: T
    * All close and within relative tolerance (1e-05)
Comparing:
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/657-meridional-mean-2d/meridional_mean_2d/MERRA2/MERRA2-T-JJA-global_test.nc
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/main/meridional_mean_2d/MERRA2/MERRA2-T-JJA-global_test.nc
    * var_key: T
    * All close and within relative tolerance (1e-05)
Comparing:
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/657-meridional-mean-2d/meridional_mean_2d/MERRA2/MERRA2-U-ANN-global_ref.nc
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/main/meridional_mean_2d/MERRA2/MERRA2-U-ANN-global_ref.nc
    * var_key: U

Not equal to tolerance rtol=1e-05, atol=0

Mismatched elements: 1 / 9792 (0.0102%)
Max absolute difference: 1.06174015e-07
Max relative difference: 1.43005542e-05
 x: array([[10.623722, 10.599673, 10.576724, ..., 10.698371, 10.673217,
        10.64816 ],
       [12.345496, 12.317517, 12.289864, ..., 12.432171, 12.403032,...
 y: array([[10.623722, 10.599673, 10.576724, ..., 10.698371, 10.673217,
        10.64816 ],
       [12.345496, 12.317517, 12.289864, ..., 12.432171, 12.403032,...
Comparing:
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/657-meridional-mean-2d/meridional_mean_2d/MERRA2/MERRA2-U-ANN-global_test.nc
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/main/meridional_mean_2d/MERRA2/MERRA2-U-ANN-global_test.nc
    * var_key: U

Not equal to tolerance rtol=1e-05, atol=0

Mismatched elements: 1 / 6120 (0.0163%)
Max absolute difference: 1.45889873e-07
Max relative difference: 1.08735566e-05
 x: array([[ 9.712424,  9.692257,  9.675111, ...,  9.79681 ,  9.779955,
         9.765068],
       [12.482069, 12.457917, 12.434379, ..., 12.581674, 12.560391,...
 y: array([[ 9.712424,  9.692257,  9.675112, ...,  9.79681 ,  9.779955,
         9.765068],
       [12.482069, 12.457917, 12.434379, ..., 12.581674, 12.560391,...
Comparing:
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/657-meridional-mean-2d/meridional_mean_2d/MERRA2/MERRA2-U-JJA-global_ref.nc
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/main/meridional_mean_2d/MERRA2/MERRA2-U-JJA-global_ref.nc
    * var_key: U
    * All close and within relative tolerance (1e-05)
Comparing:
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/657-meridional-mean-2d/meridional_mean_2d/MERRA2/MERRA2-U-JJA-global_test.nc
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/main/meridional_mean_2d/MERRA2/MERRA2-U-JJA-global_test.nc
    * var_key: U
    * All close and within relative tolerance (1e-05)
jasonb5 commented 3 months ago

I'll take a look this week, the code assumes the dimension ordering is [..., X, Y], I'll adjust this to be more flexible.

tomvothecoder commented 3 months ago

@jasonb5 Sounds good, thanks.

tomvothecoder commented 3 months ago

@jasonb5 Unless this can be resolved by next Tues, we can push it back to the next release. I don't think it is a high priority because Jiwoo hasn't ran into it and I'm using xESMF in e3sm_diags instead.

jasonb5 commented 3 months ago

@tomvothecoder Go ahead and push this out to the next release.

tomvothecoder commented 2 months ago

@jasonb5 Sounds good