Closed tomvothecoder closed 5 months ago
Hi @chengzhuzhang, I'm just tagging you so you're aware of the issues that were fixed in this PR, differences that we can accept, and issues found on the main
branch. I will merge this PR to continue moving forward.
With the list of issues fixed, the cdat-migration-fy24
branch will be even more robust.
Hi @chengzhuzhang, I'm just tagging you so you're aware of the issues that were fixed in this PR, differences that we can accept, and issues found on the
main
branch. I will merge this PR to continue moving forward.With the list of issues fixed, the
cdat-migration-fy24
branch will be even more robust.
Got it. Thank you for working on this!
Issues fixed
ValueError: Multiple 'Z' axis dims were found in this dataset, ['ilev', 'lev']. Please drop the unused dimension(s) before performing grid operations.
-- affects_align_grids_to_lower_res()
and_apply_land_sea_mask()
ilev
coordinates with pressure variables (e.g., hyam, hybm) that are not used_get_grid()
and_drop_unused_ilev_axis()
KeyError: No attribute "units"
-- affectedcosp_bin_sum()
cosp_bin_sum()
, calling.sum()
results in Xarray dropping the original units attribute. The original units are required for the "convert_units()`" function to correctly convert units to "%". Otherwise, units are None and never set to %.keep_attrs=True
to.sum()
callOSError: No file found for 'HadISST' and 'JJA' in /global/cfs/cdirs/e3sm/diagnostics/observations/Atm/climatology
andOSError: No file found for 'HadISST' and 'ANN' in /global/cfs/cdirs/e3sm/diagnostics/observations/Atm/climatology
try and except
inDataset._get_global_attr_from_climo_dataset()
to cover cases where ref files are missing and the test dataset is being used as the ref datasetx and y nan location mismatching
-- affectsCLDTOT_TAU1.3_9.4_ISCCP
,CLDTOT_TAU1.3_ISCCP
,CLDTOT_TAU9.4_ISCCP
,CLDTOT_TAU9.4_ISCCP
,CLDLOW_TAU1.3_9.4_MISR
,CLDLOW_TAU1.3_MISR
,CLDLOW_TAU9.4_MISR
,CLDTOT_TAU1.3_9.4_MISR
,CLDTOT_TAU1.3_MISR
,CLDTOT_TAU9.4_MISR
,CLDHGH_TAU1.3_9.4_MODIS
,CLDHGH_TAU1.3_MODIS
,CLDHGH_TAU9.4_MODIS
,CLDTOT_TAU1.3_9.4_MODIS
,CLDTOT_TAU1.3_MODIS
,CLDTOT_TAU9.4_MODIS
cosp_bin_sum()
call to.sum()
results innp.nan
being replaced with 0sskipna=True
andmin_count=1
to.sum()
(related post)PminusE
- large relative differencespminuse_1()
,pminuse_2()
,pminuse_3()
) and move arithmetic into those functions wrapped bywith xr.set_options(keep_attrs=True):
QREFHT
- large relative difference and shape mismatch(180, 360), (721, 1440)
derivations.py
fromacme.py
onmain
(related PR).acme.py
ALBEDOC
(reference dataset) -x and y nan location mismatching
albedoc()
formula function dividesrsutcs / rsdt
.rsdt
contains 0s, which results in divide by zeros. Xarray and NumPy replaces these values withnp.inf
instead of throwing a "Divide By Zero" error. Meanwhile CDAT replaces these values with the floats fromrsutcs
, but they are masked so the output willnp.nan
._replace_inf_with_nan()
to update the array values for this caseCRU-TREFHT
-- land mask is not being appliedsubset_and_align_datasets()
was incorrect and did not consider"land_60S90N"
if region == "land" or region == "ocean":
toif "land" in region or "ocean" in region:"
CRU-TREFHT
-- after applying land mask, region subsetting is not being appliedsubset_and_align_datasets()
was incorrect and ignored subsettingelif region == "global":
toif "global" not in region:
Differences we can accept
There seems to be a minor difference in
nan
locations for the regridded land-sea mask produced using xCDAT + xESMF vs. CDAT + ESMF (method="bilinear"
). xCDAT has a little bit morenan
values than CDAT. This causesx and y nan location mismatch
when comparing array values. I don't know the exact reason why there are differences, but they seem insignificant enough to accept.The absolute sum, mean, and nan count values are similar across variables. The diff plots also show no visible differences. All of the scripts for debugging these variables can be found here.
TAUXY
(ocean)ERA5-TREFHT
(land)MERRA2-TREFHT
(land)Issues on
main
Some reference variable names don't align with
main
- affectsAODVIS, FLUT, FLUTC, LCWF, RESTOM, SOLIN, SWCF, FLDS, FLDSC, FSDS, FSDSC, FSNS, FSNSC, LHFLX, SHFLX, T, U, U10, PRECT, SST
main
saves the original variable name, whilecdat-migration-fy24
saves the target (derived) variable name.cdat-migration-fy24
is doing the right thing here.The following variables have large diffs due to a bug on
main
:MISRCOSP-CLDLOW_TAU1.3_9.4_MISR
,MISRCOSP-CLDLOW_TAU1.3_MISR
,MISRCOSP-CLDLOW_TAU9.4_MISR
. I believe this branch is doing the correct thing. We can revisit the comparison for these variables once the issue below is fixed onmain
.Checklist
If applicable: