search

IAMconsortium / concordia

Apache License 2.0
0 stars 3 forks source link

Dimension ordering #48

Closed coroa closed 1 week ago

coroa commented 3 months ago

@etiennesky @gidden

We just changed/fixed the dimension ordering in the last round of updates to time, sector/level, lat, lon. We talked about this at: https://github.com/IAMconsortium/concordia/issues/32#issuecomment-1952600975 , where I remarked that:

Input4MIPS had: sector, time, lat, lon for CO2[_em_anthro] and time, sector, lat, lon everywhere else. I did go for the latter everywhere (consistency).

I am now seeing in the NetCDF conventions (different to what @TimotheeBrgs remarked by email), that the most compliant ordering would probably be:

etiennesky commented 3 months ago

Reading the CF conventionss All other dimensions should, whenever possible, be placed to the left of the spatiotemporal dimensions.. Doesn't that imply `time, sector,lat, lon, sector for _em_anthro and openburning? which is what I see for CO-em-anthro_input4MIPs_emissions_CMIP_CEDS-2017-05-18_gn_200001-201412.nc and CO-em-anthro_input4MIPs_emissions_ScenarioMIP_IAMC-AIM-ssp370-1-1_gn_201501-210012.nc

Hi, I think the CO2 emissions files for the CMIP6 scenarios have wrong dimensions that is CO2_em_AIR_anthro(level, time, lat, lon) and CO2_em_anthro(sector, time, lat, lon). For use in EC-Earth3 we corrected those files using ncpdq -a time,level,lat,lon and ncpdq -a time,sector,lat,lon, respectively.

I think this is a bug and we should use the "proper" ordering for CO2, identical to the other species.

coroa commented 3 months ago

All other dimensions should, whenever possible, be placed to the left of the spatiotemporal dimensions.

Uh, wow. Sorry, i agree i read this wrong.

I think it does say to use: sector, time, lat, lon and time, level, lat, lon. I guess it is meant to make it easy to read a full blob of 3 or 4 dimensional spatiotemporal data at a time. Which is a bit weird since that does not seem to be what cdo expects, or is it?

We currently do have time, sector, lat, lon and time, level, lat, lon and I would be open to keeping it (since that was also the majority of the input4MIPS files), but i am also fine with changing -em-anthro and openburning to sector, time, lat, lon.

coroa commented 3 months ago

@etiennesky I'll make that your call! :)

TimotheeBrgs commented 3 months ago

Hi, catching up with my emails after some time off. Thank you Matt and Jonas for the follow-up by email and on Github. The aim of my email was the FillValue issue #47 that I spotted, that is now closed. I was not especially expecting a fix on the dimension reordering because I already had to do this reordering for processing CMIP6 input4MIPs files, and you aim to follow input4MIPs structure.

That said, if CMIP6 files have a wrong ordering, feel free to fix it in the RESCUE files. It will add a specificity in the workflow of RESCUE file processing compared to CMIP6, but it will hopefully be in line with CMIP7 files.