Issues with LUC files - round 2

[dariak-bsc]

Hi all,
Errors & warnings found with the file checker - preliminary results:

1. (Warning) Some attributes are missing, e.g. dataset_category, grid_label, source_id, table_id, target_mip
   
2. (Error) There are missing values in the data
3. (Error) In the reference files, latitude and longitude are called 'lat' and 'lon' correspondingly, but in the new files they are called 'latitude' and 'longitude'

@etiennesky

[coroa]

Hi @dariak-bsc ,

thank you for submitting those issues with your checker!

@pascal-sauer @tscheypidi FYI and please comment!

[ShraddhaGupta28]

Hi all, The reference land use files have annual time resolution for the LU states, transitions and management, however the RESCUE land use scenarios came with time gaps of 5 years/10 years. The model (JSBACH) of course requires the files to have an annual time resolution. Can we expect the next version of the land use files to come with annual time resolution, or do we have to solve the issue ourselves by interpolating ? Thanks !

[pascal-sauer]

Hi and sorry for the late reply.

(Warning) Some attributes are missing, e.g. dataset_category, grid_label, source_id, table_id, target_mip

What are the expected values for these attributes?

(Error) There are missing values in the data

I'll double check.

(Error) In the reference files, latitude and longitude are called 'lat' and 'lon' correspondingly, but in the new files they are called 'latitude' and 'longitude'

Should be possible to just rename these, I'll check.

[pascal-sauer]

Hi @ShraddhaGupta28, the magpie model produces output with these time steps (5/10 years). Additional time steps would always simply be interpolated, which massively increases files sizes and leads to (admittedly fixable) memory problems on our side. So for the iterating phase where we still expect bug fixes and changes in response to feedback we would stick with the 5/10 year time steps, and kindly ask you to do the interpolation on your end.

[ShraddhaGupta28]

Hi @pascal-sauer,

There is also a problem regarding the generic grid coordinates when comparing the reference files and the RESCUE land use files. This causes an issue when remapping the new files using CDO for preparing the ESM input cdo gencon (Abort): Unsupported generic coordinates !

On looking closer using cdo griddes <infile.nc>, the following difference is found between the reference files and the new files:

• For the reference files.

  cdi  warning (cdfScanVarAttr): NetCDF: Variable not found - bounds_time
  cdi  warning (cdfScanVarAttr): NetCDF: Variable not found - bounds_lat
  cdi  warning (cdfScanVarAttr): NetCDF: Variable not found - bounds_lon
  #
  # gridID 1
  #
  gridtype  = lonlat
  gridsize  = 1036800
  xsize     = 1440
  ysize     = 720
  xname     = lon
  xlongname = "longitude"
  xunits    = "degrees_east"
  yname     = lat
  ylongname = "latitude"
  yunits    = "degrees_north"
  xfirst    = -179.875
  xinc      = 0.25
  yfirst    = 89.875
  yinc      = -0.25
  #
  # gridID 2
  #
  gridtype  = generic
  gridsize  = 1440
  xsize     = 2
  ysize     = 720
  yname     = lat
  ylongname = "latitude"
  yunits    = "degrees_north"
  yfirst    = 89.875
  yinc      = -0.25
  #
  # gridID 3
  #
  gridtype  = generic
  gridsize  = 2880
  xsize     = 1440
  ysize     = 2
  xname     = lon
  xlongname = "longitude"
  xunits    = "degrees_east"
  xfirst    = -179.875
  xinc      = 0.25
  #
  # gridID 4
  #
  gridtype  = generic
  gridsize  = 2
  xsize     = 2
  cdo    griddes: Processed 121 variables [0.13s 33MB]

• For the new files:

  cdi  warning (cdf_set_dim): Inconsistent dimension definition for latitude! dimid = 0;  type = 1;  newtype = 2
  cdi  warning (cdf_set_dim): Inconsistent dimension definition for longitude! dimid = 0;  type = 2;  newtype = 1
  #
  # gridID 1
  #
  gridtype  = lonlat
  gridsize  = 1036800
  xsize     = 1440
  ysize     = 720
  xname     = longitude
  xlongname = "longitude"
  xunits    = "degrees_east"
  yname     = latitude
  ylongname = "latitude"
  yunits    = "degrees_north"
  xfirst    = -179.875
  xinc      = 0.25
  yfirst    = 89.875
  yinc      = -0.25
  cdo    griddes: Processed 12 variables [0.22s 33MB]

[pascal-sauer]

Sure, I can add the bounds variables

[ShraddhaGupta28]

Hi @pascal-sauer, I think the problem is not about the missing bounds because the bounds are missing in the reference files too. But rather the problem is something about the missing generic grids latitude and longitude. In my previous comment, you see that cdo griddes on the reference file yields 4 gridIDs (see 1st bullet) out of which one is of gridtype latlon while the other three are gridtype generic, while for the new RESCUE Land use files (2nd bullet in the previous comment) one gets only 1 gridID of gridtype latlon. I think these generic gridtypes for lat and lon are important for CDO remapping commands.

[pascal-sauer]

The bounds in the reference files are actually present, but not called like the attributes suggest. They are called lat_bounds instead of bounds_lat etc. I'll now include them as bounds_lat as the attribute indicates.

[pascal-sauer]

Hm I'll check what cdogriddes returns. Maybe renaming to lon/lat as opposed to longitude/latitude already helps. I did set all attributes which are set in the reference files now, so I hope cdo will treat the files exactly like the reference files now. (I did not recreate/upload the dataset yet, but I already put the changes in our pipeline in place)

[dariak-bsc]

Hi @pascal-sauer , I checked the latest data /forcings/land_use/2024-04-25:

• there are some missing values but in fact it seems that the mask for the data is a little different from the mask of the reference files (I am going to make a plot with these points as discussed with @etiennesky );
• multiple-management and multiple-transition files don't have units.

[pascal-sauer]

Thanks for checking Daria, I'll add units and reupload. On the mask with missing values: My understanding is that you check with Etienne if this is fine, and come back to me in case we need to change something, yes?

[dariak-bsc]

Hi @pascal-sauer and @etiennesky , I think maybe we could discuss the mask and the missing values in tomorrow's meeting. (I ran the checker with the mask taken not from the reference file but from the corresponding variable in the file which is being checked, e.g. if I am checking the variable c3ann in the file multiple-states_input4MIPs_landState_RESCUE_PIK-MAgPIE-4-7-C-SSP2EU-NPi-2024.4.25_gn_2015-2100.nc then I am taking the missing values in c3ann at the first timestep as the mask to check the following timesteps. And in this case, I am not getting any missing values at the following timesteps. When I take the mask from the reference file, then I get missing values.)

@pascal-sauer could you please confirm that the variables fharv_c3per, fharv_c4per, flood, combf in multiple-management files are not required?

[pascal-sauer]

Yes, I can confirm we agreed that fharv_c3per, fharv_c4per, flood, combf are not required.

[pascal-sauer]

I uploaded all files again with units. While I was working on the units I replaced unit "1" with "share of X" to make it clearer to which other variable a relative variable relates.

[gidden]

Hi @pascal-sauer - if this issue is done, could you please close? Thanks!

[pascal-sauer]

@dariak-bsc is there anything to be done on the missing values mask? If not we can close this.

[dariak-bsc]

So from what I found (this comment), the forcings landuse mask is different from the reference files. I am not sure what we should do about it, what do you think @pascal-sauer @etiennesky @gidden ? Do we consider the missing values as missing, or I should modify the code and use not the reference mask but the one used in the forcings landuse files?

[dariak-bsc]

This is the difference between the forcings landuse mask and the reference mask:

[etiennesky]

perhaps if @pascal-sauer can run the checker he can see the problem in the generated files

[gidden]

Ok, given that one part of this is still live, let's close this bulk issue and open a new one.