IAMconsortium / concordia

Apache License 2.0
0 stars 3 forks source link

Issues with emissions files #24

Closed dariak-bsc closed 5 months ago

dariak-bsc commented 10 months ago

Errors found with the file checker:

lat does not correspond to the expected one (lat length is 280 but the reference lat length is 360)

these files `- Checking file 13/172: BC-em-openburning_input4MIPs_emissions_RESCUE_IIASA-PIK-REMIND-MAgPIE__3.2.0-4.6.7-RESCUE-Tier1-Direct-2023-09-14-EocBudg1150-OAE_off-2023-10-07_gn_202001-210012.nc
- Checking file 14/172: BC-em-openburning_input4MIPs_emissions_RESCUE_IIASA-PIK-REMIND-MagPIE__3.2.0-4.6.7-RESCUE-Tier1-Direct-2023-09-14-EocBudg500-OAE_off-2023-10-07_gn_202001-210012.nc
- Checking file 15/172: BC-em-openburning_input4MIPs_emissions_RESCUE_IIASA-PIK-REMIND-MagPIE__3.2.0-4.6.7-RESCUE-Tier1-Direct-2023-09-14-PkBudg1150-OAE_off-2023-10-07_gn_202001-210012.nc
- Checking file 16/172: BC-em-openburning_input4MIPs_emissions_RESCUE_IIASA-PIK-REMIND-MagPIE__3.2.0-4.6.7-RESCUE-Tier1-Direct-2023-09-14-PkBudg1150-OAE_on-2023-10-07_gn_202001-210012.nc
- Checking file 17/172: BC-em-openburning_input4MIPs_emissions_RESCUE_IIASA-PIK-REMIND-MagPIE__3.2.0-4.6.7-RESCUE-Tier1-Direct-2023-09-14-PkBudg500-OAE_off-2023-10-07_gn_202001-210012.nc
- Checking file 18/172: BC-em-openburning_input4MIPs_emissions_RESCUE_IIASA-PIK-REMIND-MagPIE__3.2.0-4.6.7-RESCUE-Tier1-Direct-2023-09-14-PkBudg500-OAE_on-2023-10-07_gn_202001-210012.nc
- Checking file 31/172: CH4-em-openburning_input4MIPs_emissions_RESCUE_IIASA-PIK-REMIND-MagPIE__3.2.0-4.6.7-RESCUE-Tier1-Direct-2023-09-14-EocBudg1150-OAE_off-2023-10-07_gn_202001-210012.nc
- Checking file 32/172: CH4-em-openburning_input4MIPs_emissions_RESCUE_IIASA-PIK-REMIND-MagPIE__3.2.0-4.6.7-RESCUE-Tier1-Direct-2023-09-14-EocBudg500-OAE_off-2023-10-07_gn_202001-210012.nc
- Checking file 33/172: CH4-em-openburning_input4MIPs_emissions_RESCUE_IIASA-PIK-REMIND-MagPIE__3.2.0-4.6.7-RESCUE-Tier1-Direct-2023-09-14-PkBudg1150-OAE_off-2023-10-07_gn_202001-210012.nc
- Checking file 34/172: CH4-em-openburning_input4MIPs_emissions_RESCUE_IIASA-PIK-REMIND-MagPIE__3.2.0-4.6.7-RESCUE-Tier1-Direct-2023-09-14-PkBudg1150-OAE_on-2023-10-07_gn_202001-210012.nc
- Checking file 35/172: CH4-em-openburning_input4MIPs_emissions_RESCUE_IIASA-PIK-REMIND-MagPIE__3.2.0-4.6.7-RESCUE-Tier1-Direct-2023-09-14-PkBudg500-OAE_off-2023-10-07_gn_202001-210012.nc
- Checking file 36/172: CH4-em-openburning_input4MIPs_emissions_RESCUE_IIASA-PIK-REMIND-MagPIE__3.2.0-4.6.7-RESCUE-Tier1-Direct-2023-09-14-PkBudg500-OAE_on-2023-10-07_gn_202001-210012.nc
- Checking file 49/172: CO-em-openburning_input4MIPs_emissions_RESCUE_IIASA-PIK-REMIND-MAgPIE__3.2.0-4.6.7-RESCUE-Tier1-Direct-2023-09-14-EocBudg1150-OAE_off-2023-10-07_gn_202001-210012.nc
- Checking file 50/172: CO-em-openburning_input4MIPs_emissions_RESCUE_IIASA-PIK-REMIND-MagPIE__3.2.0-4.6.7-RESCUE-Tier1-Direct-2023-09-14-EocBudg500-OAE_off-2023-10-07_gn_202001-210012.nc
- Checking file 51/172: CO-em-openburning_input4MIPs_emissions_RESCUE_IIASA-PIK-REMIND-MAgPIE__3.2.0-4.6.7-RESCUE-Tier1-Direct-2023-09-14-PkBudg1150-OAE_off-2023-10-07_gn_202001-210012.nc
- Checking file 52/172: CO-em-openburning_input4MIPs_emissions_RESCUE_IIASA-PIK-REMIND-MAgPIE__3.2.0-4.6.7-RESCUE-Tier1-Direct-2023-09-14-PkBudg1150-OAE_on-2023-10-07_gn_202001-210012.nc
- Checking file 53/172: CO-em-openburning_input4MIPs_emissions_RESCUE_IIASA-PIK-REMIND-MAgPIE__3.2.0-4.6.7-RESCUE-Tier1-Direct-2023-09-14-PkBudg500-OAE_off-2023-10-07_gn_202001-210012.nc
- Checking file 54/172: CO-em-openburning_input4MIPs_emissions_RESCUE_IIASA-PIK-REMIND-MAgPIE__3.2.0-4.6.7-RESCUE-Tier1-Direct-2023-09-14-PkBudg500-OAE_on-2023-10-07_gn_202001-210012.nc
- Checking file 67/172: CO2-em-removal_input4MIPs_emissions_RESCUE_IIASA-PIK-REMIND-MAgPIE__3.2.0-4.6.7-RESCUE-Tier1-Direct-2023-09-14-EocBudg1150-OAE_off-2023-10-07_gn_202001-210012.nc
- Checking file 68/172: CO2-em-removal_input4MIPs_emissions_RESCUE_IIASA-PIK-REMIND-MAgPIE__3.2.0-4.6.7-RESCUE-Tier1-Direct-2023-09-14-EocBudg500-OAE_off-2023-10-07_gn_202001-210012.nc
- Checking file 69/172: CO2-em-removal_input4MIPs_emissions_RESCUE_IIASA-PIK-REMIND-MAgPIE__3.2.0-4.6.7-RESCUE-Tier1-Direct-2023-09-14-PkBudg1150-OAE_off-2023-10-07_gn_202001-210012.nc
- Checking file 70/172: CO2-em-removal_input4MIPs_emissions_RESCUE_IIASA-PIK-REMIND-MAgPIE__3.2.0-4.6.7-RESCUE-Tier1-Direct-2023-09-14-PkBudg1150-OAE_on-2023-10-07_gn_202001-210012.nc
- Checking file 71/172: CO2-em-removal_input4MIPs_emissions_RESCUE_IIASA-PIK-REMIND-MagPIE__3.2.0-4.6.7-RESCUE-Tier1-Direct-2023-09-14-PkBudg500-OAE_off-2023-10-07_gn_202001-210012.nc
- Checking file 72/172: CO2-em-removal_input4MIPs_emissions_RESCUE_IIASA-PIK-REMIND-MAgPIE__3.2.0-4.6.7-RESCUE-Tier1-Direct-2023-09-14-PkBudg500-OAE_on-2023-10-07_gn_202001-210012.nc
- Checking file 85/172: NH3-em-openburning_input4MIPs_emissions_RESCUE_IIASA-PIK-REMIND-MAgPIE__3.2.0-4.6.7-RESCUE-Tier1-Direct-2023-09-14-EocBudg1150-OAE_off-2023-10-07_gn_202001-210012.nc
- Checking file 86/172: NH3-em-openburning_input4MIPs_emissions_RESCUE_IIASA-PIK-REMIND-MAgPIE__3.2.0-4.6.7-RESCUE-Tier1-Direct-2023-09-14-EocBudg500-OAE_off-2023-10-07_gn_202001-210012.nc
- Checking file 87/172: NH3-em-openburning_input4MIPs_emissions_RESCUE_IIASA-PIK-REMIND-MAgPIE__3.2.0-4.6.7-RESCUE-Tier1-Direct-2023-09-14-PkBudg1150-OAE_off-2023-10-07_gn_202001-210012.nc
- Checking file 88/172: NH3-em-openburning_input4MIPs_emissions_RESCUE_IIASA-PIK-REMIND-MAgPIE__3.2.0-4.6.7-RESCUE-Tier1-Direct-2023-09-14-PkBudg1150-OAE_on-2023-10-07_gn_202001-210012.nc
- Checking file 89/172: NH3-em-openburning_input4MIPs_emissions_RESCUE_IIASA-PIK-REMIND-MagPIE__3.2.0-4.6.7-RESCUE-Tier1-Direct-2023-09-14-PkBudg500-OAE_off-2023-10-07_gn_202001-210012.nc
- Checking file 90/172: NH3-em-openburning_input4MIPs_emissions_RESCUE_IIASA-PIK-REMIND-MagPIE__3.2.0-4.6.7-RESCUE-Tier1-Direct-2023-09-14-PkBudg500-OAE_on-2023-10-07_gn_202001-210012.nc
- Checking file 103/172: NOx-em-openburning_input4MIPs_emissions_RESCUE_IIASA-PIK-REMIND-MAgPIE__3.2.0-4.6.7-RESCUE-Tier1-Direct-2023-09-14-EocBudg1150-OAE_off-2023-10-07_gn_202001-210012.nc
- Checking file 104/172: NOx-em-openburning_input4MIPs_emissions_RESCUE_IIASA-PIK-REMIND-MAgPIE__3.2.0-4.6.7-RESCUE-Tier1-Direct-2023-09-14-EocBudg500-OAE_off-2023-10-07_gn_202001-210012.nc
- Checking file 105/172: NOx-em-openburning_input4MIPs_emissions_RESCUE_IIASA-PIK-REMIND-MAgPIE__3.2.0-4.6.7-RESCUE-Tier1-Direct-2023-09-14-PkBudg1150-OAE_off-2023-10-07_gn_202001-210012.nc
- Checking file 106/172: NOx-em-openburning_input4MIPs_emissions_RESCUE_IIASA-PIK-REMIND-MAgPIE__3.2.0-4.6.7-RESCUE-Tier1-Direct-2023-09-14-PkBudg1150-OAE_on-2023-10-07_gn_202001-210012.nc
- Checking file 107/172: Nox-em-openburning_input4MIPs_emissions_RESCUE_IIASA-PIK-REMIND-MagPIE__3.2.0-4.6.7-RESCUE-Tier1-Direct-2023-09-14-PkBudg500-OAE_off-2023-10-07_gn_202001-210012.nc
- Checking file 108/172: Nox-em-openburning_input4MIPs_emissions_RESCUE_IIASA-PIK-REMIND-MagPIE__3.2.0-4.6.7-RESCUE-Tier1-Direct-2023-09-14-PkBudg500-OAE_on-2023-10-07_gn_202001-210012.nc
- Checking file 121/172: OC-em-openburning_input4MIPs_emissions_RESCUE_IIASA-PIK-REMIND-MAgPIE__3.2.0-4.6.7-RESCUE-Tier1-Direct-2023-09-14-EocBudg1150-OAE_off-2023-10-07_gn_202001-210012.nc
- Checking file 122/172: OC-em-openburning_input4MIPs_emissions_RESCUE_IIASA-PIK-REMIND-MAgPIE__3.2.0-4.6.7-RESCUE-Tier1-Direct-2023-09-14-EocBudg500-OAE_off-2023-10-07_gn_202001-210012.nc
- Checking file 123/172: OC-em-openburning_input4MIPs_emissions_RESCUE_IIASA-PIK-REMIND-MAgPIE__3.2.0-4.6.7-RESCUE-Tier1-Direct-2023-09-14-PkBudg1150-OAE_off-2023-10-07_gn_202001-210012.nc
- Checking file 124/172: OC-em-openburning_input4MIPs_emissions_RESCUE_IIASA-PIK-REMIND-MAgPIE__3.2.0-4.6.7-RESCUE-Tier1-Direct-2023-09-14-PkBudg1150-OAE_on-2023-10-07_gn_202001-210012.nc
- Checking file 125/172: OC-em-openburning_input4MIPs_emissions_RESCUE_IIASA-PIK-REMIND-MAgPIE__3.2.0-4.6.7-RESCUE-Tier1-Direct-2023-09-14-PkBudg500-OAE_off-2023-10-07_gn_202001-210012.nc
- Checking file 126/172: OC-em-openburning_input4MIPs_emissions_RESCUE_IIASA-PIK-REMIND-MagPIE__3.2.0-4.6.7-RESCUE-Tier1-Direct-2023-09-14-PkBudg500-OAE_on-2023-10-07_gn_202001-210012.nc
- Checking file 139/172: Sulfur-em-openburning_input4MIPs_emissions_RESCUE_IIASA-PIK-REMIND-MAgPIE__3.2.0-4.6.7-RESCUE-Tier1-Direct-2023-09-14-EocBudg1150-OAE_off-2023-10-07_gn_202001-210012.nc
- Checking file 140/172: Sulfur-em-openburning_input4MIPs_emissions_RESCUE_IIASA-PIK-REMIND-MAgPIE__3.2.0-4.6.7-RESCUE-Tier1-Direct-2023-09-14-EocBudg500-OAE_off-2023-10-07_gn_202001-210012.nc
- Checking file 141/172: Sulfur-em-openburning_input4MIPs_emissions_RESCUE_IIASA-PIK-REMIND-MAgPIE__3.2.0-4.6.7-RESCUE-Tier1-Direct-2023-09-14-PkBudg1150-OAE_off-2023-10-07_gn_202001-210012.nc
- Checking file 142/172: Sulfur-em-openburning_input4MIPs_emissions_RESCUE_IIASA-PIK-REMIND-MAgPIE__3.2.0-4.6.7-RESCUE-Tier1-Direct-2023-09-14-PkBudg1150-OAE_on-2023-10-07_gn_202001-210012.nc
- Checking file 143/172: Sulfur-em-openburning_input4MIPs_emissions_RESCUE_IIASA-PIK-REMIND-MagPIE__3.2.0-4.6.7-RESCUE-Tier1-Direct-2023-09-14-PkBudg500-OAE_off-2023-10-07_gn_202001-210012.nc
- Checking file 144/172: Sulfur-em-openburning_input4MIPs_emissions_RESCUE_IIASA-PIK-REMIND-MagPIE__3.2.0-4.6.7-RESCUE-Tier1-Direct-2023-09-14-PkBudg500-OAE_on-2023-10-07_gn_202001-210012.nc
- Checking file 157/172: VOC-em-openburning_input4MIPs_emissions_RESCUE_IIASA-PIK-REMIND-MAgPIE__3.2.0-4.6.7-RESCUE-Tier1-Direct-2023-09-14-EocBudg1150-OAE_off-2023-10-07_gn_202001-210012.nc
- Checking file 158/172: VOC-em-openburning_input4MIPs_emissions_RESCUE_IIASA-PIK-REMIND-MAgPIE__3.2.0-4.6.7-RESCUE-Tier1-Direct-2023-09-14-EocBudg500-OAE_off-2023-10-07_gn_202001-210012.nc
- Checking file 159/172: VOC-em-openburning_input4MIPs_emissions_RESCUE_IIASA-PIK-REMIND-MAgPIE__3.2.0-4.6.7-RESCUE-Tier1-Direct-2023-09-14-PkBudg1150-OAE_off-2023-10-07_gn_202001-210012.nc
- Checking file 160/172: VOC-em-openburning_input4MIPs_emissions_RESCUE_IIASA-PIK-REMIND-MAgPIE__3.2.0-4.6.7-RESCUE-Tier1-Direct-2023-09-14-PkBudg1150-OAE_on-2023-10-07_gn_202001-210012.nc
- Checking file 161/172: VOC-em-openburning_input4MIPs_emissions_RESCUE_IIASA-PIK-REMIND-MagPIE__3.2.0-4.6.7-RESCUE-Tier1-Direct-2023-09-14-PkBudg500-OAE_off-2023-10-07_gn_202001-210012.nc
- Checking file 162/172: VOC-em-openburning_input4MIPs_emissions_RESCUE_IIASA-PIK-REMIND-MagPIE__3.2.0-4.6.7-RESCUE-Tier1-Direct-2023-09-14-PkBudg500-OAE_on-2023-10-07_gn_202001-210012.nc`

NaNs - some files have all missing values at certain timesteps

these files - Checking file 111/172: OC-em-AIR-anthro_input4MIPs_emissions_RESCUE_IIASA-PIK-REMIND-MAgPIE__3.2.0-4.6.7-RESCUE-Tier1-Direct-2023-09-14-PkBudg1150-OAE_off-2023-10-07_gn_202001-210012.nc
ERROR: Unexpected NaN(s) encountered in OC_em_AIR_anthro at timesteps=[72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107]
- Checking file 112/172: OC-em-AIR-anthro_input4MIPs_emissions_RESCUE_IIASA-PIK-REMIND-MAgPIE__3.2.0-4.6.7-RESCUE-Tier1-Direct-2023-09-14-PkBudg1150-OAE_on-2023-10-07_gn_202001-210012.nc
ERROR: Unexpected NaN(s) encountered in OC_em_AIR_anthro at timesteps=[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107]
- Checking file 113/172: OC-em-AIR-anthro_input4MIPs_emissions_RESCUE_IIASA-PIK-REMIND-MagPIE__3.2.0-4.6.7-RESCUE-Tier1-Direct-2023-09-14-PkBudg500-OAE_off-2023-10-07_gn_202001-210012.nc
ERROR: Unexpected NaN(s) encountered in OC_em_AIR_anthro at timesteps=[72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107]
- Checking file 114/172: OC-em-AIR-anthro_input4MIPs_emissions_RESCUE_IIASA-PIK-REMIND-MAgPIE__3.2.0-4.6.7-RESCUE-Tier1-Direct-2023-09-14-PkBudg500-OAE_on-2023-10-07_gn_202001-210012.nc
ERROR: Unexpected NaN(s) encountered in OC_em_AIR_anthro at timesteps=[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107]


General remark:
I had some problems while trying to read/manipulate the files with CDO and ncview, there is something with the time variable. This is probably still ok (my code with Python/xarray is running) but just to let you know...

problem description This is happening with different files, it's not only the problem with the one file in the example below. ncview shows the plots but gives this warning: ```bash ncview: netcdf_dim_value: unknown data type (10) for dimension time ``` CDO is not able to recognize timesteps, e.g. `cdo showtime` says either `NetCDF: HDF error`, or ```bash cdo showtime BC-em-AIR-anthro_input4MIPs_emissions_RESCUE_IIASA-PIK-REMIND-MAgPIE__3.2.0-4.6.7-RESCUE-Tier1-Direct-2023-09-14-EocBudg1150-OAE_off-2023-10-07_gn_202001-210012.nc 00:00:00 cdo showtime: Processed 1 variable over 1 timestep ( 0.01s ) ``` (it recognizes only 1 timestep instead of 108; though `ncdump -v time` is showing everything correctly)
or ```bash cdo showtime BC-em-anthro_input4MIPs_emissions_RESCUE_IIASA-PIK-REMIND-MAgPIE__3.2.0-4.6.7-RESCUE-Tier1-Direct-2023-09-14-EocBudg1150-OAE_off-2023-10-07_gn_202001-210012.nc Warning (cdfScanVarAttr): Time must be the first dimension! Unsupported array structure, skipped variable BC_em_anthro! Warning (cdfInqContents): No data arrays found! cdo showtime: Open failed on >BC-em-anthro_input4MIPs_emissions_RESCUE_IIASA-PIK-REMIND-MAgPIE__3.2.0-4.6.7-RESCUE-Tier1-Direct-2023-09-14-EocBudg1150-OAE_off-2023-10-07_gn_202001-210012.nc< Unsupported file structure ``` After rearranging the dimensions as `ncpdq -a time,sector,lat,lon,bound `, cdo showtime stops giving warnings/errors but shows e.g. ```bash 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 cdo showtime: Processed 1 variable over 108 timesteps [0.16s 65MB]. ```


Filenames:
Also, as discussed previously, there are extra "_" in the filenames which require postprocessing, currently I am doing this in my code:

@etiennesky