Closed olafurrognvaldsson closed 4 years ago
I am unable to reproduce this issue. If I uncomment the sigma_limit: 5
line from DropDigits.yml
and run against the 1.4G file from #20 (downloader@binary.belgingur.is:wod_test_data/jobs/2016/10/20161024-africa-50/model/wrfout_d01_2016-10-24_06:00:00
) then I get this written to console:
(DropDigits) logi@saxi:~/belgingur.is/WrfUtils.git/DropDigits$ ./DropDigits.py /home/logi/wod_test_data/jobs/2016/10/20161024-africa-50/model/wrfout_d01_2016-10-24_06:00:00
2020-01-15 17:05:16 | INFO | Configured logging from /home/logi/belgingur.is/WrfUtils.git/DropDigits/logging.yml
2020-01-15 17:05:16 | INFO | Load config from DropDigits.yml
2020-01-15 17:05:16 | INFO |
2020-01-15 17:05:16 | INFO | Opening input dataset /home/logi/wod_test_data/jobs/2016/10/20161024-africa-50/model/wrfout_d01_2016-10-24_06:00:00
2020-01-15 17:05:16 | INFO | Include all variables except []
2020-01-15 17:05:16 | INFO | Excluded variables: <none>
2020-01-15 17:05:16 | INFO | Included variables: Times, XLAT, XLONG, U, V, W, PH, PHB, T, TKE, P, ALT, PB, Q2, T2, PSFC, U10, V10, ITIMESTEP, RELHUM, RH2, MSLP, QVAPOR, QCLOUD, QRAIN, TSLB, SMOIS, GRDFLX, ACSNOM, SNOW, SNOWH, TKE_PBL, SINALPHA, COSALPHA, HGT, RAINC, RAINNC, SWDOWN, GLW, XLAT_U, XLONG_U, XLAT_V, XLONG_V, PBLH, REGIME, SNOWC, SR, BN2, SAVE_TOPO_FROM_REAL, SEED1, SEED2, LAKEMASK, SST
2020-01-15 17:05:16 | INFO | Dimensional limits
2020-01-15 17:05:16 | INFO | Spinup is 0h = 0 steps
2020-01-15 17:05:16 | INFO | Margin is 10 cells
2020-01-15 17:05:17 | INFO | 3D variables limited to 5 levels which reach at least 309m above sea level and 309m above surface level
2020-01-15 17:05:17 | INFO | Creating output file ./wrfout_d01_2016-10-24_06:00:00_reduced.nc
2020-01-15 17:05:17 | INFO | Setting/updating global file attributes for output file
2020-01-15 17:05:17 | INFO | Copy 114 attributes
2020-01-15 17:05:17 | INFO | Add attributes:
2020-01-15 17:05:17 | INFO | history = Converted with DropDigits.py at 2020-05-15 17:01:17 by logi on saxi
2020-01-15 17:05:17 | INFO | source = /home/logi/wod_test_data/jobs/2016/10/20161024-africa-50/model/wrfout_d01_2016-10-24_06:00:00
2020-01-15 17:05:17 | INFO | institution = Belgingur
2020-01-15 17:05:17 | INFO | description = Reduced version of: /home/logi/wod_test_data/jobs/2016/10/20161024-africa-50/model/wrfout_d01_2016-10-24_06:00:00
2020-01-15 17:05:17 | INFO | Add output dimensions:
2020-01-15 17:05:17 | INFO | Time (28)
2020-01-15 17:05:17 | INFO | DateStrLen (19)
2020-01-15 17:05:17 | INFO | west_east (164)
2020-01-15 17:05:17 | INFO | south_north (179)
2020-01-15 17:05:17 | INFO | west_east_stag (165)
2020-01-15 17:05:17 | INFO | bottom_top (5)
2020-01-15 17:05:17 | INFO | south_north_stag (180)
2020-01-15 17:05:17 | INFO | bottom_top_stag (6)
2020-01-15 17:05:17 | INFO | soil_layers_stag (4)
2020-01-15 17:05:17 | INFO | Excluded dimensions: <none>
2020-01-15 17:05:17 | INFO | Create output variables with overrides:
2020-01-15 17:05:17 | INFO | Times |S1 None … None [Time,DateStrLen]
2020-01-15 17:05:17 | INFO | XLAT f4 -3.4e+38 … 3.4e+38 [Time,south_north,west_east]
2020-01-15 17:05:17 | INFO | XLONG f4 -3.4e+38 … 3.4e+38 [Time,south_north,west_east]
2020-01-15 17:05:17 | INFO | U i2*0.01 -327.68 … 327.67 [Time,bottom_top,south_north,west_east_stag]
2020-01-15 17:05:17 | INFO | V i2*0.01 -327.68 … 327.67 [Time,bottom_top,south_north_stag,west_east]
2020-01-15 17:05:17 | INFO | W i2*0.01 -327.68 … 327.67 [Time,bottom_top_stag,south_north,west_east]
2020-01-15 17:05:17 | INFO | PH f4 -3.4e+38 … 3.4e+38 [Time,bottom_top_stag,south_north,west_east]
2020-01-15 17:05:17 | INFO | PHB f4 -3.4e+38 … 3.4e+38 [Time,bottom_top_stag,south_north,west_east]
2020-01-15 17:05:17 | INFO | T i2*0.01 -327.68 … 327.67 [Time,bottom_top,south_north,west_east]
2020-01-15 17:05:17 | INFO | TKE i2*0.001-32.768 0.0 … 65.535 [Time,bottom_top,south_north,west_east]
2020-01-15 17:05:17 | INFO | P i2*0.2 -6553.6 … 6553.400000000001 [Time,bottom_top,south_north,west_east]
2020-01-15 17:05:17 | INFO | ALT i2*0.001-32.768 0.0 … 65.535 [Time,bottom_top,south_north,west_east]
2020-01-15 17:05:17 | INFO | PB i2*2-65536 0 … 131070 [Time,bottom_top,south_north,west_east]
2020-01-15 17:05:17 | INFO | Q2 f4 -3.4e+38 … 3.4e+38 [Time,south_north,west_east]
2020-01-15 17:05:17 | INFO | T2 i2*0.01-327.68 0.0 … 655.35 [Time,south_north,west_east]
2020-01-15 17:05:17 | INFO | PSFC i2*2-65536 0 … 131070 [Time,south_north,west_east]
2020-01-15 17:05:17 | INFO | U10 i2*0.01 -327.68 … 327.67 [Time,south_north,west_east]
2020-01-15 17:05:17 | INFO | V10 i2*0.01 -327.68 … 327.67 [Time,south_north,west_east]
2020-01-15 17:05:17 | INFO | ITIMESTEP i4 -2147483648 … 2147483647 [Time]
2020-01-15 17:05:17 | INFO | RELHUM i2*0.01-327.68 0.0 … 655.35 [Time,bottom_top,south_north,west_east]
2020-01-15 17:05:17 | INFO | RH2 i2*0.01-327.68 0.0 … 655.35 [Time,south_north,west_east]
2020-01-15 17:05:17 | INFO | MSLP i2*2-65536 0 … 131070 [Time,south_north,west_east]
2020-01-15 17:05:17 | INFO | QVAPOR i2*8e-06-0.0262144 -0.2359296 … 0.2883504 [Time,bottom_top,south_north,west_east]
2020-01-15 17:05:17 | INFO | QCLOUD i2*8e-06-0.0262144 -0.2359296 … 0.2883504 [Time,bottom_top,south_north,west_east]
2020-01-15 17:05:17 | INFO | QRAIN i2*8e-06-0.0262144 -0.2359296 … 0.2883504 [Time,bottom_top,south_north,west_east]
2020-01-15 17:05:17 | INFO | TSLB i2*0.01-327.68 0.0 … 655.35 [Time,soil_layers_stag,south_north,west_east]
2020-01-15 17:05:17 | INFO | SMOIS i2*0.0001-3.2768 0.0 … 6.5535 [Time,soil_layers_stag,south_north,west_east]
2020-01-15 17:05:17 | INFO | GRDFLX i2*0.1 -3276.8 … 3276.7000000000003 [Time,south_north,west_east]
2020-01-15 17:05:17 | INFO | ACSNOM i2*0.1-3276.8 0.0 … 6553.5 [Time,south_north,west_east]
2020-01-15 17:05:17 | INFO | SNOW f4 -3.4e+38 … 3.4e+38 [Time,south_north,west_east]
2020-01-15 17:05:17 | INFO | SNOWH i2*0.001-32.768 0.0 … 65.535 [Time,south_north,west_east]
2020-01-15 17:05:17 | INFO | TKE_PBL i2*0.001-32.768 0.0 … 65.535 [Time,bottom_top_stag,south_north,west_east]
2020-01-15 17:05:17 | INFO | SINALPHA i2*0.0001 -3.2768 … 3.2767 [Time,south_north,west_east]
2020-01-15 17:05:17 | INFO | COSALPHA i2*0.0001-3.2768 0.0 … 6.5535 [Time,south_north,west_east]
2020-01-15 17:05:17 | INFO | HGT i2*0.2-4000 -2553.6000000000004 … 10553.400000000001 [Time,south_north,west_east]
2020-01-15 17:05:17 | INFO | RAINC f4 -3.4e+38 … 3.4e+38 [Time,south_north,west_east]
2020-01-15 17:05:17 | INFO | RAINNC f4 -3.4e+38 … 3.4e+38 [Time,south_north,west_east]
2020-01-15 17:05:17 | INFO | SWDOWN i2*0.1-3276.8 0.0 … 6553.5 [Time,south_north,west_east]
2020-01-15 17:05:17 | INFO | GLW i2*0.01-327.68 0.0 … 655.35 [Time,south_north,west_east]
2020-01-15 17:05:17 | INFO | XLAT_U f4 -3.4e+38 … 3.4e+38 [Time,south_north,west_east_stag]
2020-01-15 17:05:17 | INFO | XLONG_U f4 -3.4e+38 … 3.4e+38 [Time,south_north,west_east_stag]
2020-01-15 17:05:17 | INFO | XLAT_V f4 -3.4e+38 … 3.4e+38 [Time,south_north_stag,west_east]
2020-01-15 17:05:17 | INFO | XLONG_V f4 -3.4e+38 … 3.4e+38 [Time,south_north_stag,west_east]
2020-01-15 17:05:17 | INFO | PBLH f4 -3.4e+38 … 3.4e+38 [Time,south_north,west_east]
2020-01-15 17:05:17 | INFO | REGIME i1 -128 … 127 [Time,south_north,west_east]
2020-01-15 17:05:17 | INFO | SNOWC i1 -128 … 127 [Time,south_north,west_east]
2020-01-15 17:05:17 | INFO | SR i1*0.01 -1.28 … 1.27 [Time,south_north,west_east]
2020-01-15 17:05:17 | INFO | BN2 i2*1e-06 -0.032768 … 0.032767 [Time,bottom_top,south_north,west_east]
2020-01-15 17:05:17 | INFO | SAVE_TOPO_FROM_REAL i2*0.0001-3.2768 0.0 … 6.5535 [Time]
2020-01-15 17:05:17 | INFO | SEED1 i2*0.0001-3.2768 0.0 … 6.5535 [Time]
2020-01-15 17:05:17 | INFO | SEED2 i2*0.0001-3.2768 0.0 … 6.5535 [Time]
2020-01-15 17:05:17 | INFO | LAKEMASK f4 -3.4e+38 … 3.4e+38 [Time,south_north,west_east]
2020-01-15 17:05:17 | INFO | SST i2*0.01-327.68 0.0 … 655.35 [Time,south_north,west_east]
2020-01-15 17:05:17 | INFO | Copying data in chunks of 128 time steps
2020-01-15 17:05:17 | INFO | Chunk[0:28]: 2016-10-24 06:00:00 - 2016-10-25 09:00:00
2020-01-15 17:05:17 | INFO | Variable Min Max
2020-01-15 17:05:17 | INFO | Times N/A N/A
2020-01-15 17:05:17 | INFO | XLAT -36.32 37.91
2020-01-15 17:05:18 | INFO | XLONG -21.65 51.65
2020-01-15 17:05:19 | INFO | U -25.28 22.67
2020-01-15 17:05:20 | INFO | V -25.06 36.79
2020-01-15 17:05:21 | INFO | W -0.68 0.36
2020-01-15 17:05:23 | INFO | PH -116.28 491.29
2020-01-15 17:05:24 | INFO | PHB -1,414.61 32,060.61
2020-01-15 17:05:25 | INFO | T -18.88 22.09
2020-01-15 17:05:25 | INFO | TKE 0.00 0.00
2020-01-15 17:05:26 | INFO | P -689.10 2,952.23
2020-01-15 17:05:27 | INFO | ALT 0.80 1.19
2020-01-15 17:05:28 | INFO | PB 67,634.44 101,373.15
2020-01-15 17:05:28 | INFO | Q2 0.00 0.03
2020-01-15 17:05:28 | INFO | T2 272.67 314.89
2020-01-15 17:05:28 | INFO | PSFC 72,961.61 104,602.08
2020-01-15 17:05:28 | INFO | U10 -15.94 16.04
2020-01-15 17:05:29 | INFO | V10 -17.36 19.75
2020-01-15 17:05:29 | INFO | ITIMESTEP 0.00 397.00
2020-01-15 17:05:30 | INFO | RELHUM 2.89 100.00
2020-01-15 17:05:30 | INFO | RH2 6.57 100.00
2020-01-15 17:05:30 | INFO | MSLP 99,311.56 102,925.48
2020-01-15 17:05:31 | INFO | QVAPOR 0.00 0.02
2020-01-15 17:05:32 | INFO | QCLOUD 0.00 0.00
2020-01-15 17:05:32 | INFO | QRAIN 0.00 0.00
2020-01-15 17:05:33 | INFO | TSLB 273.16 318.79
2020-01-15 17:05:33 | INFO | SMOIS 0.02 1.00
2020-01-15 17:05:33 | INFO | GRDFLX -344.14 141.04
2020-01-15 17:05:33 | INFO | ACSNOM 0.00 0.80
2020-01-15 17:05:33 | INFO | SNOW 0.00 8.05
2020-01-15 17:05:33 | INFO | SNOWH 0.00 0.05
2020-01-15 17:05:34 | INFO | TKE_PBL 0.10 5.60
2020-01-15 17:05:34 | INFO | SINALPHA -0.00 -0.00
2020-01-15 17:05:34 | INFO | COSALPHA 1.00 1.00
2020-01-15 17:05:34 | INFO | HGT -144.20 2,788.55
2020-01-15 17:05:34 | INFO | RAINC 0.00 67.00
2020-01-15 17:05:34 | INFO | RAINNC -0.00 78.88
2020-01-15 17:05:34 | INFO | SWDOWN 0.00 1,162.79
2020-01-15 17:05:34 | INFO | GLW 0.00 478.03
2020-01-15 17:05:34 | INFO | XLAT_U -36.32 37.91
2020-01-15 17:05:34 | INFO | XLONG_U -21.88 51.88
2020-01-15 17:05:34 | INFO | XLAT_V -36.50 38.09
2020-01-15 17:05:34 | INFO | XLONG_V -21.65 51.65
2020-01-15 17:05:35 | INFO | PBLH 0.00 5,478.49
2020-01-15 17:05:35 | INFO | REGIME 0.00 0.00
2020-01-15 17:05:35 | INFO | SNOWC 0.00 0.42
2020-01-15 17:05:35 | INFO | SR -0.35 1.00
2020-01-15 17:05:36 | INFO | BN2 -0.00 0.01
2020-01-15 17:05:36 | INFO | SAVE_TOPO_FROM_REAL 0.00 0.00
2020-01-15 17:05:36 | INFO | SEED1 0.00 0.00
2020-01-15 17:05:36 | INFO | SEED2 0.00 0.00
2020-01-15 17:05:36 | INFO | LAKEMASK 0.00 0.00
2020-01-15 17:05:36 | INFO | SST 267.71 337.07
2020-01-15 17:05:36 | INFO | Size: 1,385,386 MB -> 76,262 MB, reduced to 5.5% in 20.5 s
And the beginning of ncdump
looks like this:
netcdf wrfout_d01_2016-10-24_06\:00\:00_reduced {
dimensions:
Time = 28 ;
DateStrLen = 19 ;
west_east = 164 ;
south_north = 179 ;
west_east_stag = 165 ;
bottom_top = 5 ;
south_north_stag = 180 ;
bottom_top_stag = 6 ;
soil_layers_stag = 4 ;
@olafurrognvaldsson can you show me output from a failing run and perhaps even a config and data file that fails to run?
What happens if you do e.g. ncdump -v T2 wrfout_d01_2016-10-24_06\:00\:00_reduced
?
...
:source = "/home/logi/wod_test_data/jobs/2016/10/20161024-africa-50/model/wrfout_d01_2016-10-24_06:00:00" ;
:institution = "Belgingur" ;
:description = "Reduced version of: /home/logi/wod_test_data/jobs/2016/10/20161024-africa-50/model/wrfout_d01_2016-10-24_06:00:00" ;
data:
T2 =
-3974, -3977, -3963, -3953, -3954, -3966, -3979, -3991, -3999, -3995,
-3988, -3981, -3979, -3997, -4027, -4045, -4063, -4072, -4076, -4070,
-4062, -4061, -4072, -4082, -4096, -4129, -4142, -4114, -4083, -4069,
-4073, -4104, -4122, -4118, -4103, -4067, -4019, -3975, -3943, -3919,
-3899, -3882, -3864, -3847, -3832, -3828, -3841, -3872, -3899, -3920,
-3949, -3973, -4000, -4031, -4065, -4096, -4118, -4120, -4122, -4119,
-4114, -4106, -4097, -4086, -4075, -4058, -4038, -4020, -4006, -3995,
-3972, -3958, -3962, -3967, -3966, -3950, -3932, -3906, -3879, -3854,
-3824, -3790, -3761, -3747, -3754, -3765, -3778, -3783, -3776, -3768,
-3753, -3732, -3711, -3695, -3684, -3671, -3641, -3578, -3504, -3475,
-3475, -3487, -3504, -3531, -3565, -3589, -3585, -3584, -3601, -3627,
-3652, -3668, -3681, -3689, -3688, -3683, -3680, -3675, -3667, -3661,
-3657, -3654, -3649, -3652, -3654, -3659, -3660, -3664, -3676, -3696,
-3707, -3706, -3709, -3711, -3706, -3705, -3713, -3719, -3712, -3709,
-3712, -3716, -3713, -3702, -3686, -3677, -3672, -3674, -3666, -3666,
-3661, -3661, -3675, -3669, -3652, -3650, -3669, -3677, -3690, -3705,
-3713, -3729, -3732, -3740,
-3969, -3975, -3969, -3966, -3963, -3967, -3977, -3986, -3987, -3974,
-3959, -3952, -3954, -3974, -4005, -4027, -4053, -4067, -4068, -4063,
-4057, -4054, -4062, -4070, -4077, -4094, -4096, -4074, -4046, -4036,
-4044, -4067, -4060, -4020, -3989, -3953, -3914, -3899, -3884, -3867,
...
Interesting, try changing the sign in the yml-file so it reads add_offset: 327.68
for variable T2
(these are some mighty low values ... :)
That sounds like a completely separate issue?
The shit gets faunkier. When I run this here
#!/bin/bash
#
source _python_env_
#
for i in `ls /riv/scrooge/RAV2/2018-19/wrfout_d01_2018-08-25*`
do
echo "/home/or/Belgingur-Utils-20191227/WrfUtils/DropDigits/DropDigits.py " $i
/home/or/Belgingur-Utils-20191227/WrfUtils/DropDigits/DropDigits.py $i
done
I get this error message
2020-01-15 20:08:16 | ERROR | Uncaught exception in main()
Traceback (most recent call last):
File "/home/or/Belgingur-Utils-20191227/WrfUtils/DropDigits/DropDigits.py", line 486, in <module>
main()
File "/home/or/Belgingur-Utils-20191227/WrfUtils/DropDigits/DropDigits.py", line 309, in main
errors = process_file(in_file, out_file, config=config, overrides=overrides)
File "/home/or/Belgingur-Utils-20191227/WrfUtils/DropDigits/DropDigits.py", line 438, in process_file
in_chunk: np.ndarray = in_var[c_start:c_end, 0:var_max_k, margin:max_j - margin, margin:max_i - margin]
File "netCDF4/_netCDF4.pyx", line 4095, in netCDF4._netCDF4.Variable.__getitem__
File "/home/or/miniconda3/envs/DropDigits/lib/python3.7/site-packages/netCDF4/utils.py", line 330, in _StartCountStride
raise ValueError("slicing expression exceeds the number of dimensions of the variable")
ValueError: slicing expression exceeds the number of dimensions of the variable
Traceback (most recent call last):
File "/home/or/Belgingur-Utils-20191227/WrfUtils/DropDigits/DropDigits.py", line 486, in <module>
main()
File "/home/or/Belgingur-Utils-20191227/WrfUtils/DropDigits/DropDigits.py", line 309, in main
errors = process_file(in_file, out_file, config=config, overrides=overrides)
File "/home/or/Belgingur-Utils-20191227/WrfUtils/DropDigits/DropDigits.py", line 438, in process_file
in_chunk: np.ndarray = in_var[c_start:c_end, 0:var_max_k, margin:max_j - margin, margin:max_i - margin]
File "netCDF4/_netCDF4.pyx", line 4095, in netCDF4._netCDF4.Variable.__getitem__
File "/home/or/miniconda3/envs/DropDigits/lib/python3.7/site-packages/netCDF4/utils.py", line 330, in _StartCountStride
raise ValueError("slicing expression exceeds the number of dimensions of the variable")
ValueError: slicing expression exceeds the number of dimensions of the variable
But now I'm not going to stare at this longer tonight
I need more of the output. That doesn't give any indication which variable it is failing on or which variables succeeded.
@olafurrognvaldsson Can you please trigger the error by running DropDigits on exactly one netCDF file with the new --verbose
flag and attach both the full console output to this issue along with the config file used and the path of the netCDF file. That should isolate the problem.
DropDigits.yml.log DropDigits.sh.log dropdigits.log
The error appears when the code tries to process the variable float ZNU(Time, bottom_top)
The way the code is run is shown in DropDigits.sh.log
(I can't upload files ending in .sh or .yml, hence the extra ".log").
OK, and the way that ZNU
and ZNW
are different from other variables I've seen is that they have a vertical dimension but no planar dimensions:
2020-01-16 12:32:52 | INFO | ZNU i2*0.0001-3.2768 0.0 … 6.5535 [Time,bottom_top]
2020-01-16 12:32:52 | INFO | ZNW i2*0.0001-3.2768 0.0 … 6.5535 [Time,bottom_top_stag]
and
<class 'netCDF4._netCDF4.Variable'>
int16 ZNU(Time, bottom_top)
description: eta values on half (mass) levels
least_significant_digit: 4
scale_factor: 0.0001
add_offset: -3.2768
FieldType: 104
MemoryOrder: Z
units:
stagger:
unlimited dimensions:
current shape = (56, 5)
filling on, default _FillValue of -32767 used
<class 'netCDF4._netCDF4.Variable'>
int16 ZNW(Time, bottom_top_stag)
description: eta values on full (w) levels
least_significant_digit: 4
scale_factor: 0.0001
add_offset: -3.2768
FieldType: 104
MemoryOrder: Z
units:
stagger: Z
unlimited dimensions:
current shape = (56, 6)
filling on, default _FillValue of -32767 used
while the code assumes that if there is a vertical dimension that needs to be sliced then there will be planar dimensions as well. This is something I can fix:
2020-01-16 12:32:54 | DEBUG | var_name: ZNU
2020-01-16 12:32:54 | DEBUG | in_var.shape: (56, 64)
2020-01-16 12:32:54 | DEBUG | in_slice: [0:56,0:5,10:46,10:54]
should be:
2020-01-16 12:32:54 | DEBUG | var_name: ZNU
2020-01-16 12:32:54 | DEBUG | in_var.shape: (56, 64)
2020-01-16 12:32:54 | DEBUG | in_slice: [0:56,0:5]
@olafurrognvaldsson Try it again please?
@logi works like a charm !!
There is still an assumption in the code that if there is one planar dimension (staggered or not) exactly when the other planar dimension is present (independently staggered or not).
DropDigits writes no values in the new netCDF file when used with
sigma_limit = xx