Utility to create digit-dropped wrfout-like files from GFS&CFS

[logi]

@beneduzi Am I correct that this requires your pending WOD commit for the downloaded upstream data to have enough variables that we can run this utility against them?

[logi]

@kstanislawska once our upstream files have the needed variables, can you verify this issue by creating a netcdf file from some nomads data and then generate a point forecast from that using the existing utility?

[kstanislawska]

I created a nc file from one of my upstream forecast files using gfs_upstream.yml and it went through fine. I went on to creating a point forecast file and got this error:

  File "/home/karolina/WrfUtils.git/PointForecast/wrfout_reader.py", line 29, in __init__
    self.start_date = parse_wrf_date(nc_data.START_DATE)
  File "netCDF4/_netCDF4.pyx", line 2388, in netCDF4._netCDF4.Dataset.__getattr__ (netCDF4/_netCDF4.c:20474)
  File "netCDF4/_netCDF4.pyx", line 2333, in netCDF4._netCDF4.Dataset.getncattr (netCDF4/_netCDF4.c:19433)
  File "netCDF4/_netCDF4.pyx", line 1112, in netCDF4._netCDF4._get_att (netCDF4/_netCDF4.c:4451)
AttributeError: NetCDF: Attribute not found

@beneduzi could you make sure that we have START_DATE variable in the file and also check that Times variable is there and is consistent with the WRF output (so it is capitalized at least ;))? Which variables do we need to add to upstream so we get this fully functional? I see a lot of values being empty in the created file. times has only the last value, XLAT and XLONG seem to be empty.

[beneduzi]

@kstanislawska We need to add acpcp, apcp, gust, gfluxsfc, csnow, sdwe and short-wave and long-wave downward radiation flux.

[logi]

@beneduzi what are the variable names that need to be added to the *_VARS constants https://github.com/Belgingur/WOD/blob/master/troupe/src/upstream/nomads_urls.py#L750

[beneduzi]

ACPCP, APCP, DLWRF, DSWRF, GUST For, rain, radiation and wind gust. CSNOW for snow coverage

[kstanislawska]

I created a .nc file using one of the recent forecast files downloaded by troupe, with config file gfs_upstream.yml. There are some problems with ranges and precision in some variables:

>>> print(data.variables['XLONG'][:])
[[65525 65526 65526 ... 39 39 40]
 [65525 65526 65526 ... 39 39 40]
 [65525 65526 65526 ... 39 39 40]
 ...
 [65525 65526 65526 ... 39 39 40]
 [65525 65526 65526 ... 39 39 40]
 [65525 65526 65526 ... 39 39 40]]
>>> print(data.variables['XLONG'][20,:])
[65525 65526 65526 65527 65527 65528 65528 65529 65529 65530 65530 65531
 65531 65532 65532 65533 65533 65534 65534 -- -- 0 0 0 1 1 2 2 3 3 4 4 5 5
 6 6 7 7 8 8 9 9 10 10 11 11 12 12 13 13 14 14 15 15 16 16 17 17 18 18 19
 19 20 20 21 21 22 22 23 23 24 24 25 25 26 26 27 27 28 28 29 29 30 30 31
 31 32 32 33 33 34 34 35 35 36 36 37 37 38 38 39 39 40]
>>> print(data.variables['XLAT'][:])
[[35 35 35 ... 35 35 35]
 [35 35 35 ... 35 35 35]
 [36 36 36 ... 36 36 36]
 ...
 [70 70 70 ... 70 70 70]
 [70 70 70 ... 70 70 70]
 [71 71 71 ... 71 71 71]]

Times variable has most of the content empty.

>>> print(data.variables['Times'][:])
['' '' '' '' '' '' '' '' '' '' '' '' '' '' '' '' '2018-02-25_00:00:00']

Also some contents of all other variables are empty.

>>> print(data.variables['T2'][10,:])
[[-- -- -- ... -- -- --]
 [-- -- -- ... -- -- --]
 [-- -- -- ... -- -- --]
 ...
 [-- -- -- ... -- -- --]
 [-- -- -- ... -- -- --]
 [-- -- -- ... -- -- --]]

[beneduzi]

This is strange, in my local tests the variables were ok. I will take another look at this.

[kstanislawska]

@beneduzi I still see the following problems with the code:

• XLONG spans 0..360 instead of -180..180
• Times variable is wrong (it prints [b'2'] instead of the full date)
• SINALPHA field is 1 and should be 0

I tested this against the global upstream and not tested the sub-area upstreams. When we get the Times and XLONG variable right, I can start testing against the point forecast code.

This is how Times variable looks in ts-wrfouts:

>>> print (data.variables['Times'])
<class 'netCDF4._netCDF4.Variable'>
|S1 Times(Time, DateStrLen)
unlimited dimensions: 
current shape = (13, 19)
filling on, default _FillValue of  used

>>> print (data.variables['Times'][:])
[[b'2' b'0' b'1' b'8' b'-' b'0' b'3' b'-' b'2' b'2' b'_' b'0' b'9' b':'
  b'0' b'0' b':' b'0' b'0']
 [b'2' b'0' b'1' b'8' b'-' b'0' b'3' b'-' b'2' b'2' b'_' b'1' b'0' b':'
  b'0' b'0' b':' b'0' b'0']
 [b'2' b'0' b'1' b'8' b'-' b'0' b'3' b'-' b'2' b'2' b'_' b'1' b'1' b':'
  b'0' b'0' b':' b'0' b'0']
...

[kstanislawska]

What is GHT? Should that not be HGT as in ts-wrfouts? What is TSLB? I have not seen this variable anywhere in the post-processing before.