search

akrherz / oats

"Oats & Horses" modelling work
Apache License 2.0
0 stars 0 forks source link

Variable extraction for ER_LU10_IR10 #9

Open akrherz opened 7 years ago

akrherz commented 7 years ago

As per Dr Arritt, please create a variable extraction from the d02 files of:

Step 2 is then to produce 'daily' 12-12z averages.

akrherz commented 7 years ago

Use wrfncxnj

akrherz commented 7 years ago

running this command to see what fun can be had (needed to patch a python glob issue)

/opt/miniconda2/bin/python wrfncxnj.py -o T2.nc -v T2 -f "../../ER_LU10_IR10/d02_original/wrfout_d02_*"
akrherz commented 7 years ago

Command hit a bad file:

    START_DATE: 1995-08-23_12:00:00
Traceback (most recent call last):
  File "wrfncxnj.py", line 115, in <module>
    wnt.checkStep(wrfnc)
  File "/mnt/nrel/akrherz/wrfncxnj-0.1_r2183/wrfncxnj_base.py", line 776, in checkStep
    t1 = strptime(charr2str(incTimes[1]))
  File "/mnt/nrel/akrherz/wrfncxnj-0.1_r2183/wrfncxnj_base.py", line 319, in strptime
    date = ncdf.num2date(0, units="Days since %s" % str)
  File "netCDF4/_netCDF4.pyx", line 5491, in netCDF4._netCDF4.num2date (netCDF4/_netCDF4.c:66113)
  File "netCDF4/_netCDF4.pyx", line 5284, in netCDF4._netCDF4._dateparse (netCDF4/_netCDF4.c:63025)
  File "netcdftime/_netcdftime.pyx", line 927, in netcdftime._netcdftime._parse_date (netcdftime/_netcdftime.c:17475)
  File "netcdftime/_netcdftime.pyx", line 939, in netcdftime._netcdftime._parse_date (netcdftime/_netcdftime.c:17184)
ValueError: Unable to parse date string ''

moved the file into the questionable folder and started the process again.

akrherz commented 7 years ago

The problematic file was actually wrfout_d02_1995-08-23_02:00:00, which had a Times[1] = ''

akrherz commented 7 years ago

Found another bad file :( wrfout_d02_1996-03-14_06\:00\:00

akrherz commented 7 years ago

Sigh, another bad file near 1996-07-05_20:30:00. I'd best write another script to find these pre-emptively

akrherz commented 7 years ago

OK, so I sort of abandoned using these wrf files as is due to the very slow access of them. Instead, I am now rewritting them using nccopy to get rid of the unlimited time dimension. Having a fixed time dimension means that reading variables over all times at once is snappy fast. I just now did a test running wrfncxnj over them and it was very fast and appeared to work!

akrherz commented 7 years ago

So the nccopy process finished yesterday, so I am back at it now. A script was written to check over the files to find any further Times issues and here are what it found:

So am running the wrf script again and will cross fingers!

akrherz commented 7 years ago

Horray, T2 extraction finished, but is 47 GB in size :/ Will discuss this further with Dr Arritt before proceeding further

akrherz commented 7 years ago

decision was to remove the original files (copied from hard drives) and run with the nccopy files. generating the other variables now

akrherz commented 7 years ago

Cloud fraction extraction failed with:

/opt/miniconda2/bin/python wrfncxnj.py -o CLDFRA.nc -v CLDFRA -f "../../ER_LU10_IR10/d02_original_nccopy/wrfout_d02_*"
Error, no pressure levels found in the input files

SMOIS failed with

Traceback (most recent call last):
  File "wrfncxnj.py", line 142, in <module>
    oncvar.set_ncvars(vars[varname])
  File "/mnt/nrel/akrherz/wrfncxnj-0.1_r2183/wrfncxnj_base.py", line 934, in set_ncvars
    self.ncvars = self.onc.createVariable(varobj.standard_abbr, np.float32, self.dims)
  File "netCDF4/_netCDF4.pyx", line 2293, in netCDF4._netCDF4.Dataset.createVariable (netCDF4/_netCDF4.c:18399)
  File "netCDF4/_netCDF4.pyx", line 3215, in netCDF4._netCDF4.Variable.__init__ (netCDF4/_netCDF4.c:30249)
  File "/opt/miniconda2/lib/python2.7/site-packages/netCDF4/utils.py", line 41, in _find_dim
    raise ValueError("cannot find dimension %s in this group or parent groups" % dimname)
ValueError: cannot find dimension slev in this group or parent groups
akrherz commented 7 years ago

After code review, need to add -z to command for CLDFRA and fix table

akrherz commented 7 years ago

And add -s to the command for soil moisture

akrherz commented 7 years ago

So to get the "daily" 12z to 12z averages, we have the following dance

  1. Remove unlimited time dimension nccopy -u -4 T2.nc T2_fixed.nc
  2. Overwrite our original file mv T2_fixed.nc T2.nc
  3. Copy file to allow for faked time cp T2.nc T2_faketime.nc
  4. Add 12 hours to the time variable python add12.py T2_faketime.nc
  5. Use cdo to do daymean cdo daymean T2_faketime.nc T2_daymean.nc
  6. remove old file rm T2_faketime.nc