Issue with post.bash output file dates

[ailishgraham]

For the past week or so I have noticed that all of my post processed files have strange time values (e.g. days since 2000 values of e60 e-28 etc). I am using a copy of pp.ncl from the WRFotron folder and the updated post.bash. I think the date values are linked to an error I started getting in post.bash (shown below). I thought the issue may be due to the change of nco version to 4.6.0 (to fix the restart file bug), based on googling the error message and help threads suggesting it is a library bug (and ncl/nco being dependent on each other?), but when I try re-running post.bash with nco version 4.8.1 the error is still appearing.

The errors I get for every chunk are:

PP'ing 20190915000000 deleting tmp_wrfout_d01_2019-09-15_00:00:00.nc fatal:Could not create (tmp_wrfout_d01_2019-09-15_00:00:00.nc)

fatal:ut_inv_calendar: Invalid specification string

fatal:["Execute.c":8637]:Execute: Error occurred at or near line 55 in file $NCARG_ROOT/lib/ncarg/nclscripts/wrf/WRF_contributed.ncl

fatal:["Execute.c":8637]:Execute: Error occurred at or near line 283 in file $NCARG_ROOT/lib/ncarg/nclscripts/wrf/WRF_contributed.ncl

fatal:["Execute.c":8637]:Execute: Error occurred at or near line 22 in file /nobackup/ee15amg/Australia_fires/WRFotron2.0_2019/pp.ncl

fatal:Error writing file variable attribute, either the file or the variable (timestamp) are undefined

fatal:["Execute.c":8637]:Execute: Error occurred at or near line 23 in file /nobackup/ee15amg/Australia_fires/WRFotron2.0_2019/pp.ncl

fatal:Error writing file variable attribute, either thefile or the variable (timestamp) are undefined

fatal:["Execute.c":8637]:Execute: Error occurred at or near line 24 in file /nobackup/ee15amg/Australia_fires/WRFotron2.0_2019/pp.ncl

fatal:Either file (fileout) isn't defined or variable (td_2m) is not a variable in the file

fatal:["Execute.c":8637]:Execute: Error occurred at or near line 33 in file /nobackup/ee15amg/Australia_fires/WRFotron2.0_2019/pp.ncl

fatal:Either file (fileout) isn't defined or variable (td) is not a variable in the file

fatal:["Execute.c":8637]:Execute: Error occurred at or near line 35 in file /nobackup/ee15amg/Australia_fires/WRFotron2.0_2019/pp.ncl

I get the same error for every wrfout file. Has anyone ever seen this before? Is it worth trying the python processing instead to diagnose whether this is an nco problem or an issue with my output (I hope this isn't the case :( )?

[lukeconibear]

I've recently been testing the CEMAC method and have also had this issue with pp.ncl not running correctly, and the additional variables not being created (e.g. AOD550_sfc). The scripts breaks from the start trying to load the NCAR functions, which then causes other issues:

fatal:ut_inv_calendar: Invalid specification string
^Mfatal:["Execute.c":8637]:Execute: Error occurred at or near line 55 in file $NCARG_ROOT/lib/ncarg/nclscripts/wrf/WRF_contributed.ncl

I've not had these errors before with my manual compilation which used NCL6.6.2.

Helen thought this may be due to NCL6.5.0, which is the version the CEMAC method has.

Though seeing as NCL has given us many problems over the years and that it is not maintained anymore, I've replaced pp.ncl with postprocessing.py. I've used wrf-python to try and keep it consistent with pp.ncl. I've tested it, and it works correctly for both WRFChem3.7.1 and WRFChem4.2, which removes the need to fix the NCL bug. I'd recommend you use this from now on and you will also need the updated post.bash.

This is currently running in serial, and Helen and I were looking into enabling the OpenMP support within wrf-python to parallelise it again.

At the moment there is a separate NCO bug for using NETCDF4 which affects WRFChem4.2, though to my memory you're using WRFChem3.7.1 so this is not relevant for you.

I think your raw wrfout files will be fine, you'll just need to re-run the postprocessing for the ones affected by this NCL issue above which don't have the additional variables.

[ailishgraham]

Yes I had this same issue too, lots of variables were left out. Very glad to hear it's an issue in the post processing as I had suspected.

You're right, I'm using WRFChem3.7.1 so the second issue won't affect me. I had seen the python version of pp.ncl and wondered if this would fix the issue, thanks for creating this. I will download both the python code and new post.bash and do a test now. Just to check the variable WRFChem_path in the python code is the path to the model code? i.e in 'python postprocessing.py inFile outFile WRFChem_path'.

Thanks for the help!

[lukeconibear]

Yes, the final variable WRFChem_path is for the main model e.g. /nobackup/${USER}/WRFChem3.7.1. This is to determine whether it needs to do the NETCDF4 conversion for versions newer than WRFChem4. Within post.bash these are already set for you using $WRFdir which was defined in config.bash. So the example: python postprocessing.py inFile outFile WRFChem_path Becomes: python ${pyPpScript} ${inFile} tmp_${outFile} $WRFdir

I forgot to mention that you will also need to replace the postprocessing script within config.bash: nclPpScript=${chainDir}/pp.ncl with: pyPpScript=${chainDir}/postprocessing.py

[ailishgraham]

This all makes sense now. I updated the scripts and the test worked so have submitted the rest of the runs. Thanks for the help!

[lukeconibear]

Great, you're welcome, thanks for letting me know.