barronh
commented
3 years ago - OMPROFOZ
- skipping data even when present: this can occur if the data filters are removing all valid points.
- OMPROFOZv003.py was using a strict filtering requirements. Increase RMS and AverageResidual to 3 helps
- pressure was inverted and not supported until now! Latest version correctly handles pressure and inverts output to match CMAQ.
- assertion errors: I suspect these are when the shape of the data cannot be concatenated for some reason. For example, the Xtrack dimensions are not equal. I would advocate removing those files.
- SceneLER should not be required. The sat2cmaq.py has been retooled to only use explicitly requested variables.
I believe that all these issues have been addressed. Please confirm and we'll close this issue.
I've used the updated sat2cmaq.py processing script and OMNO2v003.py and OMPROFOZv003.py configuration scripts to process 2004 - 2017 OMPROFOZ and 2004 - 2019 OMNO2d into daily files for the hemispheric CMAQ domain.
This worked really nicely for almost the entire dataset. Below are some exceptions where I ran into trouble, there probably are multiple reasons for that. Any help would be appreciated.
1. 2016 OMPROFOZ
run script on local machine: /work/MOD3EVAL/css/CMAQv53_TS/work/satellite/OMPROFOZ/run_convertsatomprofoz_daily_2016.csh
log file on local machine: /work/MOD3EVAL/css/CMAQv53_TS/work/satellite/OMPROFOZ/logs/convertsatomprofoz_daily_2016_1164387.txt
downloaded L2 input data on local machine: /work/MOD3EVAL/css/CMAQv53_TS/work/satellite/OMPROFOZ/avdc.gsfc.nasa.gov
"assertion error": 2/10 and 3/9
"no data; skipping" error despite non-empty downloaded L2 directories: 3/8, 3/13, 3/16, 3/18, 5/1, 6/13, 6/17, 6/18, 6/27, 6/29, 7/1, 7/4, 7/7, 7/13, 7/14, 8/1, 8/3, 8/11
2. 2017 OMPROFOZ
run script on local machine: /work/MOD3EVAL/css/CMAQv53_TS/work/satellite/OMPROFOZ/run_convertsatomprofoz_daily_2017.csh
log file on local machine: /work/MOD3EVAL/css/CMAQv53_TS/work/satellite/OMPROFOZ/logs/convertsatomprofoz_daily_2017_1164444.txt
downloaded L2 input data on local machine: /work/MOD3EVAL/css/CMAQv53_TS/work/satellite/OMPROFOZ/avdc.gsfc.nasa.gov
"assertion error": 6/28
"no data; skipping" error despite non-empty downloaded L2 directories: 6/15, 6/18, 10/18, 10/27, 10/28, 10/31, 11/1, 11/2, 11/5, 11/9, 11/14, 11/16, 11/18, 11/25, 12/7, 12/8, 12/23, 12/25
"zero-size array to reduction operation minimum which has no identity": 6/13, 10/24
3. 2019 OMNO2d
run script on local machine: /work/MOD3EVAL/css/CMAQv53_TS/work/satellite/OMNO2d/run_conversatomno2d_2019.csh
log file on local machine: /work/MOD3EVAL/css/CMAQv53_TS/work/satellite/OMNO2d/logs/convertsatomprofoz_daily_2019_1164950.txt
downloaded L2 input data on local machine: /work/MOD3EVAL/css/CMAQv53_TS/work/satellite/OMNO2d/aura.gesdisc.eosdis.nasa.gov
"KeyError: 'SceneLER'" (see below for full traceback): 1/27, 1/28, 2/28, 3/1, 4/1, 4/2, 5/3, 5/4, 6/4, 6/5, 7/6, 7/7, 8/7, 8/8, 9/1, 9/9, 10/10, 10/11, 11/11, 11/12
Traceback (most recent call last): File "scripts/sat2cmaq.py", line 172, in
process()
File "scripts/sat2cmaq.py", line 59, in process
omf, omgf = openpaths(inpaths)
File "scripts/sat2cmaq.py", line 54, in openpaths
omf = omfs[0].stack(omfs[1:], datatgtdim)
File "/work/ROMO/anaconda_envs/basic38/lib/python3.8/site-packages/PseudoNetCDF/core/files.py", line 1835, in stack
[f.variables[varkey][:] for f_ in fs], axis=axisi)
File "/work/ROMO/anaconda_envs/basic38/lib/python3.8/site-packages/PseudoNetCDF/core/files.py", line 1835, in
[f .variables[varkey][:] for f_ in fs], axis=axisi)
KeyError: 'SceneLER'
Besides these issues, the only instances were daily files weren't produced by sat2cmaq.py were those where the underlying L2 datafiles are missing from the ftp directories. Below is a list of those cases for reference:
OMPROFOZ:
2004 data started October 1, 2004. No data November 19 - December 1, 204
2006 no data available February 28 - March 2, 2006
2008 no data available September 28 - 29, 2008
2013 no data available November 12, 2013
2016 no data available May 30 - June 12, 2016 (14 missing days)
2017 no data available March 13 - 16, 2017
OMNO2d:
2004 data started October 1, 2004. No data November 19 - December 1, 204
2006 no data available February 28 - March 2, 2006
2008 no data available September 28 - 29, 2008
2016 no data available May 30 - June 9 and June 11 - June 12, 2016
2017 no data available March 13 - 16, 2017