As of met-6.0, the -job go_index and -job ss_index do not support the "-by" option:
WARNING:
WARNING: The -by option is ignored for the "go_index" job type.
WARNING:
This task is to add that support, particularly to handle "-by FCST_INIT_BEG". Listed below is an email chain which describes the test data to be used on mandan:
(1) I tried running with the "-by" option and see the following:
WARNING:
WARNING: The -by option is ignored for the "go_index" job type.
WARNING:
Perhaps we should make this option work!
(2) One other consideration. Those output directories contain 4 types of point_stat output:
ADPSFC, ADPUPA, WINDS, and ADPSFC_MPR
About 97.5% of the lines existing in the ADPSFC_MPR output type... which isn't used in the GO Index computations!
There is 32GB of MET output for each version...
du -sch find /d4/projects/VERSION_TNE/v3.5.1 -name "*.stat"
But only 1.7GB of that is non-MPR data...
du -sch find /d4/projects/VERSION_TNE/v3.5.1 -name "*.stat" | egrep -v MPR
When I skipped over the ADPSFC_MPR files, it took about 4 minutes to process the 9,087 point_stat non-MPR files... and compute Not having to read/discard several million MPR lines makes it run much, much faster.
Granted, this is lumping all FCST_INIT_BEG times together... but making the "-by" option work for GO_Index would make this all very easy.
[MET-814] created by johnhg
dwfncar
commented
7 years ago
As of met-6.0, the -job go_index and -job ss_index do not support the "-by" option:
WARNING:
WARNING: The -by option is ignored for the "go_index" job type.
WARNING:
This task is to add that support, particularly to handle "-by FCST_INIT_BEG". Listed below is an email chain which describes the test data to be used on mandan:
(1) I tried running with the "-by" option and see the following:
WARNING:
WARNING: The -by option is ignored for the "go_index" job type.
WARNING:
Perhaps we should make this option work!
(2) One other consideration. Those output directories contain 4 types of point_stat output:
ADPSFC, ADPUPA, WINDS, and ADPSFC_MPR
About 97.5% of the lines existing in the ADPSFC_MPR output type... which isn't used in the GO Index computations!
There is 32GB of MET output for each version...
du -sch
find /d4/projects/VERSION_TNE/v3.5.1 -name "*.stat"But only 1.7GB of that is non-MPR data...
du -sch
find /d4/projects/VERSION_TNE/v3.5.1 -name "*.stat" | egrep -v MPRWhen I skipped over the ADPSFC_MPR files, it took about 4 minutes to process the 9,087 point_stat non-MPR files... and compute Not having to read/discard several million MPR lines makes it run much, much faster.
time /d4/projects/CODE/MET/met-5.2/bin/stat_analysis \
-lookin /d4/projects/VERSION_TNE/v3..1/AFWAOC//metprd/point_statWINDS \
-lookin /d4/projects/VERSION_TNE/v3..1/AFWAOC//metprd/point_statADPUPA \
-lookin /d4/projects/VERSION_TNE/v3..1/AFWAOC//metprd/pointstat*ADPSFC[0-9]* \
-job go_index -by fcst_init_beg \
-model AFWAOC_WRFv3.8.1_d01 -model AFWAOC_WRFv3.5.1_d01 \
-vx_mask CONUS -interp_mthd BILIN -out go_index.tmp \
-v 4 -log run_go_index.log
Granted, this is lumping all FCST_INIT_BEG times together... but making the "-by" option work for GO_Index would make this all very easy. [MET-814] created by johnhg