Add support for EPA AirNow ASCII data in ASCII2NC

DanielAdriaansen commented 2 years ago

Describe the New Feature

Per dtcenter/METplus#1515 and @PerryShafran-NOAA, the EPA will switch from providing BUFR to providing ASCII data to NOAA. The new feature is to add support for this new dataset to ASCII2NC.

Acceptance Testing

Sample files exist on seneca here: /home/dadriaan/projects/airnow/shafran_data/

From dtcenter/METplus#1515:

There are several files here to look at, but they both provide the same data. The file with all the data is the HourlyAQObs data. The HourlyData is a neater looking file with the same data, but the lat/lon of each station is not provided there, so you'll have to use the HourlyAQObs file, which is a CSV file.

It would be great to be able to also read in the daily file, which I think we can use to validate daily model items. That is the daily_data.dat file, and there are two of them. Unfortunately, they are in the HourlyData format, but you should be able to use the lat/lon of the HourlyAQObs file to find these stations. Here's the fact sheet on the daily file:

As you can see, they don't make this easy. However, we could just stick with hourly data and just sum/average them the way we are doing now in PB2NC.

There are four file types: 1) "HourlyAQObs" (Docs: HourlyAQObsFactSheet.pdf) 2) "HourlyData" (Docs: HourlyDataFactSheet.pdf) 3) "daily_data" (Docs: unknown) 4) "daily_data_v2" (Docs: DailyDataFactSheet.pdf)

The documentation for file type 4) only describes "daily_data_v2", and no information was provided about the "daily_data" file from the user. @JohnHalleyGotway notes:

...looking at a diff of the two, I recommend using daily_data_v2.dat. The former does NOT contain lat/lon info for each site and would require a lookup table for that. The v2 version DOES contain the lat/lon info (it would seems) and would therefore be simpler to process.

Time Estimate

1 day of work

Sub-Issues

[ ] Document different file type metadata
[ ] Add support for reading these files in ASCII2NC
[ ] Test support

Relevant Deadlines

NONE.

Funding Source

2792541

Define Related Issue(s)

Consider the impact to the other METplus components.

[ ] METplus: need to update ASCII2NC wrapper

New Feature Checklist

See the METplus Workflow for details.

[x] Complete the issue definition above, including the Time Estimate and Funding source.
[x] Fork this repository or create a branch of develop. Branch name: feature_<Issue Number>_<Description>
[ ] Complete the development and test your changes.
[ ] Add/update log messages for easier debugging.
[ ] Add/update unit tests.
[ ] Add/update documentation.
[ ] Push local changes to GitHub.
[ ] Submit a pull request to merge into develop. Pull request: feature <Issue Number> <Description>
[ ] Define the pull request metadata, as permissions allow. Select: Reviewer(s) and Linked issues Select: Repository level development cycle Project for the next official release Select: Milestone as the next official version
[ ] Iterate until the reviewer(s) accept and merge your changes.
[ ] Delete your fork or branch.
[ ] Close this issue.

Ho-ChunHuang-NOAA commented 1 year ago

John:

Please confirm whether I should use (Dec 7 email above)

module load met/feature_2142

or

module load met/feature_2142_airnow_take2

as mentioned in different email

Thanks.

Ho-Chun Huang, Ph.D.

JohnHalleyGotway commented 1 year ago

@Ho-ChunHuang-NOAA, only 1 of those 2 options would actually work Hera. And it's the first one:

module load intel/2022.1.2
module load anaconda/latest
module use -a /contrib/met/modulefiles/
module load met/feature_2142

As for the location issue, it is not desirable for the ascii2nc tool to automatically pull data from the internet when it is run. Instead, it'll use what is provided to it via the $MET_AIRNOW_STATIONS environment variable. If that's not set, it'll default to the table file included in the repo. I realize that that's out of date, but providing a default does provide some value.

For your use, I'd recommend configuring your run of ascii2nc via METplus with the $MET_AIRNOW_STATIONS environment variable set and pointing to the stations file for the current day. Assuming you have a mechanism for pulling that data onto WCOSS2 every day, @georgemccabe can advise on the easiest way to configure METplus to define the $MET_AIRNOW_STATIONS environment variable to point to it.

Does that work?

Ho-ChunHuang-NOAA commented 1 year ago

Hi, John :

Thanks for the clarification. I will go ahead with the testing.

The EPA ASCII files ingestion is now operational. Daily AirNOW ASCII file are located in wcoss2:/lfs/h1/ops/prod/dcom/yyyymmdd/airnow

@georgemccabe https://github.com/georgemccabe do you have suggestion configuring METplus to define the $MET_AIRNOW_STATIONS environment variable?

Ho-Chun Huang, Ph.D.

georgemccabe commented 1 year ago

Hi @Ho-ChunHuang-NOAA, you can set this environment in the use case METplus config file. You will want to set this variable in the [user_env_vars] section. This will set the environment variable for the METplus commands only. You should be able to reference filename template tags in them. At the bottom of your METplus config file for this use case, add the following:

[user_env_vars]
MET_AIRNOW_STATIONS = /lfs/h1/ops/prod/dcom/{valid?fmt=%Y%m%d}/airnow/Monitoring_Site_Locations_V2.dat

Please let me know if that does not work.

Ho-ChunHuang-NOAA commented 1 year ago

Hi, George:

Thanks for the instruction. I have not constructed the METplus code for ASCII2NC but the info will be helpful in the next face combining ASCII2NC & POINT_STAT processes.

Will report back later.

Thanks again for the help.

Ho-Chun Huang, Ph.D.

Physical Scientist III, Contractor with Lynker in Support of

NOAA/NWS/NCEP/EMC, U.S. Department of Commerce

5830 University Research Ct., Rm. 2792

College Park, MD 20740

@. @.>

301-683-3958

On Fri, Dec 9, 2022 at 4:53 PM George McCabe @.***> wrote:

Hi @Ho-ChunHuang-NOAA https://github.com/Ho-ChunHuang-NOAA, you can set this environment in the use case METplus config file. You will want to set this variable in the [user_env_vars] section. This will set the environment variable for the METplus commands only. You should be able to reference filename template tags in them. At the bottom of your METplus config file for this use case, add the following:

[user_env_vars] MET_AIRNOW_STATIONS = /lfs/h1/ops/prod/dcom/{valid?fmt=%Y%m%d}/airnow/Monitoring_Site_Locations_V2.dat

Please let me know if that does not work.

— Reply to this email directly, view it on GitHub https://github.com/dtcenter/MET/issues/2142#issuecomment-1344815902, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALPHE3BVTRW73BAQRMQJT2LWMOS67ANCNFSM5TZOWZZA . You are receiving this because you were mentioned.Message ID: @.***>

JohnHalleyGotway commented 1 year ago

@Ho-ChunHuang-NOAA any reason why we shouldn't proceed with the MET-11.0.0 release today? Should I proceed with merging these changes and creating the release?

Ho-ChunHuang-NOAA commented 1 year ago

Hi, John:

I have no objection to the merge to MET-11.0.0 release.

Ho-Chun Huang, Ph.D.

Ho-ChunHuang-NOAA commented 1 year ago

Hi, John:

Can you give me your suggestion?

Users who used EPA ASCII AirNOW data reported that there is a negative values in the concentration field. People suggest it should be treated as concentration =0.

Is there a option in ASCII2NC to reset the values to 0 if it is negative and if yes can you show to do it?

Or we can reset the value at the POINT_STAT? i.e., using OBS_VAR1_OPTIONS =. I do not want to to OBS_VAR1_THRESH = >=0 because it is a valid data.

I try to go to discussion and search for the negative or reset, but the keyword search did not come back with useful information.

Ho-ChunHuang-NOAA commented 1 year ago

Are these the correct setting in POINT_STAT?

FCST_VAR1_NAME = OZCON FCST_VAR1_LEVELS = A1 FCST_VAR1_OPTIONS = set_attr_name = "OZCON1"; OBS_VAR1_NAME= OZONE OBS_VAR1_LEVELS= A1 OBS_VAR1_OPTIONS = message_type = "AIRNOW_HOURLY_AQOBS"; censor_thresh = [ <=0.0 ]; censor_val=[ 0.0 ];

FCST_VAR1_NAME = PMTF FCST_VAR1_LEVELS = L1 OBS_VAR1_NAME= PM25 OBS_VAR1_LEVELS= A1 OBS_VAR1_OPTIONS = message_type = "AIRNOW_HOURLY_AQOBS"; censor_thresh = [ <=0.0 ]; censor_val=[ 0.0 ];

Ho-ChunHuang-NOAA commented 1 year ago

John:

I successfully create the hourly o3/pm25 statistic, but I encounter problem using daily files.

matching criteria is FCST_VAR1_NAME = OZMAX1 FCST_VAR1_LEVELS = L1 FCST_VAR1_OPTIONS = valid_time= "{valid?fmt=%Y%m%d?shift=1d}_04"; OBS_VAR1_NAME= OZONE-1HR OBS_VAR1_LEVELS=A1 OBS_VAR1_OPTIONS = message_type = "AIRNOW_DAILY_V2"; valid_time= "{valid?fmt=%Y%m%d}_000000";

FCST_VAR1_NAME = OZMAX8 FCST_VAR1_LEVELS = L1 FCST_VAR1_OPTIONS = valid_time= "{valid?fmt=%Y%m%d?shift=1d}_11"; OBS_VAR1_NAME= OZONE-8HR OBS_VAR1_LEVELS=A8 OBS_VAR1_OPTIONS = message_type = "AIRNOW_DAILY_V2"; valid_time= "{valid?fmt=%Y%m%d}_000000";

FCST_VAR1_NAME = PMTF FCST_VAR1_LEVELS = A23 FCST_VAR1_OPTIONS = valid_time= "{valid?fmt=%Y%m%d?shift=1d}_04";set_attr_name = "PMAVE"; OBS_VAR1_NAME= PM2.5-24hr OBS_VAR1_LEVELS=A24 OBS_VAR1_OPTIONS = message_type = "AIRNOW_DAILY_V2"; valid_time= "{valid?fmt=%Y%m%d}_000000";

I still have problem in daily ascii file. in obs_var name and valid time DEBUG 2: Processing OZMAX1/L1 versus OZONE-1HR/A1, for observation type AIRNOW_DAILY_V2, over region SRockies, for interpolation method NEAREST(1), using 0 matched pairs. DEBUG 2: Number of matched pairs = 0 DEBUG 2: Observations processed = 4078 DEBUG 2: Rejected: station id = 0 DEBUG 2: Rejected: obs var name = 3081 DEBUG 2: Rejected: valid time = 997

DEBUG 2: Processing OZMAX8/L1 versus OZONE-8HR/A8, for observation type AIRNOW_DAILY_V2, over region FULL, for interpolation method NEAREST(1), using 0 matched pairs. DEBUG 2: Number of matched pairs = 0 DEBUG 2: Observations processed = 4078 DEBUG 2: Rejected: station id = 0 DEBUG 2: Rejected: obs var name = 3125 DEBUG 2: Rejected: valid time = 953

DEBUG 2: Processing PMTF/A23 versus PM2.5-24hr/A24, for observation type AIRNOW_DAILY_V2, over region FULL, for interpolation method NEAREST(1), using 0 matched pairs. DEBUG 2: Number of matched pairs = 0 DEBUG 2: Observations processed = 4078 DEBUG 2: Rejected: station id = 0 DEBUG 2: Rejected: obs var name = 2851 DEBUG 2: Rejected: valid time = 1227

Ho-ChunHuang-NOAA commented 1 year ago

Is "-" and "." allow in the point_stat variable name matching ? Should I used the quotation mark " in the OBS_VAR1_NAME?

Ho-ChunHuang-NOAA commented 1 year ago

set new obs window

12/12 17:49:32.519 metplus INFO: METPLUS_FCST_FIELD=field = [{ name="OZMAX1"; level="L1"; valid_time= "20221119_04"; }]; 12/12 17:49:32.519 metplus INFO: METPLUS_INTERP_DICT= 12/12 17:49:32.519 metplus INFO: METPLUS_MASK_GRID=grid = []; 12/12 17:49:32.519 metplus INFO: METPLUS_MASK_POLY=poly = []; 12/12 17:49:32.519 metplus INFO: METPLUS_MASK_SID=sid = []; 12/12 17:49:32.519 metplus INFO: METPLUS_MESSAGE_TYPE=message_type = ["ONLYSF"]; 12/12 17:49:32.519 metplus INFO: METPLUS_MET_CONFIG_OVERRIDES= 12/12 17:49:32.519 metplus INFO: METPLUS_MODEL=model = "PROD_ASCII"; 12/12 17:49:32.519 metplus INFO: METPLUS_OBS_FIELD=field = [{ name="OZONE-1HR"; level="A1"; message_type = "AIRNOW_DAILY_V2"; valid_time= "20221118_000000"; }]; 12/12 17:49:32.519 metplus INFO: METPLUS_OBS_QUALITY= 12/12 17:49:32.519 metplus INFO: METPLUS_OBS_WINDOW_DICT=obs_window = {beg = 864000000;end = -864000000;} 12/12 17:49:32.519 metplus INFO: METPLUS_OUTPUT_FLAG_DICT= 12/12 17:49:32.519 metplus INFO: METPLUS_OUTPUT_PREFIX= 12/12 17:49:32.519 metplus INFO: METPLUS_REGRID_DICT=regrid = {to_grid = NONE;} 12/12 17:49:32.520 metplus INFO: MET_TMP_DIR=/lfs/h2/emc/physics/noscrub/ho-chun.huang/metplus_aq/tmp 12/12 17:49:32.520 metplus INFO: MODEL=PROD_ASCII 12/12 17:49:32.520 metplus INFO: OBS_FIELD={ name="OZONE-1HR"; level="A1"; message_type = "AIRNOW_DAILY_V2"; valid_time= "20221118_000000"; } 12/12 17:49:32.520 metplus INFO: OBS_WINDOW_BEGIN=864000000 12/12 17:49:32.520 metplus INFO: OBS_WINDOW_END=-864000000

result are the same

DEBUG 1: Forecast File: /lfs/h2/emc/ptmp/ho-chun.huang/com/aqm/prod_ascii/cs.20221118/aqm.t06z.max_1hr_o3.148.grib2 DEBUG 1: Observation File: /lfs/h2/emc/vpppg/noscrub/ho-chun.huang/dcom_ascii2nc_airnow/20221118/airnow_daily_20221118.nc DEBUG 3: MetGrib2DataFile::data_plane_array() -> Found exact match for "OZMAX1/L1" in GRIB2 record 1 field 1 of GRIB2 file "/lfs/h2/emc/ptmp/ho-chun.huang/com/aqm/prod_ascii/cs.20221118/aqm.t06z.max_1hr_o3.148.grib2"

:::::::::::::::::::::::::::::::::::::: :::::::::::::::::::::::::::::::::::::

DEBUG 2: Processing OZMAX1/L1 versus OZONE-1HR/A1, for observation type AIRNOW_DAILY_V2, over region FULL, for interpolation method NEAREST(1), using 0 matched pairs. DEBUG 2: Number of matched pairs = 0 DEBUG 2: Observations processed = 4088 DEBUG 2: Rejected: station id = 0 DEBUG 2: Rejected: obs var name = 3111 DEBUG 2: Rejected: valid time = 977

Ho-ChunHuang-NOAA commented 1 year ago

For OZMAX8

DEBUG 2: Processing OZMAX8/L1 versus OZONE-8HR/A8, for observation type AIRNOW_DAILY_V2, over region FULL, for interpolation method NEAREST(1), using 0 matched pairs. DEBUG 2: Number of matched pairs = 0 DEBUG 2: Observations processed = 4088 DEBUG 2: Rejected: station id = 0 DEBUG 2: Rejected: obs var name = 3143 DEBUG 2: Rejected: valid time = 945

For PMAVE DEBUG 2: Processing PMTF/A23 versus PM2.5-24hr/A24, for observation type AIRNOW_DAILY_V2, over region FULL, for interpolation method NEAREST(1), using 0 matched pairs. DEBUG 2: Number of matched pairs = 0 DEBUG 2: Observations processed = 4088 DEBUG 2: Rejected: station id = 0 DEBUG 2: Rejected: obs var name = 2841 DEBUG 2: Rejected: valid time = 1247

JohnHalleyGotway commented 1 year ago

@Ho-ChunHuang-NOAA, please try multiplying your settings by -1. Here's what I see in the log messages you sent:

12/12 17:49:32.519 metplus INFO: METPLUS_OBS_WINDOW_DICT=obs_window = {beg = 864000000;end = -864000000;}

These are both relative to the forecast valid time. The end value should be >= beg value. But you have the signs switched. Please try setting beg = -864000000 and end = 864000000.

Perhaps we should add a sanity check, and have MET error out if end < beg?

Ho-ChunHuang-NOAA commented 1 year ago

Update the obs_window, the error message are the same

12/12 19:50:18.452 metplus INFO: METPLUS_OBS_WINDOW_DICT=obs_window = {beg = -864000000;end = 864000000;}

DEBUG 2: Processing OZMAX1/L1 versus OZONE-1HR/A1, for observation type AIRNOW_DAILY_V2, over region FULL, for interpolation method NEAREST(1), using 0 matched pairs. DEBUG 2: Number of matched pairs = 0 DEBUG 2: Observations processed = 4088 DEBUG 2: Rejected: station id = 0 DEBUG 2: Rejected: obs var name = 3111 DEBUG 2: Rejected: valid time = 977

DEBUG 2: Processing OZMAX8/L1 versus OZONE-8HR/A8, for observation type AIRNOW_DAILY_V2, over region FULL, for interpolation method NEAREST(1), using 0 matched pairs. DEBUG 2: Number of matched pairs = 0 DEBUG 2: Observations processed = 4088 DEBUG 2: Rejected: station id = 0 DEBUG 2: Rejected: obs var name = 3143 DEBUG 2: Rejected: valid time = 945

DEBUG 2: Processing PMTF/A23 versus PM2.5-24hr/A24, for observation type AIRNOW_DAILY_V2, over region CONUS_Central, for interpolation method NEAREST(1), using 0 matched pairs. DEBUG 2: Number of matched pairs = 0 DEBUG 2: Observations processed = 4088 DEBUG 2: Rejected: station id = 0 DEBUG 2: Rejected: obs var name = 2841 DEBUG 2: Rejected: valid time = 1247

JohnHalleyGotway commented 1 year ago

@Ho-ChunHuang-NOAA, I asked Julie to pull down those two sample files from WCOSS2:

DEBUG 1: Forecast File: /lfs/h2/emc/ptmp/ho-chun.huang/com/aqm/prod_ascii/cs.20221118/aqm.t06z.max_1hr_o3.148.grib2
DEBUG 1: Observation File: /lfs/h2/emc/vpppg/noscrub/ho-chun.huang/dcom_ascii2nc_airnow/20221118/airnow_daily_20221118.nc

And I ran Point-Stat directly with this data:

/d1/projects/MET/MET_releases/met-10.1.0/bin/point_stat \
aqm.t06z.max_1hr_o3.148.grib2 airnow_daily_20221118.nc \
PointStatConfig.txt -outdir out -v 3 -log run_ps.log

I used MET version 10.1.0 because presumably that's what you're running on WCOSS2. Point-Stat (both versions 10.1.0 and 11.0.0) produce 888 matched pairs:

DEBUG 2: Processing OZMAX1/L1 versus OZONE-1HR/A1, for observation type AIRNOW_DAILY_V2, over region FULL, for interpolation method NEAREST(1), using 888 matched pairs.
DEBUG 3: Number of matched pairs   = 888
DEBUG 3: Observations processed    = 4088
DEBUG 3: Rejected: obs var name    = 3111
DEBUG 3: Rejected: off the grid    = 89

Note that you should probably remove the valid_time setting from:

12/12 17:49:32.520 metplus INFO: OBS_FIELD={ name="OZONE-1HR"; level="A1"; message_type = "AIRNOW_DAILY_V2"; valid_time= "20221118_000000"; }

It serves no purpose here. Instead the obs_window settings are used to filter point observations by time.

Here are my config and log files: PointStatConfig.txt run_ps.log

Perhaps you could try running Point-Stat directly using this config file to take the METplus python wrappers out of the equation while debugging?

Ho-ChunHuang-NOAA commented 1 year ago

Hi, John:

I followed your instructions and used your PointStatConfig.txt https://github.com/dtcenter/MET/files/10212097/PointStatConfig.txt and got the same result.

I will remove the valid time in OBS_VAR1_OPTIONS and rerun my script.

Ho-Chun Huang, Ph.D.

Physical Scientist III, Contractor with Lynker in Support of

NOAA/NWS/NCEP/EMC, U.S. Department of Commerce

5830 University Research Ct., Rm. 2792

College Park, MD 20740

@. @.>

301-683-3958

On Mon, Dec 12, 2022 at 4:19 PM johnhg @.***> wrote:

@Ho-ChunHuang-NOAA https://github.com/Ho-ChunHuang-NOAA, I asked Julie to pull down those two sample files from WCOSS2:

DEBUG 1: Forecast File: /lfs/h2/emc/ptmp/ho-chun.huang/com/aqm/prod_ascii/cs.20221118/aqm.t06z.max_1hr_o3.148.grib2 DEBUG 1: Observation File: /lfs/h2/emc/vpppg/noscrub/ho-chun.huang/dcom_ascii2nc_airnow/20221118/airnow_daily_20221118.nc

And I ran Point-Stat directly with this data:

/d1/projects/MET/MET_releases/met-10.1.0/bin/point_stat \ aqm.t06z.max_1hr_o3.148.grib2 airnow_daily_20221118.nc \ PointStatConfig.txt -outdir out -v 3 -log run_ps.log

I used MET version 10.1.0 because presumably that's what you're running on WCOSS2. Point-Stat (versions 10.1.0 and 11.0.0) both produce 888 matched pairs:

DEBUG 2: Processing OZMAX1/L1 versus OZONE-1HR/A1, for observation type AIRNOW_DAILY_V2, over region FULL, for interpolation method NEAREST(1), using 888 matched pairs. DEBUG 3: Number of matched pairs = 888 DEBUG 3: Observations processed = 4088 DEBUG 3: Rejected: obs var name = 3111 DEBUG 3: Rejected: off the grid = 89

Note that you should probably remove the valid_time setting from:

12/12 17:49:32.520 metplus INFO: OBS_FIELD={ name="OZONE-1HR"; level="A1"; message_type = "AIRNOW_DAILY_V2"; valid_time= "20221118_000000"; }

It serves no purpose here. Instead the obs_window settings are used to filter point observations by time.

Here are my config and log files: PointStatConfig.txt https://github.com/dtcenter/MET/files/10212097/PointStatConfig.txt run_ps.log https://github.com/dtcenter/MET/files/10212096/run_ps.log

Perhaps you could try running Point-Stat directly using this config file to take the METplus python wrappers out of the equation while debugging?

— Reply to this email directly, view it on GitHub https://github.com/dtcenter/MET/issues/2142#issuecomment-1347344065, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALPHE3G5N6BKE5DBXENRDVLWM6JE7ANCNFSM5TZOWZZA . You are receiving this because you were mentioned.Message ID: @.***>

Ho-ChunHuang-NOAA commented 1 year ago

I remove the OBS_VAR1_OPTIONS validtime description and re-run, I got the same result as with validtime description.

This is the only place I specify the OBS_VAR1_OPTIONS with valid_time. FCST_VAR1_NAME = OZMAX1 FCST_VAR1_LEVELS = L1 FCST_VAR1_OPTIONS = valid_time= "{valid?fmt=%Y%m%d?shift=1d}_04"; OBS_VAR1_NAME= OZONE-1HR OBS_VAR1_LEVELS=A1 OBS_VAR1_OPTIONS = message_type = "AIRNOW_DAILY_V2";

https://www.emc.ncep.noaa.gov/mmb/hchuang/ftp/run_ozmax1_2142.log

JohnHalleyGotway commented 1 year ago

Thanks for sending that log file. I'm pretty stumped. Its as if the METplus obs_window settings are actually being used by the Point-Stat tool. I clearly see that the environment variable is being set the way we want:

12/12 22:00:13.342 metplus INFO: METPLUS_OBS_WINDOW_DICT=obs_window = {beg = -864000000;end = 864000000;}

Next, I'd try running MET at verbosity level 10 (-v 10). Here's what I'm looking for in the output:

|    |    Name  = obs_window
|    |    Type  = DictionaryType
|    |    Dict/Array Value = 
|    |    |    Nentries = 2
|    |    |    Address  = 0x56158e382010
|    |    |    Parent   = 0x56158d542aa8
|    |    |    IsArray  = false
|    |    |    Dictionary Entry[0] ...
|    |    |    |    Name  = beg
|    |    |    |    Type  = IntegerType
|    |    |    |    Integer Value = -864000000
|    |    |    Dictionary Entry[1] ...
|    |    |    |    Name  = end
|    |    |    |    Type  = IntegerType
|    |    |    |    Integer Value = 864000000

BUT this is not written to the log output, its written to standard out. I'm not sure if METplus captures that output or not.

This would confirm that MET is parsing the obs_window as expected. If you see in your output that its some value other than 864000000, that'd indicate some issue with config options being overwritten.

Ho-ChunHuang-NOAA commented 1 year ago

Hi, John:

I need to take care of other issues at this moment. I need time to analyze what you said, I have no clue what is the problem at this moment. I will investigate it and get back to you on this issue later.

While working on the 2020 cases, I discovered the timestamps in HourlyAQObs* between 2020 and 2022 are different, Can current ascii2nc handle the year specification of 2 digit and 4 digits?

"09/01/20"

"09/01/2022"

p.s. HourlyData* using a two-digit year between 2020 and 2022. Current year daily_data_v2.dat using a two-digit year.

Ho-Chun Huang, Ph.D.

Physical Scientist III, Contractor with Lynker in Support of

NOAA/NWS/NCEP/EMC, U.S. Department of Commerce

5830 University Research Ct., Rm. 2792

College Park, MD 20740

@. @.>

301-683-3958

On Tue, Dec 13, 2022 at 7:03 PM johnhg @.***> wrote:

Thanks for sending that log file. I'm pretty stumped. Its as if the METplus obs_window settings are actually being used by the Point-Stat tool. I clearly see that the environment variable is being set the way we want:

12/12 22:00:13.342 metplus INFO: METPLUS_OBS_WINDOW_DICT=obs_window = {beg = -864000000;end = 864000000;}

Next, I'd try running MET at verbosity level 10 (-v 10). Here's what I'm looking for in the output:

| | Name = obs_window | | Type = DictionaryType | | Dict/Array Value = | | | Nentries = 2 | | | Address = 0x56158e382010 | | | Parent = 0x56158d542aa8 | | | IsArray = false | | | Dictionary Entry[0] ... | | | | Name = beg | | | | Type = IntegerType | | | | Integer Value = -864000000 | | | Dictionary Entry[1] ... | | | | Name = end | | | | Type = IntegerType | | | | Integer Value = 864000000

BUT this is not written to the log output, its written to standard out. I'm not sure if METplus captures that output or not.

This would confirm that MET is parsing the obs_window as expected. If you see in your output that its some value other than 864000000, that'd indicate some issue with config options being overwritten.

— Reply to this email directly, view it on GitHub https://github.com/dtcenter/MET/issues/2142#issuecomment-1350114483, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALPHE3DGBBTTMNVF4NBBVF3WNEFENANCNFSM5TZOWZZA . You are receiving this because you were mentioned.Message ID: @.***>

Ho-ChunHuang-NOAA commented 1 year ago

Got it, I will repeat the run with -v 10.

georgemccabe commented 1 year ago

Hi @Ho-ChunHuang-NOAA, it looks like you are using your own MET config file that does not contain the environment variables that are set by the METplus wrappers. There are many warnings in your log file that mention that the environment variables are not utilized:

12/12 22:00:13.331 metplus (command_builder.py:153) WARNING: Environment variable ${METPLUS_OBS_WINDOW_DICT} is not utilized in MET config file: /lfs/h2/emc/vpppg/save/ho-chun.huang/METplus-4.0.0/parm/met_config/PointStatConfig_AIRNOW_max_Bukovsky_G148

Removing or commenting out POINT_STAT_CONFIG_FILE in your METplus config file will use the wrapped MET config file that is provided with the METplus wrappers (METplus/parm/met_config/PointStatConfig_wrapped) and the values from the environment variables set by the wrapper will be used. If you have changed any values from the defaults in PointStatConfig_AIRNOW_max_Bukovsky_G148, then you will need to set the appropriate settings in your METplus config file. You can compare this file to the default config file found in the share/met/config directory of the MET install. The path to the default file will also be output in the point_stat log output. If you have issues reconciling the differences, you could send me the PointStatConfig_AIRNOW_max_Bukovsky_G148 file.

Ho-ChunHuang-NOAA commented 1 year ago

Hi, George:

Thanks for the tip. I will review my setting as suggested.

Ho-Chun Huang, Ph.D.

Ho-ChunHuang-NOAA commented 1 year ago

Hi, George:

I use the template you suggested but the results still lead to zero number of matches. Can you take a look at the PointStatConfig I used and let me know your comment? Thank you for your help.

The files can be found in https://www.emc.ncep.noaa.gov/mmb/hchuang/ftp/

PointStatConfig_AIRNOW_max_test run_2142.log

I did cp /apps/ops/prod/libs/intel/19.1.3.304/met/10.1.2/share/met/config/PointStatConfig_default . then edit PointStatConfig_default to PointStatConfig_AIRNOW_max_test. Then I set POINT_STAT_CONFIG_FILE ={PARM_BASE}/met_config/PointStatConfig_AIRNOW_max_test

If I do not modified, the hardwire fcst_field and obs_field and not masking region leads to no observation matching information.

Modification I made to avoid hardwire field < model = "${MODEL}";

model = "WRF"; 43c43 < cat_thresh = [ >50, >60, >65, >70, >75, >85, >105, >125, >150 ];

cat_thresh = [ NA ]; 56c56,84 < field = [ ${FCST_FIELD} ];

field = [ { name = "SPFH"; level = [ "P500" ]; cat_thresh = [ >80.0 ]; },
  {
    name       = "TMP";
    level      = [ "P500" ];
    cat_thresh = [ >273.0 ];
  },

  {
    name       = "HGT";
    level      = [ "P500" ];
    cat_thresh = [ >0.0 ];
  },

  {
    name       = "UGRD";
    level      = [ "P500" ];
    cat_thresh = [ >5.0 ];
  },

  {
    name       = "VGRD";
    level      = [ "P500" ];
    cat_thresh = [ >5.0 ];
57a86 ]; 59,60d87 < obs = { < field = [ ${OBS_FIELD} ]; 61a89 obs = fcst; 69c97 < message_type = [];

message_type = [ "ADPUPA" ]; 179,182c207,208 < // grid = [ "FULL" ]; < // poly = []; < // poly = [ "/lfs/h2/emc/vpppg/noscrub/emc.vpppg/verification/EVS_fix/masks/Bukovsky_G148_Appalachia.nc", "/lfs/h2/emc/vpppg/noscrub/emc.vpppg/verification/EVS_fix/masks/Bukovsky_G148_CONUS_Central.nc", "/lfs/h2/emc/vpppg/noscrub/emc.vpppg/verification/EVS_fix/masks/Bukovsky_G148_CONUS_East.nc", "/lfs/h2/emc/vpppg/noscrub/emc.vpppg/verification/EVS_fix/masks/Bukovsky_G148_CONUS.nc", "/lfs/h2/emc/vpppg/noscrub/emc.vpppg/verification/EVS_fix/masks/Bukovsky_G148_CONUS_South.nc", "/lfs/h2/emc/vpppg/noscrub/emc.vpppg/verification/EVS_fix/masks/Bukovsky_G148_CONUS_West.nc", "/lfs/h2/emc/vpppg/noscrub/emc.vpppg/verification/EVS_fix/masks/Bukovsky_G148_CPlains.nc", "/lfs/h2/emc/vpppg/noscrub/emc.vpppg/verification/EVS_fix/masks/Bukovsky_G148_DeepSouth.nc", "/lfs/h2/emc/vpppg/noscrub/emc.vpppg/verification/EVS_fix/masks/Bukovsky_G148_GreatBasin.nc", "/lfs/h2/emc/vpppg/noscrub/emc.vpppg/verification/EVS_fix/masks/Bukovsky_G148_GreatLakes.nc", "/lfs/h2/emc/vpppg/noscrub/emc.vpppg/verification/EVS_fix/masks/Bukovsky_G148_Mezquital.nc", "/lfs/h2/emc/vpppg/noscrub/emc.vpppg/verification/EVS_fix/masks/Bukovsky_G148_MidAtlantic.nc", "/lfs/h2/emc/vpppg/noscrub/emc.vpppg/verification/EVS_fix/masks/Bukovsky_G148_NorthAtlantic.nc", "/lfs/h2/emc/vpppg/noscrub/emc.vpppg/verification/EVS_fix/masks/Bukovsky_G148_NPlains.nc", "/lfs/h2/emc/vpppg/noscrub/emc.vpppg/verification/EVS_fix/masks/Bukovsky_G148_NRockies.nc", "/lfs/h2/emc/vpppg/noscrub/emc.vpppg/verification/EVS_fix/masks/Bukovsky_G148_PacificNW.nc", "/lfs/h2/emc/vpppg/noscrub/emc.vpppg/verification/EVS_fix/masks/Bukovsky_G148_PacificSW.nc", "/lfs/h2/emc/vpppg/noscrub/emc.vpppg/verification/EVS_fix/masks/Bukovsky_G148_Prairie.nc", "/lfs/h2/emc/vpppg/noscrub/emc.vpppg/verification/EVS_fix/masks/Bukovsky_G148_Southeast.nc", "/lfs/h2/emc/vpppg/noscrub/emc.vpppg/verification/EVS_fix/masks/Bukovsky_G148_Southwest.nc", "/lfs/h2/emc/vpppg/noscrub/emc.vpppg/verification/EVS_fix/masks/Bukovsky_G148_SPlains.nc", "/lfs/h2/emc/vpppg/noscrub/emc.vpppg/verification/EVS_fix/masks/Bukovsky_G148_SRockies.nc" ]; < poly = [ "/lfs/h2/emc/vpppg/noscrub/emc.vpppg/verification/EVS_fix/masks/Bukovsky_G148_CONUS.nc" ];

grid = [ "FULL" ]; poly = []; 243,244c269,270 < fho = STAT; < ctc = STAT;

fho = NONE; ctc = NONE; 249c275 < sl1l2 = STAT;

sl1l2 = NONE; 267c293 < tmp_dir = "/lfs/h2/emc/ptmp/Ho-Chun.Huang/tmp";

tmp_dir = "/tmp";

Ho-Chun Huang, Ph.D.

Physical Scientist III, Contractor with Lynker in Support of

NOAA/NWS/NCEP/EMC, U.S. Department of Commerce

5830 University Research Ct., Rm. 2792

College Park, MD 20740

@. @.>

301-683-3958

On Wed, Dec 14, 2022 at 10:43 AM Ho-Chun Huang - NOAA Affiliate < @.***> wrote:

Hi, George:

Thanks for the tip. I will review my setting as suggested.

Ho-Chun Huang, Ph.D.

Physical Scientist III, Contractor with Lynker in Support of

NOAA/NWS/NCEP/EMC, U.S. Department of Commerce

5830 University Research Ct., Rm. 2792

College Park, MD 20740

@. @.>

301-683-3958

On Wed, Dec 14, 2022 at 10:32 AM George McCabe @.***> wrote:

Hi @Ho-ChunHuang-NOAA https://github.com/Ho-ChunHuang-NOAA, it looks like you are using your own MET config file that does not contain the environment variables that are set by the METplus wrappers. There are many warnings in your log file that mention that the environment variables are not utilized:

12/12 22:00:13.331 metplus (command_builder.py:153) WARNING: Environment variable ${METPLUS_OBS_WINDOW_DICT} is not utilized in MET config file: /lfs/h2/emc/vpppg/save/ho-chun.huang/METplus-4.0.0/parm/met_config/PointStatConfig_AIRNOW_max_Bukovsky_G148

Removing or commenting out POINT_STAT_CONFIG_FILE in your METplus config file will use the wrapped MET config file that is provided with the METplus wrappers (METplus/parm/met_config/PointStatConfig_wrapped) and the values from the environment variables set by the wrapper will be used. If you have changed any values from the defaults in PointStatConfig_AIRNOW_max_Bukovsky_G148, then you will need to set the appropriate settings in your METplus config file. You can compare this file to the default config file found in the share/met/config directory of the MET install. The path to the default file will also be output in the point_stat log output. If you have issues reconciling the differences, you could send me the PointStatConfig_AIRNOW_max_Bukovsky_G148 file.

— Reply to this email directly, view it on GitHub https://github.com/dtcenter/MET/issues/2142#issuecomment-1351638990, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALPHE3BC4UQM26U6YHGUEALWNHR5VANCNFSM5TZOWZZA . You are receiving this because you were mentioned.Message ID: @.***>

georgemccabe commented 1 year ago

Hi @Ho-ChunHuang-NOAA, I was suggesting that you use the PointStatConfig_wrapped file that is provided with the METplus install instead of modifying the defaults by hand. Using the PointStatConfig_AIRNOW_max_test file still produces the warnings that the environment variables are not in your file, so it does not include the variable that sets the obswindow values. It also referenced environment variables that are deprecated and will be soon removed from support (only environment variables that start with METPLUS will be set by METplus to use in MET config files).

You should set the following in your METplus config file:

POINT_STAT_CONFIG_FILE = {PARM_BASE}/met_config/PointStatConfig_wrapped
POINT_STAT_MASK_POLY = /lfs/h2/emc/vpppg/noscrub/emc.vpppg/verification/EVS_fix/masks/Bukovsky_G148_CONUS.nc
POINT_STAT_OUTPUT_FLAG_FHO = STAT
POINT_STAT_OUTPUT_FLAG_CTC = STAT
POINT_STAT_OUTPUT_FLAG_SL1L2 = STAT
POINT_STAT_MET_CONFIG_OVERRIDES = cat_thresh = [ >50, >60, >65, >70, >75, >85, >105, >125, >150 ]; message_type   = [];

This should set the desired overrides and also read the obs_window values that you have set. Let me know if this does not fix the issue.

I'm not sure which METplus config file to put these changes because it looks like you are passing in many config files to run_metplus.py. Previously I had helped @PerryShafran-NOAA update a CAM use case to make changes like this and reduce the number of config files used to run in this discussion. It would be good to update all of the use cases that you run in this way. I am happy help with these updates and/or meet with you to discuss what is needed to change.

Ho-ChunHuang-NOAA commented 1 year ago

Hi, George:

I followed the instructions in the previous email, using /apps/ops/prod/libs/intel/19.1.3.304/metplus/4.1.4/parm/met_config/PointStatConfig_wrapped.

The error message occurred, what is the "METPLUS_HSS_EC_VALUE" value I should give in the top script? I have never used this variable before.

DEBUG 1: Default Config File: /apps/ops/prod/libs/intel/19.1.3.304/met/10.1.2/share/met/config/PointStatConfig_default DEBUG 1: User Config File: /lfs/h2/emc/vpppg/save/ho-chun.huang/METplus-4.0.0/parm/met_config/PointStatConfig_wrapped ERROR : ERROR : replace_env() -> unable to get value for environment variable "METPLUS_HSS_EC_VALUE" ERROR :

Thank you.

Ho-Chun Huang, Ph.D.

Physical Scientist III, Contractor with Lynker in Support of

NOAA/NWS/NCEP/EMC, U.S. Department of Commerce

5830 University Research Ct., Rm. 2792

College Park, MD 20740

@. @.>

301-683-3958

On Wed, Dec 14, 2022 at 1:26 PM George McCabe @.***> wrote:

Hi @Ho-ChunHuang-NOAA https://github.com/Ho-ChunHuang-NOAA, I was suggesting that you use the PointStatConfig_wrapped file that is provided with the METplus install instead of modifying the defaults by hand. Using the PointStatConfig_AIRNOW_max_test file still produces the warnings that the environment variables are not in your file, so it does not include the variable that sets the obswindow values. It also referenced environment variables that are deprecated and will be soon removed from support (only environment variables that start with METPLUS will be set by METplus to use in MET config files).

You should set the following in your METplus config file:

POINT_STAT_CONFIG_FILE = {PARM_BASE}/met_config/PointStatConfig_wrapped POINT_STAT_MASK_POLY = /lfs/h2/emc/vpppg/noscrub/emc.vpppg/verification/EVS_fix/masks/Bukovsky_G148_CONUS.nc POINT_STAT_OUTPUT_FLAG_FHO = STAT POINT_STAT_OUTPUT_FLAG_CTC = STAT POINT_STAT_OUTPUT_FLAG_SL1L2 = STAT POINT_STAT_MET_CONFIG_OVERRIDES = cat_thresh = [ >50, >60, >65, >70, >75, >85, >105, >125, >150 ]; message_type = [];

This should set the desired overrides and also read the obs_window values that you have set. Let me know if this does not fix the issue.

I'm not sure which METplus config file to put these changes because it looks like you are passing in many config files to run_metplus.py. Previously I had helped @PerryShafran-NOAA https://github.com/PerryShafran-NOAA update a CAM use case to make changes like this and reduce the number of config files used to run in this discussion https://github.com/dtcenter/METplus/discussions/996#discussioncomment-1304157. It would be good to update all of the use cases that you run in this way. I am happy help with these updates and/or meet with you to discuss what is needed to change.

— Reply to this email directly, view it on GitHub https://github.com/dtcenter/MET/issues/2142#issuecomment-1351911751, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALPHE3DBOQDXGXXYLDNPODDWNIGOPANCNFSM5TZOWZZA . You are receiving this because you were mentioned.Message ID: @.***>

georgemccabe commented 1 year ago

@Ho-ChunHuang-NOAA, it looks like you are running METplus 4.0.0 using the param file from 4.1.4. METPLUS_HSS_EC_VALUE is not set by METplus 4.0, but it is set in 4.1. The environment variable is referenced in the 4.1.4 config file, but it is not set by METplus 4.0.

Ho-ChunHuang-NOAA commented 1 year ago

Hi, George:

I am using metplus/4.1.4. METplus-4.0.0 is simply the directory I have not changed.

module purge export HPC_OPT=/apps/ops/prod/libs module use /apps/ops/prod/libs/modulefiles/compiler/intel/19.1.3.304/ module load intel module load gsl module load python/3.8.6 module load netcdf/4.7.4 module load met/10.1.2 module load metplus/4.1.4

Ho-Chun Huang, Ph.D.

Physical Scientist III, Contractor with Lynker in Support of

NOAA/NWS/NCEP/EMC, U.S. Department of Commerce

5830 University Research Ct., Rm. 2792

College Park, MD 20740

@. @.>

301-683-3958

On Wed, Dec 14, 2022 at 5:42 PM George McCabe @.***> wrote:

@Ho-ChunHuang-NOAA https://github.com/Ho-ChunHuang-NOAA, it looks like you are running METplus 4.0.0 using the param file from 4.1.4. METPLUS_HSS_EC_VALUE is not set by METplus 4.0, but it is set in 4.1. The environment variable is referenced in the 4.1.4 config file, but it is not set by METplus 4.0.

— Reply to this email directly, view it on GitHub https://github.com/dtcenter/MET/issues/2142#issuecomment-1352318384, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALPHE3EWMDUSFIQLN6ZKBGTWNJENVANCNFSM5TZOWZZA . You are receiving this because you were mentioned.Message ID: @.***>

georgemccabe commented 1 year ago

The last log file you sent shows that you are running METplus v4.0.0:

12/14 16:49:29.713 metplus INFO: Running METplus v4.0.0 called with command: /lfs/h2/emc/vpppg/save/ho-chun.huang/METplus-4.0.0/ush/run_metplus.py ...

After you run the module commands, could you run which run_metplus.py to see which version of that script is being used? It looks like it should be 4.1.4, but that isn't consistent with what is shown in the logs.

georgemccabe commented 1 year ago

@Ho-ChunHuang-NOAA, you could also check the call to run_metplus.py. The command may include the full path to the script which may point to an older version.

dtcenter / MET