search

NCAR / i-wrf

Integrated End-To-End WRF Containerized Framework
https://i-wrf.org
Apache License 2.0
6 stars 1 forks source link

Run Hurricane Matthew test case in METplus container #48

Closed jaredalee closed 1 month ago

jaredalee commented 4 months ago

Describe the Task

Using the WRF output generated by #46 and the observation data obtained in #47, generate some sample verification statistics with the METplus container. Document what commands/steps were used. This will also serve as another test to make sure that #5 is working properly.

Time Estimate

Estimate the amount of work required here. Issues should represent approximately 1 to 3 days of work.

Sub-Issues

Consider breaking the task down into sub-issues.

Relevant Deadlines

List relevant project deadlines here or state NONE.

Funding Source

7790013

Define the Metadata

Assignee

Labels

Projects and Milestone

Task Checklist

jaredalee commented 3 months ago

As a placeholder until the WRF output from the Hurricane Matthew test case from the container is available, the output from a standalone version of WRF for this case is here: /glade/derecho/scratch/jaredlee/nsf_i-wrf/matthew/wrf

JohnHalleyGotway commented 3 months ago

@georgemccabe, @jaredalee, and @JohnHalleyGotway met on April 17th for a working session.

Steps:

georgemccabe commented 3 months ago

In branch https://github.com/NCAR/i-wrf/tree/feature/48_matthew_metplus, I created the directory structure for the use cases from @JohnHalleyGotway's comment above. I also committed the in-progress METplus configuration file for the Hurricane Matthew example. The config file has been set to read the WRF data (using a hard-coded path).

jprestop commented 2 months ago

@JohnHalleyGotway and @georgemccabe The latest METplus beta releases are now installed on Casper. The Existing Builds page has been updated to reflected these installations. Please let me know if you have any questions or problems.

jaredalee commented 2 months ago

George & I just finished a working meeting, where we set up the METplus config file (in the commit referenced just above) to:

a) read in the WRF output file successfully b) read in the METAR obs file successfully c) read and calculate stats for 2-m T, 10-m U, and 10-m V as an initial test

We uncovered a bug in MET where it doesn't currently read in multiple obs files. George will create an issue and then fix that. Next time we'll add configurations to handle processing the RAOB obs file and determine if MET is properly interpolating native-grid WRF output to pressure levels, or if WRF needs to be re-run to output variables on specific pressure levels in a different file.

JohnHalleyGotway commented 2 months ago

During the i-wrf tech team meeting on 5/3/24, @dudhia recommended the following:

Note that these changes are rather large and should probable be done via a separate issue.

For this issue while working on the hurricane Matthew instructions, please just scope out the idea, and if feasible, write up a separate issue to describe this potential future enhancement.

georgemccabe commented 2 months ago

These notes expand on the previous comment and should be added to the new issue when it is created:

There is already a METplus wrapper for the GFDLTracker. I created a Docker image on DockerHub that contains the tracker that is used to run it for the METplus automated tests.

https://github.com/dtcenter/METplus/blob/develop/.github/actions/run_tests/Dockerfile.run_gfdl

This Dockerfile could be copied and modified to create a Docker container that includes METplus and the tracker.

The image created includes patches so it can be compiled using GCC instead of Intel. We could instead compile it using Intel since we already compile WRF using Intel. Here is the Dockerfile that builds the tracker using GCC: https://github.com/dtcenter/METplus/blob/develop/internal/scripts/docker_env/Dockerfile.gfdl-tracker

JohnHalleyGotway commented 2 months ago

Here's a list of recommended improvements to MET/METplus:

  1. In MET Point-Stat, we're requesting only VL1L2, not VCNT, but are getting a warning about computing wind direction stats for 0-vectors:

    WARNING: VL1L2Info::compute_stats() -> Skipping 85 of 459 vector pairs for which the direction difference is undefined.
WARNING: Set the "wind_thresh" and "wind_logic" configuration options to exclude zero vectors.

    Only warn about this when VCNT output is requested.

  2. In METplus PointStat wrapper, add support for setting wind_thresh and wind_logic that we used for this use case but are not currently supported by the wrapper. When this is added, you can replace the following override settings:

    POINT_STAT_MET_CONFIG_OVERRIDES = wind_thresh   = [ >2 ]; wind_logic = INTERSECTION;

  3. dtcenter/METplus#2567 to fix the Point-Stat wrapper bug for multiple observation files.

  4. dtcenter/METplus#1514 to create a MADIS2NC wrapper.

georgemccabe commented 2 months ago

I was able to get a successful METplus run using apptainer on casper.

To load apptainer and pull the METplus image:

module load apptainer

cd ${SCRATCH}
apptainer pull docker://dtcenter/metplus:6.0.0-beta4

This creates a local file called metplus_6.0.0-beta4.sif.

Create local output directory:

mkdir ${SCRATCH}/metplus_out

Set environment variable to bind directories to container (note: this can also be accomplished by passing the value on the command line using the --bind argument)

LOCAL_METPLUS_CONFIG_DIR=/glade/u/home/mccabe/i-wrf/use_cases/Hurricane_Matthew/METplus
LOCAL_FCST_INPUT_DIR=/glade/derecho/scratch/jaredlee/nsf_i-wrf/matthew
LOCAL_UPPER_AIR_OBS_INPUT_DIR=/glade/campaign/ral/wsap/i-wrf/data/hurr-matthew/madis/point/raob/netcdf
LOCAL_SURFACE_OBS_INPUT_DIR=/glade/campaign/ral/wsap/i-wrf/data/hurr-matthew/madis/point/metar/netcdf
LOCAL_OUTPUT_DIR=${SCRATCH}/metplus_out

export APPTAINER_BIND="${LOCAL_METPLUS_CONFIG_DIR}:/config,${LOCAL_FCST_INPUT_DIR}:/data/input/wrf,${LOCAL_UPPER_AIR_OBS_INPUT_DIR}:/data/input/obs/raob,${LOCAL_SURFACE_OBS_INPUT_DIR}:/data/input/obs/metar,${LOCAL_OUTPUT_DIR}:/data/output"

Execute the run_metplus.py command inside the container to run the use case:

apptainer exec metplus_6.0.0-beta4.sif /metplus/METplus/ush/run_metplus.py /config/PointStat_matthew.conf

Check that the output data was created locally:

ls ${SCRATCH}/metplus_out/point_stat -1

point_stat_surface_360000L_20161007_120000V_cnt.txt point_stat_surface_360000L_20161007_120000V.stat point_stat_surface_360000L_20161007_120000V_vcnt.txt point_stat_upper_air_360000L_20161007_120000V_cnt.txt point_stat_upper_air_360000L_20161007_120000V.stat point_stat_upper_air_360000L_20161007_120000V_vcnt.txt

If you need to run on casper outside of apptainer, you can override the input directory values with the actual paths on casper:

module use /glade/work/dtcrt/METplus/casper/components/METplus/installations/modulefiles
module load metplus/6.0.0-beta4

LOCAL_FCST_INPUT_DIR=/glade/derecho/scratch/jaredlee/nsf_i-wrf/matthew
LOCAL_UPPER_AIR_OBS_INPUT_DIR=/glade/campaign/ral/wsap/i-wrf/data/hurr-matthew/madis/point/raob/netcdf
LOCAL_SURFACE_OBS_INPUT_DIR=/glade/campaign/ral/wsap/i-wrf/data/hurr-matthew/madis/point/metar/netcdf
LOCAL_OUTPUT_DIR=${SCRATCH}/metplus_out2

run_metplus.py \
/glade/u/home/mccabe/i-wrf/use_cases/Hurricane_Matthew/METplus/PointStat_matthew.conf \
config.FCST_POINT_STAT_INPUT_DIR=${LOCAL_FCST_INPUT_DIR} \
surface.OBS_POINT_STAT_INPUT_DIR=${LOCAL_SURFACE_OBS_INPUT_DIR} \
upper_air.OBS_POINT_STAT_INPUT_DIR=${LOCAL_UPPER_AIR_OBS_INPUT_DIR} \
config.OUTPUT_BASE=${LOCAL_OUTPUT_DIR}

Resources: Apptainer Quick Start Apptainer Bind Paths and Mounts dtcenter/metplus DockerHub tags METplus 6.0 Existing Builds

georgemccabe commented 2 months ago

I was able to create a Docker "artifact" that can be pulled and mounted using apptainer. This is what I had to do on casper to create it:

module load apptainer

To create volume (directory obs existed in current directory that contained observation data):

mksquashfs obs/ input_obs.squashfs
apptainer sif new input_obs.sif
apptainer sif add --datatype 4 --partarch 2 --partfs 1 --parttype 3 input_obs.sif input_obs.squashfs

To push volume sif (artifact) to DockerHub:

apptainer remote login --username georgemccabe oras://registry-1.docker.io
# enter authentication token
apptainer push -U input_obs.sif oras://registry-1.docker.io/ncar/iwrf:data-matthew-input-obs

Then to run I had to pull the artifact using oras, then bind it to container:

cd ${SCRATCH}
mkdir metplus_out

apptainer pull docker://dtcenter/metplus:feature_1514_madis2nc-pull_request

apptainer pull data-matthew-input-obs.sif oras://registry-1.docker.io/ncar/iwrf:data-matthew-input-obs

LOCAL_METPLUS_CONFIG_DIR=/glade/u/home/mccabe/i-wrf/use_cases/Hurricane_Matthew/METplus
LOCAL_FCST_INPUT_DIR=/glade/derecho/scratch/jaredlee/nsf_i-wrf/matthew
LOCAL_OUTPUT_DIR=${SCRATCH}/metplus_out

export APPTAINER_BIND="${LOCAL_METPLUS_CONFIG_DIR}:/config,${LOCAL_FCST_INPUT_DIR}:/data/input/wrf,${LOCAL_OUTPUT_DIR}:/data/output,data-matthew-input-obs.sif:/data/input/obs:image-src=/"

apptainer exec /glade/derecho/scratch/mccabe/metplus-dev_feature_1514_madis2nc-pull_request.sif /metplus/METplus/ush/run_metplus.py /config/PointStat_matthew.conf

Resources: