search

NCAR / container-dtc-nwp

End-to-end NWP systems in containers.
https://dtcenter.org/community-code/numerical-weather-prediction-nwp-containers
28 stars 13 forks source link

Support a METviewer customization option to compare MET output from multiple runs. #78

Closed JohnHalleyGotway closed 2 years ago

JohnHalleyGotway commented 2 years ago

(1) Set PROJ_DIR = top level dir that contains multiple cases (should already be set correctly)

(2) Create $PROJ_DIR/metviewer directory like this. mkdir -p metviewer/mysql

(3) Create modified YML file.

cp $PROJ_DIR/container-dtc-nwp/components/metviewer/docker-compose.yml \
   $PROJ_DIR/container-dtc-nwp/components/metviewer/docker-compose-cases.yml

(4) Edit/modify like this:

OLD YML

    volumes:
        - ${CASE_DIR}/metviewer/mysql:/var/lib/mysql
    volumes:
      - ${PROJ_DIR}/container-dtc-nwp/components/scripts:/scripts
      - ${CASE_DIR}/metprd:/data
      - ${CASE_DIR}/metviewer/output/xml:/opt/tomcat/webapps/metviewer_output/xml
      - ${CASE_DIR}/metviewer/output/plots:/opt/tomcat/webapps/metviewer_output/plots
      - ${CASE_DIR}/metviewer/output/data:/opt/tomcat/webapps/metviewer_output/data
      - ${CASE_DIR}/metviewer/output/scripts:/opt/tomcat/webapps/metviewer_output/scripts

New YML

    volumes:
        - ${PROJ_DIR}/metviewer/mysql:/var/lib/mysql
    volumes:
      - ${PROJ_DIR}/container-dtc-nwp/components/scripts:/scripts
      - ${PROJ_DIR}/cases:/data
      - ${PROJ_DIR}/metviewer/output/xml:/opt/tomcat/webapps/metviewer_output/xml
      - ${PROJ_DIR}/metviewer/output/plots:/opt/tomcat/webapps/metviewer_output/plots
      - ${PROJ_DIR}/metviewer/output/data:/opt/tomcat/webapps/metviewer_output/data
      - ${PROJ_DIR}/metviewer/output/scripts:/opt/tomcat/webapps/metviewer_output/scripts

Make sure multi runs are stored in a top-level "cases" directory

my_container_tutorial

(5) Startup metviewer running 'docker-compose -f docker-compose-cases.yml up -d'

(6) Load the data by running an updated version of metv_load_all.ksh script 3 times:

docker exec -it metviewer /scripts/common/metv_load_all.ksh mv_sandy  /data/sandy1
docker exec -it metviewer /scripts/common/metv_load_all.ksh mv_sandy  /data/sandy2
docker exec -it metviewer /scripts/common/metv_load_all.ksh mv_sandy3 /data/sandy3

Task: Enhance metv_load_all.ksh to support running with 1 command line arg (existing) or 2 command line args (new). metv_load_case.ksh {db name} {path in docker to case directory which contains a metprd sub-dir} Note, when running with 2 args, the metv_load_all.ksh needs to replace:

/data/{met_tool}

with

/data/sandy1/metprd/{met_tool}

Where "sandy1" is the "basename" of the 2nd command line arg and "metprd" is a constant.

michelleharrold commented 2 years ago

@JohnHalleyGotway -- after chatting yesterday, the process laid out above got us ~90% of the way to the goal, but there were some roadblocks that caused me to take a slightly different path. Would love to get your feedback.

I created a branch (feature/issue_78_metv) with my changes. Instead of having one script (metv_load_all.ksh) that can handle all of the desired behavior we want, I ended up having two: metv_load_all.ksh (original unchanged script) and metv_load_cases.ksh (new script that handles loading MET data from multiple cases; will be used in the customization section of the tutorial). I also created a new yml file that specifically handles having MET output under one top-level directory (docker-compose-cases.yml; to be used in the customization section of the tutorial). Having one load script got a little tricky depending on if you wanted to pivot between running with the original yml file or the new yml file. For a cleaner, more straightforward customization experience, it seems to make sense to have separate scripts/yml file for the set-up and loading. Thoughts"?

JohnHalleyGotway commented 2 years ago

@michelleharrold, that seems like a fine approach to me... especially if it actually works!

From a volume-mounting perspective, the metprd output directories probably really do need to live in the same top-level directory. Sure would be nice if there were a way to avoid that requirement so that students could load derecho, sandy, and snow output into the same database without having to move data around first.

There are some minor edits I'd recommend to the header of the new script and what-not, but I could just suggest those edits via the PR review process.

michelleharrold commented 2 years ago

@JohnHalleyGotway -- agreed about being able to avoid the requirement to have the data live in the same top-level directory. Kind of a bummer, but I couldn't figure out a good way around it (for now).

I also added a Docker compose YML file for AWS, so until we can figure out what is going on with AWS and METviewer, I am going to hold off on the PR until I can test everything.