search

NCAR / container-dtc-nwp

End-to-end NWP systems in containers.
https://dtcenter.org/community-code/numerical-weather-prediction-nwp-containers
28 stars 13 forks source link

change the value of url.output #93 #95

Closed TatianaBurek closed 1 year ago

TatianaBurek commented 1 year ago

Pull Request Testing

Pull Request Checklist

fossell commented 1 year ago

@TatianaBurek @michelleharrold - I tried to test this but didn't have success. I'm guessing I'm missing a step somewhere, I'm just not familiar enough with this component. I mostly followed the instructions we currently have for metviewer with singularity when just doing batch scripts. So maybe procedure have changed or there's something new? I didn't notice any glaring errors when I did this, but there was a lot of output to the screen and I was doing something else while waiting for builds, so who knows. Here is what I did:

cd /home/ec2-user

# Moved the original repo provided with the AMI, clone new, checkout feature branch
mv container-dtc-nwp container-dtc-nwp_save
git clone https://github.com/NCAR/container-dtc-nwp
git checkout -b feature_93-aws_port origin/feature_93-aws_port

# Build the new docker image for singularity
cd ${PROJ_DIR}/container-dtc-nwp/components/metviewer/METviewer
docker build -t nwp-container-metviewer_singularity -f Dockerfile_for_Singularity .
docker images

# Build the singularity sandbox from teh new docker image. Note use of "docker-daemon://" is to use the local build of docker image.
singularity build --sandbox --fix-perms --force metv4singularity docker-daemon://nwp-container-metviewer_singularity:latest

# Start the singularity instance
singularity instance start --writable metv4singularity metv
singularity instance list

# Initialize the singularity instance and make writable
singularity exec --writable instance://metv bash init_singularity.sh
[enter]

# Load the database
cd ${PROJ_DIR}/container-dtc-nwp/components/scripts/common
singularity shell instance://metv
./metv_load_singularity.ksh mv_sandy ${CASE_DIR}

Then tried to open the GUI using my instance IPV4, e.g.: http://[IPv4]/metviewer/metviewer1.jsp

But it just sat there spinning. I also then tried to just run the batch plots options, but that didn't work either. (i.e. while in the singularity shell:

cd ${PROJ_DIR}/container-dtc-nwp/components/scripts/sandy_20121027/metviewer
./metv_plot_singularity.ksh ${CASE_DIR}

But no plots were generated in ${CASE_DIR}/metviewer/plots. So maybe there was some issue earlier on. Am I missing something or can you provide a list of steps to successfully launch the GUI so I can see where I went wrong?

TatianaBurek commented 1 year ago

@fossell Was docker-daemon://nwp-container-metviewer_singularity:latest updated to METviewer 4.1.0? If not, you need to build the singularity sandbox from a new docker image by saving it into .tar and then use .tar in singularity build --sandbox command

fossell commented 1 year ago

@TatianaBurek - I built the "docker-daemon://nwp-container-metviewer_singularity:latest" using the Dockerfile_for_Singularity file in container-dtc-nwp/components/metviewer/METviewer from your feature branch. It says "ENV METVIEWER_GIT_NAME v4.1.0" in that file, so I think so? But maybe there was an error building that image. To build it, I checked out your feature branch and I did: 

cd ${PROJ_DIR}/container-dtc-nwp/components/metviewer/METviewer
docker build -t nwp-container-metviewer_singularity -f Dockerfile_for_Singularity .

I'll try again and monitor it, maybe there was an error.

michelleharrold commented 1 year ago

@TatianaBurek and @fossell - I wanted to report I also tested this, and I had the same issue as @fossell regarding accessing the GUI via http://[IPv4]/metviewer/metviewer1.jsp -- I went through and tried all the steps again but was able to replicate the behavior.

Other than selected our specified security group, I did not do anything special with setting up my instance. I also double checked I was using METviewer v4.1.0.

TatianaBurek commented 1 year ago

To access the GUI use this URL: http://[IPv4]**:8080**/metviewer/ for example: http://35.175.144.92:8080/metviewer/

michelleharrold commented 1 year ago

@TatianaBurek and @fossell - using the URL Tatiana listed, I was able to access the GUI!

I did try this: cd ${PROJ_DIR}/container-dtc-nwp/components/scripts/sandy_20121027/metviewer ./metv_plot_singularity.ksh ${CASE_DIR}

But nothing happened -- upon further investigation, the /METviewer/bin/mv_batch.sh script is empty and has zero size (the other scripts under /bin seem fine), so I think that is why the batch plots are not being generated.

TatianaBurek commented 1 year ago

I also noticed that mv_batch.sh is empty and working on fixing this

fossell commented 1 year ago

@TatianaBurek, @michelleharrold - Just wanted to update that I'm now able to view the GUI and make plots using http://[IPv4]:8080/metviewer/ and with the recent mods also able to generate batch plots using ./metv_plot_singularity.ksh with Singularity.

I'll proceed with comprehensive end-to-end testing for both Docker and Singularity to ensure nothing else broke in the process and that we are ready for this to be merged for release.

fossell commented 1 year ago

@TatianaBurek, @michelleharrold - In testing Docker on AWS for Sandy, I'm not able to pull up the GUI now. I built all docker images from scratch using this feature branch. METviewer docker image built successfully. All components ran without issue. For METviewer step I followed online tutorial:

cd ${PROJ_DIR}/container-dtc-nwp/components/metviewer
docker-compose -f docker-compose-AWS.yml up -d

[+] Running 34/34 ⠿ metviewer Pulled 23.4s ⠿ 2d473b07cdd5 Already exists 0.0s ⠿ 2a71f7a3b30d Already exists 0.0s ⠿ 0bff2878611b Already exists 0.0s ⠿ 3054cff78acf Already exists 0.0s ⠿ 85caa936b897 Already exists 0.0s ⠿ e023c53ff84d Already exists 0.0s ⠿ e3ef979ddfd1 Pull complete 0.1s ⠿ ebc93fd0f4b5 Pull complete 0.2s ⠿ d9ca6e647d39 Pull complete 0.2s ⠿ cd49dfe2750a Pull complete 4.3s ⠿ d42e55f00ba5 Pull complete 11.8s ⠿ f7f06c188805 Pull complete 13.0s ⠿ 0158a69d9058 Pull complete 13.0s ⠿ 733a1b5e611b Pull complete 13.6s ⠿ cea76ee1bd6f Pull complete 13.8s ⠿ b72e671e8170 Pull complete 14.2s ⠿ 016c7058ba3f Pull complete 15.1s ⠿ b7c27ebfc108 Pull complete 16.5s ⠿ d33ee1a1755c Pull complete 22.8s ⠿ 863b3822d37c Pull complete 23.0s ⠿ e50bd2a24157 Pull complete 23.1s ⠿ db Pulled 9.5s ⠿ 66fb34780033 Pull complete 6.5s ⠿ ef4ccd63cdb4 Pull complete 6.6s ⠿ d6f28a94c51f Pull complete 6.6s ⠿ 7feea2a503b5 Pull complete 6.8s ⠿ 71dd5852ecd9 Pull complete 6.8s ⠿ 2ff5c3b24fd5 Pull complete 6.9s ⠿ 88a546386a61 Pull complete 7.3s ⠿ 65b18297cf83 Pull complete 7.4s ⠿ d64f23335fb8 Pull complete 9.1s ⠿ 1812e3377415 Pull complete 9.2s ⠿ 69105ed9bd35 Pull complete 9.2s [+] Running 3/3 ⠿ Network metviewer_default Created 0.0s ⠿ Container mysql_mv Started 21.8s ⠿ Container metviewer Started 

docker ps -a

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES dbfd20d79a3c dtcenter/nwp-container-metviewer:latest "/bin/sh -c '${CATAL…" 18 minutes ago Up 18 minutes 0.0.0.0:80->8080/tcp, :::80->8080/tcp metviewer eb44b13d5b17 mysql:5.7 "docker-entrypoint.s…" 18 minutes ago Up 18 minutes 33060/tcp, 0.0.0.0:6603->3306/tcp, :::6603->3306/tcp mysql_mv

docker exec -it metviewer /scripts/common/metv_load_all.ksh mv_sandy

ERROR 1008 (HY000) at line 1: Can't drop database 'mv_sandy'; database doesn't exist ---- MVLoad ----

Version: 4.0.2

Begin time: 2022-08-24 18:59:42 Parsing: /data/load_mv_sandy.xml

Permutation 1 of 4 - /data/grid_stat /data/grid_stat/grid_stat_ARW_F03_03h_030000L_20121027_210000V.stat /data/grid_stat/grid_stat_ARW_F06_03h_060000L_20121028_000000V.stat Permutation 1 of 4 complete - insert time: 0:00:00.051 stat lines: 40 mode lines: 0 mtd lines: 0

Permutation 2 of 4 - /data/point_stat /data/point_stat/point_stat_ARW_F00_ADPSFC_000000L_20121027_180000V.stat /data/point_stat/point_stat_ARW_F00_ADPSFC_MPR_000000L_20121027_180000V.stat /data/point_stat/point_stat_ARW_F00_WINDS_000000L_20121027_180000V.stat /data/point_stat/point_stat_ARW_F03_ADPSFC_030000L_20121027_210000V.stat /data/point_stat/point_stat_ARW_F03_ADPSFC_MPR_030000L_20121027_210000V.stat /data/point_stat/point_stat_ARW_F03_WINDS_030000L_20121027_210000V.stat /data/point_stat/point_stat_ARW_F06_ADPUPA_060000L_20121028_000000V.stat /data/point_stat/point_stat_ARW_F06_ADPSFC_060000L_20121028_000000V.stat /data/point_stat/point_stat_ARW_F06_ADPSFC_MPR_060000L_20121028_000000V.stat /data/point_stat/point_stat_ARW_F06_WINDS_060000L_20121028_000000V.stat Permutation 2 of 4 complete - insert time: 0:00:00.158 stat lines: 577 mode lines: 0 mtd lines: 0

Permutation 3 of 4 - /data/ensemble_stat Permutation 4 of 4 - /data/mode_stat ==== grid_stat ====

  stat_header table time total: 0:00:00.000
           stat header records: 253
           stat header inserts: 81
             line data records: 253
             line data inserts: 278
            stat group records: 0
            stat group inserts: 0
            var length records: 0
            var length inserts: 0
                   total lines: 617
                   insert size: 1
                  lines / msec: 2.924
                     num files: 12

==== mode ====

  mtd_header search time total: 0:00:00.000
 stat_header search time total: 0:00:00.027
           mode_header inserts: 0
              mode_cts inserts: 0
       mode_obj_single inserts: 0
         mode_obj_pair inserts: 0
==== mtd ====

            mtd_header inserts: 0
     mtd_3d_obj_single inserts: 0
       mtd_3d_obj_pair inserts: 0
            mtd_2d_obj inserts: 0
                   total lines: 0
                     num files: 0
                     num files: 0

==== indexes ====creating

Indexes creating 0:00:00.266

End time: 2022-08-24 18:59:42 Load total: 0:00:00.211

---- MVLoad Done ----

docker exec -it metviewer /bin/bash
/scripts/common/reset_metv_url.ksh
exit

% Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 14 100 14 0 0 306 0 --:--:-- --:--:-- --:--:-- 311 Resetting METviewer URL to http://54.174.214.208/metviewer/metviewer1.jsp Restarting METviewer web service Using CATALINA_BASE: /opt/tomcat Using CATALINA_HOME: /opt/tomcat Using CATALINA_TMPDIR: /opt/tomcat/temp Using JRE_HOME: /usr Using CLASSPATH: /opt/tomcat/bin/bootstrap.jar:/opt/tomcat/bin/tomcat-juli.jar Using CATALINA_OPTS:
Using CATALINA_BASE: /opt/tomcat Using CATALINA_HOME: /opt/tomcat Using CATALINA_TMPDIR: /opt/tomcat/temp Using JRE_HOME: /usr Using CLASSPATH: /opt/tomcat/bin/bootstrap.jar:/opt/tomcat/bin/tomcat-juli.jar Using CATALINA_OPTS:
Tomcat started. [root@dbfd20d79a3c /]# exit

Then tried to open the GUI with http://IPV4_public_IP/metviewer/metviewer1.jsp as well as http://IPV4_public_IP:8080/metviewer/metviewer1.jsp. But no luck. This isn't Singularity, but is there an update for the regular Docker version of running METviewer and viewing it on the GUI on AWS that I'm missing?

TatianaBurek commented 1 year ago

I found a few issues in the testings steps:

  1. docker-compose-AWS.yml pulls dtcenter/nwp-container-metviewer:latest image from the repo but in our case we are testing the image that was created from the branch and not the latest. So, before running docker-compose command you should edit docker-compose-AWS.yml and change dtcenter/nwp-container-metviewer:latest to dtcenter/container-dtc-nwp-dev:metviewer_feature_93-aws_port. OR change the image version, commit it into the branch and clone the updated branch. To verify the image, look at the mv_load log messages. It should be: INFO:root:METdbload Version: 1.0.0 and not Version: 4.0.2
  2. reset_metv_url.ksh script resets the URL by fixing the IP address but it did not add the port number ":8080' to it.
  3. docker-compose-AWS.yml has incorrect port mapping: "80:8080". We don't use Apache so it should be "8080:8080"
  4. I can't connect to the database. ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2) Maybe ports 6603 and/or 3306 should be opened?
fossell commented 1 year ago

@TatianaBurek - With the recent mods regarding the 8080 port, is this ready to test again? Or was there still an issue with the ERROR 2002 (HY000): mySQL error? Are you waiting on us to find a solution about 3306 ports or have your contacted Deidre? Just let us know where this stands and if you need any action on our part.

TatianaBurek commented 1 year ago

@fossell I updated files to address port 8080 issue . This change is ready for testing. The issue with the database port(s) 6603 and 3306 is still present. Getting Deidre's expertise would be advantages in this case. Should I contact her or will you?

fossell commented 1 year ago

@TatianaBurek - Thanks for the update. If you're willing to contact Deidre that might be best, I don't know the lingo how to describe what we need in terms that she needs!

TatianaBurek commented 1 year ago

After a few changes in Docker file the container is working on AWS. IMPORTANT: if you use feature_93-aws_port you need to change the image name to dtcenter/container-dtc-nwp-dev:metviewer_feature_93-aws_port

michelleharrold commented 1 year ago

@TatianaBurek and @fossell - I tested dtcenter/container-dtc-nwp-dev:metviewer_feature_93-aws_port and everything worked great -- as a note, I did use docker-compose.yml, and I had to change image, accordingly, to use dtcenter/container-dtc-nwp-dev:metviewer_feature_93-aws_port. Since AWS and non-AWS are now both using port 8080:8080, we no longer have a need for an AWS-specific docker-compose file.

fossell commented 1 year ago

@TatianaBurek, @michelleharrold - I also had success running using the docker-compose.yml file. So I think we can remove the docker-compose-AWS.yml file. We can do that in a subsequent clean up task. I also re-tested metviewer using singularity and the GUI was successfully launched again. So as for this PR, I think we are good to merge. I'll wait for @michelleharrold to approve just in case I'm fogetting anything, then I'll merge.

fossell commented 1 year ago

I might change my mind actually, may it's better to remove that file now, it has some testing environmental variables in it, like MYSQL_DIR and METVIEWER_DIR. And @michelleharrold pointed out we don't need the docker-compose-cases.yml now either.
@TatianaBurek - Could you remove the files docker-compose-AWS.yml and docker-compose-cases.yml from this branch? Then I will merge.

TatianaBurek commented 1 year ago

@fossell I updated files to address port 8080 issue . This change is ready for testing. The issue with the database port(s) 6603 and 3306 is still present. Getting Deidre's expertise would be advantages in this case. Should I contact her or will you?

TatianaBurek commented 1 year ago

docker-compose files for AWS are deleted.