search

NCAR / container-dtc-nwp

End-to-end NWP systems in containers.
https://dtcenter.org/community-code/numerical-weather-prediction-nwp-containers
28 stars 13 forks source link

Prebuilt images #14

Closed jwolff-ncar closed 4 years ago

jwolff-ncar commented 4 years ago

Update prebuilt images (hasn't been done in awhile!).

@jwolff-ncar Will do this on my laptop and then copy the saved images over to mohawk:/d2/www/dtcenter/container_nwp_tutorial/tar_files/prebuilt_images

We really want to use the save images on DockerHub: docker pull dtcenter/gsi docker pull dtcenter/wps_wrf docker pull dtcenter/upp docker pull dtcenter/met

@mkavulich Can you add NCL to this list? And does it also make sense to include the data containers? @JohnHalleyGotway Is that the right MET we should be pointing to (confirming it shouldn't be met-dev)? @jwolff-ncar and @michelleharrold Michelle and I discussed this the other day. There are currently 3 Dockerfiles for MET floating around: MET repo itself, container-dtc-met, and container-dtc-nwp. After the met-9.0 release, I think we should delete the container-dtc-met repo entirely. And then repoint the DockerHub reference to the MET repo itself. Hopefully that’ll simplify things.

We need METviewer too.

jwolff-ncar commented 4 years ago

@JohnHalleyGotway I am getting an error message when I try to compile MET.

Here is my error message: Downloading met-8.1.2 from https://dtcenter.org/sites/default/files/met-8.1.2.20191031.tar.gz % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 18716 0 18716 0 0 80423 0 --:--:-- --:--:-- --:--:-- 80672

gzip: stdin: not in gzip format tar: Child returned status 1 tar: Error is not recoverable: exiting now The command '/bin/sh -c umask 0002 && echo "Downloading met-${MET_VERSION} from ${MET_URL}" && mkdir -p /comsoftware/met && curl -SL ${MET_URL} | tar zxC /comsoftware/met && cd /comsoftware/met/met-${MET_VERSION} && LOG_FILE=/comsoftware/met/met-${MET_VERSION}/configure.log && echo "Configuring met-${MET_VERSION} and writing log file ${LOG_FILE}" && ./configure --prefix=/comsoftware/met/ --enable-grib2 --enable-mode_graphics --enable-modis --enable-lidar2nc --enable-python MET_NETCDF=/comsoftware/libs/netcdf MET_HDF=/comsoftware/libs/HDF4.2r3 MET_HDFEOS=/comsoftware/libs/hdfeos MET_FREETYPEINC=/usr/include/freetype2 MET_FREETYPELIB=/usr/lib MET_CAIROINC=/usr/include/cairo MET_CAIROLIB=/usr/lib MET_PYTHON_CC='-I/usr/include/python2.7' MET_PYTHON_LD='-lpython2.7' > ${LOG_FILE} && LOG_FILE=/comsoftware/met/met-${MET_VERSION}/make_install.log && echo "Compiling met-${MET_VERSION} and writing log file ${LOG_FILE}" && make install > ${LOG_FILE} && LOG_FILE=/comsoftware/met/met-${MET_VERSION}/make_test.log && echo "Testing met-${MET_VERSION} and writing log file ${LOG_FILE}" && make test > ${LOG_FILE} 2>&1' returned a non-zero code: 2

I was having space and memory issues earlier so it could just be my machine acting up. However, can you try the following?

git clone https://github.com/NCAR/container-dtc-nwp ${PROJ_DIR}/container-dtc-nwp cd container-dtc-nwp cd ${PROJ_DIR}/container-dtc-nwp/components cd met/MET ; docker build -t dtc-met . ; cd ../..

jwolff-ncar commented 4 years ago

@JohnHalleyGotway Oops, obviously a bad path to the tar file. Just need to find the right one. Looking now but if you can point me to it that would be great. Thanks!

jwolff-ncar commented 4 years ago

@JohnHalleyGotway I updated the Dockerfile path to the MET 8.1.2 tarball and now it compiles.

jwolff-ncar commented 4 years ago

I ran into an issue with METviewer. Suggestions? @TatianaBurek

git clone https://github.com/NCAR/container-dtc-nwp ${PROJ_DIR}/container-dtc-nwp cd container-dtc-nwp cd ${PROJ_DIR}/container-dtc-nwp/components cd metviewer/METviewer ; docker build -t dtc-metviewer . ; cd ../..

Error: Problem: package R-3.6.1-1.el8.x86_64 requires R-devel = 3.6.1-1.el8, but none of the providers can be installed

jwolff-ncar commented 4 years ago

I added NCL to DockerHub: From: /Volumes/d1/jwolff/projects/AWS/sc_test_20191219/container-dtc-nwp/components/ncl

docker build -t dtcenter/ncl .

docker push dtcenter/ncl

Note: Should connect this to the repo so if the Dockerfile changes it is updated automatically

docker pull dtcenter/ncl

jwolff-ncar commented 4 years ago

To get METviewer to build, the DTC base image needed to be modified to use centos:7 (rather than latest). Once this was done, it built fine.

jwolff-ncar commented 4 years ago

Just as a note, if updating the WRF_VERSION it needs to be done in multiple places: components/wps_wrf/Dockerfile components/scripts/case/set_env.ksh If this is not done correctly it will crash at runtime.

jwolff-ncar commented 4 years ago

The latest prebuilt images have been uploaded to Drupal.

This is done.