Hyrax docker

Current Version: Hyrax-1.17.0-1

Hyrax-1.17.0 is composed of:

OLFS version 1.18.14-1
BES version 3.21.0-46
libdap4 version 3.21.0-27

Contents

• Overview
• Quick Start
• Serving Your Data
• Server Logs
• Docker Images
  • hyrax
  • besd
  • olfs
  • ncwms
• Docker-Compose
  • hyrax
  • hyrax-wms
  • developer
• Ansible
• License
• Acknowledgements

Overview

The hyrax-docker project can build the following Docker images:

• hyrax- A complete Hyrax server in a single container. It may be built with or without a bundled ncWMS.
• besd - The BackEndServer (BES) component of Hyrax.
• olfs - The OLFS component of Hyrax, a Java Web Application deployed in Apache Tomcat.
• ncwms - The ncWMS application from our friends at the Reading e-Science Centre.
• ngap - A specialization of the Hyrax server for deployment in the NASA cloud. This variant is dependent on the injection of numerous configuration information at launch an is unlikely to be of use to people whoa are not directly involved with the NGAP project.

Each of these images can be run standalone. The besd, olfs, and the ncwms images may be combined via docker compose or ansible.

Note: We build and upload our official release containers to Docker Hub. We also produce and publish to Docker Hub snapshot containers for every CI build of hyrax, olfs, besd, and ngap images. While these CI build images have passed all of our various tests they are not official releases. Caveat Emptor.

The default configuration has the Hyrax service start up providing access to the default (test) data, but can easily be configured to serve data from the host machine (see Using Hyrax docker to serve your data).

NOTE: This code is based on work started by Gareth Williams at CSIRO and contributed to OPeNDAP. We are grateful for their support.

Quick Start

• Hyrax Data Server
• Hyrax Data Server + ncWMS2
• besd
• olfs

Build

Build and run a dockerized Hyrax, serving your own data.

Each build is based on a build recipe that specifies the binary components to be used in the build.

For our CI builds those components reside exclusively in a private S3 bucket owned by OPeNDAP. In order to build using those recipes, one needs to have credentials to access the S3 bucket. Once you have configured your AWS credentials (either using the AS configuratrion or envirnment variables) thebuild_from_ci_recipe.sh shell script will perform the retrieval and docker build using the recipe supplied, or the most recent recipe in the snapshot.time file.

Our official release builds can be (re)built by using one of the recipes located in the releases directory. The build_from_release_recipe.sh shell script located in this directory will build the latest release, or it can be told which build recipe file to utilize as the only command line parameter to the shell script.

For detailed information about each of the four Docker container images, see the section Images below.

To build the single container hyrax, clone this project

git clone https://github.com/opendap/hyrax-docker

change directory to the top of the project:

cd hyrax-docker

and then use build_from_release_recipe.sh

./build_from_release_recipe.sh

To build the docker image using the latest release binaries located at www.opendap.org/pub

To build the release from a previous version, for example 1.16.8:

./build_from_release_recipe.sh releases/hyrax-1.16.8-build-recipe

Running The Server

To run the container:

docker run -d -h hyrax -p 8080:8080 --name=hyrax_container hyrax_image

To run the container with ncWMS you'll need to tell the server where the ncWMS service is located. This can be done by utilizing the container's -n paramter to specify the endpoint like this:

docker run -d -h hyrax -p 8080:8080 --name=hyrax_container hyrax_image -n http://localhost:8080

TIP: The value of -n should be the outward facing domain name / IP address of your Docker container. If you are running a container on your local system, then the example value of http://localhost:8080 should work well. If your Hyrax container is running elsewhere (in AWS for example) you'll have to sort out what the value should be. If the -n parameter is omitted from the docker run command then the value used will be the value of --build-arg NCWMS_BASE from the docker build command. If no NCWMS_BASE was specified in the docker build command then the value defaults to https://localhost:8080 (note that this is an HTTPS transport URL)

To configure the hyrax_container so the server is accessible using a port other than 8080, such as port 80, the default port for HTTP. This can also be used to run several servers, each accessed using a different port (e.g., you can have a development server running on port 8080 and then use -p 9090:8080 to bind the Hyrax in the docker container to the localhost:9090).

To run the Dockerized Hyrax on port 80, use (note that docker is started as the super user because only root can bind ports less than 1024).

sudo docker run -d -h hyrax -p 80:8080 --name=hyrax_container hyrax_image

To stop the container

docker stop hyrax_container
docker rm hyrax_container

where the argument to docker stop is the value passed in for the --name parameter with docker run. The command docker rm removes the container and is needed only if you want to (re)start the container with different values for any of the docker run parameters.

Using Hyrax docker to serve your data

docker run --hostname hyrax --port 8080:8080 --volume /home/mydata:/usr/share/hyrax --name=hyrax_container hyrax_image

Server Logs & Serving Your Data

cd hyrax-docker/hyrax-latest
docker build -t hyrax --no-cache hyrax
prefix=`pwd`
docker run \
   --name hyrax \
   --publish 8080:8080 \
   --volume $prefix/logs:/var/log/tomcat \
   --volume $prefix/logs:/var/lib/tomcat/webapps/opendap/WEB-INF/conf/logs \
   --volume $prefix/logs:/var/log/bes \
   hyrax \
   -e support@erehwon.edu \
   -s \
   -n http://localhost:8080

cd hyrax-docker/hyrax-latest
docker build -t hyrax --no-cache hyrax
prefix=`pwd`
docker run \
   --name hyrax \
   --publish 8080:8080 \
   --volume $prefix/local_data:/usr/share/hyrax \
   hyrax \
   -e support@erehwon.edu \
   -s \
   -n http://localhost:8080

cd hyrax-docker/hyrax-latest
docker build -t hyrax_ncwms --build-arg USE_NCWMS=true --no-cache hyrax
prefix=`pwd`
docker run \
   --name hyrax \
   --publish 8080:8080 \
   --volume $prefix/local_data:/usr/share/hyrax \
   --volume $prefix/logs:/var/log/tomcat \
   --volume $prefix/logs:/var/lib/tomcat/webapps/opendap/WEB-INF/conf/logs \
   --volume $prefix/logs:/var/log/bes \
   --volume $prefix/logs:/root/.ncWMS2/logs \
   hyrax_ncwms \
   -e support@erehwon.edu \
   -s \
   -n http://localhost:8080

    volumes:
     - ./logs/olfs_tomcat:/usr/local/tomcat/logs
     - ./logs:/usr/local/tomcat/webapps/opendap/WEB-INF/conf/logs

Images

one_container_times:  n=100, min=  129.96,  mean=  131.41 +/-  0.54,  max=  133.07

two_container_times   n=100, min=   82.90,  mean=  126.42 +/- 13.23,  max=  133.03

hyrax

• SERVER_HELP_EMAIL (-e) - The email address of the support person for the service. This will be returned in error and help pages.

• FOLLOW_SYMLINKS (-s) - Instructs the server to follow symbolic links in the file system.

• NCWMS_BASE (-n) - The system needs to know the publicly accessible service base for the ncWMS (something like http://yourhost:8080). If all you want is to test it on your local system then the value of http://localhost:8080 will suffice.

• JAVA_OPTS - If JAVA_OPTS is defined in the container runtime environment, tomcat/olfs/ncWMS will include those options in the service start up. There are many options that could be passed. Of particular note is –Xmx which sets the amount of memory available. ncWMS will not work properly with low memory limits. JAVA_OPTS can be set by normal methods: on the docker run command line, or in docker-compose configuration or in your own container layer if you build on the provided containers. (This from Gareth 11 Sept 2017.)

Command Line Examples:

Command Line Options Example

Launch Hyrax using command line switches to set the admin email to (-e support@erehwon.edu), enable symbolic link traversal (-s), and set the ncWMS service base to (-n http://foo.bar.com:8080)

docker run                      \
    --name hyrax                \
    --publish 8080:8080         \
    hyrax_image                 \
    -e support@erehwon.edu      \
    -s                          \
    -n http://foo.bar.com:8080

Environment Variables Example

Launch Hyrax using command line defined environment variables to set the admin email to (-e SERVER_HELP_EMAIL=support@foo.com), enable symbolic link traversal (-s), and set the ncWMS service base to (-e NCWMS_BASE=http://foo.bar.com)

docker run \
    --name hyrax \
    --publish 8080:8080 \
    --env FOLLOW_SYMLINKS=true \
    --env SERVER_HELP_EMAIL=support@foo.com \
    --env NCWMS_BASE=http://foo.bar.com \
    hyrax_image

NOTE: The environment variables are set to the left of the image name. The command line switches occur AFTER the image name.

The Whole Enchilada

In this example we use the command line parameters to condition the server. We specify a read-only volume for data, 3 read-write volumes for collecting logs on the local disk, and finally mount our local BES configuration onto the Docker BES instance configuration.

docker run \
    --name hyrax \
    --publish 8080:8080 \
    --volume  /usr/share/data:/usr/share/hyrax:ro  \
    --volume /tmp/logs/tomcat:/var/log/tomcat \
    --volume /tmp/logs:/var/lib/tomcat/webapps/opendap/WEB-INF/conf/logs \
    --volume /tmp/logs:/var/log/bes \
    --volume /etc/bes:/etc/bes \
    hyrax \
    -e support@erehwon.edu \
    -s \
    -n http://localhost:8080

And again but this time using command line set enironment variables. Same result as just above.

docker run \
    --name hyrax \
    --publish 8080:8080 \
    --volume  /usr/share/data:/usr/share/hyrax:ro  \
    --volume /tmp/logs/tomcat:/var/log/tomcat \
    --volume /tmp/logs:/var/lib/tomcat/webapps/opendap/WEB-INF/conf/logs \
    --volume /tmp/logs:/var/log/bes \
    --volume /etc/bes:/etc/bes \
    --env FOLLOW_SYMLINKS=true \
    --env SERVER_HELP_EMAIL=support@foo.com \
    --env NCWMS_BASE=http://foo.bar.com \
    hyrax_image

Advanced Examples

In the event that greater control of the Hyrax configuration is desired, or additional disk space is required for the various BES caching activities one may utilize volume mounts to address these issues.

Map BES cache to host filesystem

docker run \
    --name hyrax \
    --publish 8080:8080 \
    --volume /tmp/bes_cache:/tmp  \
    --volume /usr/share/data:/usr/share/hyrax:ro  \
    --volume /tmp/logs/tomcat:/var/log/tomcat \
    --volume /tmp/logs:/var/lib/tomcat/webapps/opendap/WEB-INF/conf/logs \
    --volume /tmp/logs:/var/log/bes \
    --env FOLLOW_SYMLINKS=true \
    --env SERVER_HELP_EMAIL=support@foo.com \
    --env NCWMS_BASE=http://foo.bar.com \
    hyrax_image

Annontation:

• --volume /tmp/bes_cache:/tmp: Maps the docker container's /tmp dir to the docker host directory /tmp/bes_cache
• --volume /usr/share/data:/usr/share/hyrax:ro: Maps the docker container's Hyrax data directory to the docker host directory /usr/share/data
• --volume /tmp/logs/tomcat:/var/log/tomcat: Maps the docker container's Tomcat logs directory to the docker host directory /tmp/logs/tomcat
• --volume /tmp/logs:/var/lib/tomcat/webapps/opendap/WEB-INF/conf/logs: Maps the docker container's OLFS logs to the docker host directory /tmp/logs
• --volume /tmp/logs:/var/log/bes: Maps the docker container's BES log files to the docker host directory /tmp/logs

Replace default BES configuration

docker run \
    --name hyrax \
    --publish 8080:8080 \
    --volume /home/roger/bes:/etc/bes  \
    --volume /usr/share/data:/usr/share/hyrax:ro  \
    --volume /tmp/logs/tomcat:/var/log/tomcat \
    --volume /tmp/logs:/var/lib/tomcat/webapps/opendap/WEB-INF/conf/logs \
    --volume /tmp/logs:/var/log/bes \
    --env FOLLOW_SYMLINKS=true \
    --env SERVER_HELP_EMAIL=support@foo.com \
    --env NCWMS_BASE=http://foo.bar.com \
    hyrax_image

Annontation:

• --volume /home/roger/bes:/etc/bes: Replaces the docker container's BES configuration with one held in the docker host file system directory /home/roger/bes

Replace default BES & OLFS configuration

docker run \
    --name hyrax \
    --publish 8080:8080 \
    --volume /home/roger/bes:/etc/bes  \
    --volume /home/roger/olfs:/var/lib/tomcat/webapps/opendap/WEB-INF/conf \
    --volume /usr/share/data:/usr/share/hyrax:ro  \
    --volume /tmp/logs/tomcat:/var/log/tomcat \
    --volume /tmp/logs:/var/log/bes \
    --env FOLLOW_SYMLINKS=true \
    --env SERVER_HELP_EMAIL=support@foo.com \
    --env NCWMS_BASE=http://foo.bar.com \
    hyrax_image

Annontation:

• --volume /home/roger/bes:/etc/bes: Replaces the docker container's BES configuration with one held in the docker host file system directory /home/roger/bes
• --volume /home/roger/olfs:/var/lib/tomcat/webapps/opendap/WEB-INF/conf: Replaces the docker container's OLFS configuration with one held in the docker host file system directory /home/roger/olfs

besd

docker run --name besd -p 10022:10022 besd_image -e support@erehwon.edu -s

docker run --name besd -p 10022:10022 -e FOLLOW_SYMLINKS=true -e SERVER_HELP_EMAIL=support@foo.com besd_image

olfs

• NCWMS_BASE (-n) - The system needs to know the publicly accessible service base for ncWMS (something like http://yourhost:8080). If all you want is to test it on your local system then the default value of http://localhost:8080 will suffice.

Command Line Examples:

Launch the olfs using command line switches to set the ncWMS service base to (-n http://foo.bar.com:8080)

docker run --name olfs -p 8080:8080 olfs_image -n http://foo.bar.com:8080

Launch the olfs using command line defined environment variables to set the ncWMS service base to (-e NCWMS_BASE=http://foo.bar.com)

docker run --name besd -p 8080:8080 -e NCWMS_BASE=http://foo.bar.com olfs_image

ncwms

docker run --name ncwms -p 8080:8080 ncwms_image

Docker-Compose

hyrax.yml

hyrax_wms.yml

developer.yml

Ansible

License

Authors: Nathan David Potter ndp@opendap.org, Dan Holloway dholloway@opendap.org, James Gallagher jgallagher@opendap.org

This library is free software; you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation; either version 3.0 of the License, or (at your option) any later version.

This library is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more details.

You should have received a copy of the GNU Lesser General Public License along with this library; if not, write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA

You can contact OPeNDAP, Inc. at PO Box 112, Saunderstown, RI. 02874-0112.

Acknowledgements

Based on https://bitbucket.csiro.au/projects/ASC/repos/hyrax-docker/, Dec 19, 2016, by gareth.williams@csiro.au. That project was licensed under a CSIRO variation of a MIT / BSD Open Source License. The license text is in the file CSIRO_MIT_LICENSE

NOTE: Gareth wrote: Ideas have been drawn from https://github.com/Unidata/thredds-docker and various other contributions on dockerhub, including the official postgres container's exemplar use of variables with an entrypoint.