upgrading from 2.18 to 2.23 not deploying using docker

[MathewBiddle]

I was following the instructions at https://ioos.github.io/erddap-gold-standard/update-erddap.html to upgrade an ERDDAP from 2.18 to 2.23 (https://erddap.ioos.us/erddap/index.html). It seems that version 2.19 has one major TODO item:

TO DO: ERDDAP now requires Java 17 and the related Tomcat 10. You must follow the ERDDAP installation instructions (or the equivalent e.g., for Docker) to install Java 17 and Tomcat 10 and copy your [tomcat]/content directory from your Tomcat 8 installation into the new [tomcat] directory.

I assumed (probably bad on my part) this was managed in the docker container and all I needed to do was update the container to the most recent one and things would work just fine. However, that's not the case.

I know the most recent commit moved the env vars to docker-compose. Is this a new requirement for deploying >v2.23 with Docker?

I'm just trying to tease out if I need to move all my env variable declarations to docker-compose or if I need to do something else to satisfy the Java and Tomcat update...

[MathewBiddle]

What I've done:

1. Stopped the ERDDAP server: docker-compose down
2. Edited docker-compose.yml to point to the latest container https://github.com/ioos/erddap-gold-standard/blob/479a1dc50e7ddc0340587cbea6abd1f39f54d2c5/docker-compose.yml#L3 ours was still pointing to 2.18 from this commit https://github.com/ioos/erddap-gold-standard/blob/6528ef1b3e94f9a6b7c4c9294c2ac976a2656b6f/docker-compose.yml#L5
3. Save edits.
4. Spin up the container docker-compose up -d

Container looks like it's doing what it should but the logs aren't getting written and the website isn't resolving and the container keeps restarting.

$ docker ps
CONTAINER ID   IMAGE                                    COMMAND                  CREATED              STATUS                          PORTS     NAMES
f572840307dd   axiom/docker-erddap:2.23-jdk17-openjdk   "/entrypoint.sh cata…"   About a minute ago   Restarting (1) 26 seconds ago             erddap_gold_standard
$ docker ps
CONTAINER ID   IMAGE                                    COMMAND                  CREATED              STATUS                          PORTS     NAMES
f572840307dd   axiom/docker-erddap:2.23-jdk17-openjdk   "/entrypoint.sh cata…"   About a minute ago   Restarting (1) 34 seconds ago             erddap_gold_standard

If I stop the container, revert back to 2.18, and start back up again, the ERDDAP spins up as expected.

I think I'm missing something with how Java changed between 2.18 and 2.23 and what might need to be adjusted within this repo.

cc @mwengren

[MathewBiddle]

Here's the diff between where I'm at and where main is: https://github.com/ioos/erddap-gold-standard/compare/6528ef1b3e94f9a6b7c4c9294c2ac976a2656b6f...main

And running GenerateDatasetsXml.sh with 2.23 works just fine... 😕

[MathewBiddle]

AHHA! I was hiding the docer-compose output with -d:

$ docker-compose up
Creating network "erddap-gold-standard_default" with the default driver
Creating erddap_gold_standard ... done
Attaching to erddap_gold_standard
erddap_gold_standard | ERDDAP running with:  -server -d64 -Xms1G -Xmx2G -XX:+HeapDumpOnOutOfMemoryError -Djava.awt.headless=true -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -DerddapContentDirectory=/usr/local/tomcat/content/erddap/ -Djna.tmpdir=/tmp/ -Djava.security.egd=file:/dev/./urandom
erddap_gold_standard | Unrecognized option: -d64
erddap_gold_standard | Error: Could not create the Java Virtual Machine.
erddap_gold_standard | Error: A fatal exception has occurred. Program will exit.
erddap_gold_standard exited with code 1

[MathewBiddle]

In erddap/conf/setenv.sh I removed -d64 from NORMAL:

NORMAL="-server -Xms1G -Xmx2G"                                                                                                                                                                                                                                                 HEAP_DUMP="-XX:+HeapDumpOnOutOfMemoryError"
HEADLESS="-Djava.awt.headless=true"
EXTRAS="-XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled"
CONTENT_ROOT="-DerddapContentDirectory=$CATALINA_HOME/content/erddap"
JNA_DIR="-Djna.tmpdir=/tmp/"
FASTBOOT="-Djava.security.egd=file:/dev/./urandom"

Now I have this error:

$ docker-compose up
Starting erddap_gold_standard ... done
Attaching to erddap_gold_standard
erddap_gold_standard | groupadd: group 'tomcat' already exists
erddap_gold_standard | ERDDAP running with:  -server -Xms1G -Xmx2G -XX:+HeapDumpOnOutOfMemoryError -Djava.awt.headless=true -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -DerddapContentDirectory=/usr/local/tomcat/content/erddap/ -Djna.tmpdir=/tmp/ -Djava.security.egd=file:/dev/./urandom
erddap_gold_standard | Unrecognized VM option 'UseConcMarkSweepGC'
erddap_gold_standard | Error: Could not create the Java Virtual Machine.
erddap_gold_standard | Error: A fatal exception has occurred. Program will exit.
erddap_gold_standard exited with code 1

[MathewBiddle]

Seems like something here isn't working right? https://github.com/axiom-data-science/docker-erddap/blob/4c4b510be4b87b30288eea0ab4a9632fc3009dc1/files/setenv.sh#L31-L35

[MathewBiddle]

Finally got it working! Here were the changes I had to make:

docker-compose.yml points to new image:

$ diff docker-compose.yml docker-compose.yml.20240130
5c5
<     image: axiom/docker-erddap:2.23-jdk17-openjdk
---
>     image: axiom/docker-erddap:2.18

setenv.sh removed -d64 and EXTRAS:

$ diff erddap/conf/setenv.sh erddap/conf/setenv.sh.backup20240130
16c16
< NORMAL="-server -Xms1G -Xmx2G"
---
> NORMAL="-server -d64 -Xms1G -Xmx2G"
19c19
< #EXTRAS="-XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled"
---
> EXTRAS="-XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled"

Now we have 2.23 running:

The key to figuring this out was being able to trace what docker-compose was doing in https://github.com/ioos/erddap-gold-standard/issues/62#issuecomment-1917820340

[mwengren]

@MathewBiddle Glad you sorted it out! I'm not clear where you are editing the setenv.sh file, since that should be built into the Docker container from the source setenv.sh file in axiom-data-science/docker-erddap, and it doesn't exist in this repo.

When I tried to run ERDDAP 2.23 via the standard Docker command in https://ioos.github.io/erddap-gold-standard/index.html#step-3-deploy-erddap-locally rather than Docker compose, it seemed to work properly out of the box, and these lines in setenv.sh properly detected the Java version and skipped adding those parameters to the Java command. Here's the relevant output from Docker:

ERDDAP_MAX_MEMORY=2G
ERDDAP_MIN_MEMORY=1G
ERDDAP Running with:  -server -Xms1G -Xmx2G -XX:+HeapDumpOnOutOfMemoryError -Djava.awt.headless=true  -DerddapContentDirectory=/usr/local/tomcat/content/erddap/ -Djna.tmpdir=/tmp/ -Djava.security.egd=file:$
dev/./urandom

The setenv.sh file is copied to the Docker image in these lines of the Dockerfile.

It's odd that the behavior would be different when run with Docker Compose than with the standard Docker command. Maybe it's because you were working off an existing ERDDAP Docker container that had an old version of setenv.sh hanging around that required updating? I would have expected it to pull down a new image of 2.23 when you updated the docker-compose.yml file and rebuilt the container. Strange. I'm not too familiar with Docker Compose.

Troubleshooting with docker image ls and docker container ls -a might help to understand what's going on. Maybe we can take a look sometime.

[MathewBiddle]

I'm not clear where you are editing the setenv.sh file, since that should be built into the Docker container from the source setenv.sh file in axiom-data-science/docker-erddap, and it doesn't exist in this repo.

That's what I've been most confused about in this whole thing. I'll try to shutdown the container, move setenv.sh and see if a new setenv.sh is built when I start the container.

Here is my docker image ls:

$ docker image ls
REPOSITORY            TAG                  IMAGE ID       CREATED         SIZE
axiom/docker-erddap   2.23-jdk17-openjdk   2e83e717aae3   2 months ago    1.32GB
axiom/docker-erddap   <none>               b33e965d8152   11 months ago   1.57GB
axiom/docker-erddap   2.18                 dda528a72b28   23 months ago   1.37GB

[MathewBiddle]

in my docker-compose.yml I mount that file into the container.

"${PWD}/erddap/conf/setenv.sh:/usr/local/tomcat/bin/setenv.sh"

[MathewBiddle]

Okay, I think that was the issue. I was mounting setenv.sh into the container so I could specify the various environment variables (admin contact stuff mainly). See https://github.com/ioos/erddap-gold-standard/blob/479a1dc50e7ddc0340587cbea6abd1f39f54d2c5/docker-compose.yml#L18-L38

I have now updated our docker-compose.yml to include those definitions and no longer mount in the setenv.sh file. The container spins up and runs as expected now.

$ diff --side-by-side docker-compose.yml docker-compose.yml.20240130
version: '2'                                                    version: '2'
services:                                                       services:
  erddap:                                                         erddap:
    container_name: "erddap_gold_standard"                          container_name: "erddap_gold_standard"
    image: axiom/docker-erddap:2.23-jdk17-openjdk             |     image: axiom/docker-erddap:2.18
    restart: unless-stopped                                         restart: unless-stopped
    ports:                                                          ports:
      - "80:8080"                                                     - "80:8080"
      - "443:8443"                                                    - "443:8443"
    volumes:                                                        volumes:
      - "${PWD}/erddap/conf/star_ioos_us.jks:/usr/local/tomca         - "${PWD}/erddap/conf/star_ioos_us.jks:/usr/local/tomca
      - "${PWD}/erddap/conf/server.xml:/usr/local/tomcat/conf         - "${PWD}/erddap/conf/server.xml:/usr/local/tomcat/conf
      - "${PWD}/erddap/conf/web.xml:/usr/local/tomcat/conf/we         - "${PWD}/erddap/conf/web.xml:/usr/local/tomcat/conf/we
    #  - "${PWD}/erddap/conf/setenv.sh:/usr/local/tomcat/bin/ |       - "${PWD}/erddap/conf/setenv.sh:/usr/local/tomcat/bin/s
      - "${PWD}/erddap/conf/robots.txt:/usr/local/tomcat/weba         - "${PWD}/erddap/conf/robots.txt:/usr/local/tomcat/weba
      - "${PWD}/erddap/content:/usr/local/tomcat/content/erdd         - "${PWD}/erddap/content:/usr/local/tomcat/content/erdd
      - "${PWD}/erddap/data:/erddapData"                              - "${PWD}/erddap/data:/erddapData"
      - "${PWD}/datasets:/datasets"                                   - "${PWD}/datasets:/datasets"
      - "/tmp/:/usr/local/tomcat/temp/"                               - "/tmp/:/usr/local/tomcat/temp/"
    environment:                                              <
      ERDDAP_MIN_MEMORY: 1G                                   <
      ERDDAP_MAX_MEMORY: 2G                                   <
      ERDDAP_baseHttpsUrl: "https://erddap.ioos.us"           <
      ERDDAP_adminInstitution: "Integrated Ocean Observing Sy <
      ERDDAP_adminInstitutionUrl: "https://ioos.github.io/"   <
      ERDDAP_adminIndividualName: "Mathew Biddle"             <
      ERDDAP_adminPosition: "Physical Scientist"              <
      ERDDAP_adminPhone: "240-533-9470"                       <
      ERDDAP_adminAddress: "1315 East-West Highway 2nd Floor" <
      ERDDAP_adminCity: "Silver Spring"                       <
      ERDDAP_adminStateOrProvince: "MD"                       <
      ERDDAP_adminPostalCode: "20910"                         <
      ERDDAP_adminCountry: "USA"                              <
      ERDDAP_adminEmail: "data.ioos@noaa.gov"                 <

[mwengren]

Great! That explains it. Specifying those env vars in docker-compose.yml is a better way to do that so that upstream changes in the container aren't missed. That should make it easier to update going forward.