It would be really nice to be able to set Heap sizes for Java in Cantaloupe

[noahwsmith]

I think it would go here:

https://github.com/Islandora-Devops/isle-buildkit/blob/main/cantaloupe/rootfs/etc/s6-overlay/s6-rc.d/cantaloupe/run

Having CANTALOUPE_HEAP_MIN and CANTALOUPE_HEAP_MAX variables that passed into Xms and Xmx parameters in that command would do the trick.

[g7morris]

Hello everyone,

Today I was able to:

• create a new MR based on the changes suggested in this ticket - https://github.com/Islandora-Devops/isle-buildkit/pull/310/files

• run make bake cantaloupe on isle-buildkit without failure resulting in a new Docker image.

• test an isle-dc based project using this new image (local) by

  • using a different tag for cantaloupe - image: ${REPOSITORY}/cantaloupe:local
  • adding the following to the cantaloupe environment section of the docker-compose.yml

      CANTALOUPE_HEAP_MIN: "3G"
      CANTALOUPE_HEAP_MAX: "5G"

• increase and decrease the heap values dynamically. I increased to 4 GB min and 6 GB max.
  • Reviewed change in values by docker-compose exec cantaloupe bash -lc "printenv | grep HEAP"

Test 1

CANTALOUPE_HEAP_MAX=5G
CANTALOUPE_HEAPCACHE_PERSIST=false
CANTALOUPE_HEAPCACHE_PERSIST_FILESYSTEM_PATHNAME=/data/heap.cache
CANTALOUPE_HEAPCACHE_TARGET_SIZE=2G
CANTALOUPE_HEAP_MIN=3G

Test 2

CANTALOUPE_HEAP_MAX=6G
CANTALOUPE_HEAPCACHE_PERSIST=false
CANTALOUPE_HEAPCACHE_PERSIST_FILESYSTEM_PATHNAME=/data/heap.cache
CANTALOUPE_HEAPCACHE_TARGET_SIZE=2G
CANTALOUPE_HEAP_MIN=4G

@noahwsmith and @nigelgbanks Anything else I should do to test or consider?

[g7morris]

Possibly add these values to the isle-dc sample.env once this MR is merged?

[noahwsmith]

That sounds good. We'll test this out on a real project and report back...

[g7morris]

Okay coming back to this for some changes, further testing.

When I ran make bake cantaloupe on my local, everything passed however when I use the built image: islandora/cantaloupe:307-set-heap-sizes-for-java-in-cantaloupe from Dockerhub, cantaloupe & s6 crash with

bd-base-i8-staging-cantaloupe-1  | s6-rc: info: service legacy-services: starting
bd-base-i8-staging-cantaloupe-1  | s6-rc: info: service legacy-services successfully started
bd-base-i8-staging-cantaloupe-1  | Invalid initial heap size: -Xms
bd-base-i8-staging-cantaloupe-1  | Error: Could not create the Java Virtual Machine.
bd-base-i8-staging-cantaloupe-1  | Error: A fatal exception has occurred. Program will exit.
bd-base-i8-staging-cantaloupe-1  | s6-rc: info: service cantaloupe finish: executing...

[g7morris]

Still trying to get this to work.

I did add with-contenv to the run file but java is still not getting these variables correctly.

exec with-contenv s6-setuidgid cantaloupe java -Dcantaloupe.config=/opt/cantaloupe/cantaloupe.properties -Xms"${CANTALOUPE_HEAP_MIN}" -Xmx"${CANTALOUPE_HEAP_MAX}" -jar /opt/cantaloupe/cantaloupe.jar

[g7morris]

I also tried removing the curly braces, quotes etc.

-Xms$CANTALOUPE_HEAP_MIN} -Xmx$CANTALOUPE_HEAP_MAX -Xms${CANTALOUPE_HEAP_MIN} -Xmx${CANTALOUPE_HEAP_MAX} -Xms"${CANTALOUPE_HEAP_MIN}" -Xmx"${CANTALOUPE_HEAP_MAX}"

All don't work. :(

Feel like I'm missing something simple here.

[g7morris]

I've also changed to the following in the run file #!/usr/bin/with-contenv sh instead of #!/usr/bin/env bash which has also made no difference.

Should I not attempt to make a change to this command @nigelgbanks and use JAVA_OPTS instead?

[g7morris]

So changing this to

#!/usr/bin/with-contenv sh
set -e

# When bind mounting we need to ensure that we
# actually can write to the folder.
chown cantaloupe:cantaloupe /data

#exec with-contenv s6-setuidgid cantaloupe java -Dcantaloupe.config=/opt/cantaloupe/cantaloupe.properties -Xms"${CANTALOUPE_HEAP_MIN}" -Xmx"${CANTALOUPE_HEAP_MAX}" -jar /opt/cantaloupe/cantaloupe.jar

echo "CANTALOUPE_HEAP_MIN: ${CANTALOUPE_HEAP_MIN}"
echo "CANTALOUPE_HEAP_MAX: ${CANTALOUPE_HEAP_MAX}"

exec with-contenv s6-setuidgid cantaloupe java -Dcantaloupe.config=/opt/cantaloupe/cantaloupe.properties -Xms${CANTALOUPE_HEAP_MIN} -Xmx${CANTALOUPE_HEAP_MAX} -jar /opt/cantaloupe/cantaloupe.jar

Works for about 10-30 seconds but then crashes. What am I missing here?

[g7morris]

The output from logs is

bd-base-i8-staging-cantaloupe-1  | 16:49:41.713 [main] INFO  o.e.j.s.h.ContextHandler - Started o.e.j.s.ServletContextHandler@a9cd3b1{/,null,AVAILABLE}
bd-base-i8-staging-cantaloupe-1  | 16:49:41.776 [main] INFO  o.e.j.s.AbstractConnector - Started ServerConnector@229f66ed{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8182}
bd-base-i8-staging-cantaloupe-1  | 16:49:41.776 [main] INFO  o.e.j.s.Server - Started @10147ms
bd-base-i8-staging-cantaloupe-1  | s6-rc: info: service legacy-services: stopping
bd-base-i8-staging-cantaloupe-1  | s6-rc: info: service legacy-services successfully stopped
bd-base-i8-staging-cantaloupe-1  | s6-rc: info: service confd: stopping
bd-base-i8-staging-cantaloupe-1  | s6-rc: info: service cantaloupe: stopping
bd-base-i8-staging-cantaloupe-1  | s6-rc: info: service confd successfully stopped
bd-base-i8-staging-cantaloupe-1  | s6-rc: info: service cantaloupe finish: executing...
bd-base-i8-staging-cantaloupe-1  | s6-rc: info: service cantaloupe handled signal: 15...
bd-base-i8-staging-cantaloupe-1  | s6-rc: info: service cantaloupe received SIGTERM exiting gracefully
bd-base-i8-staging-cantaloupe-1  | s6-rc: info: service cantaloupe successfully stopped
bd-base-i8-staging-cantaloupe-1  | s6-rc: info: service ready: stopping
bd-base-i8-staging-cantaloupe-1  | s6-rc: info: service ready successfully stopped
bd-base-i8-staging-cantaloupe-1  | s6-rc: info: service tty: stopping

[g7morris]

So Cantaloupe can start but then from somewhere is told to stop?

[g7morris]

I would push these changes for others to test but am prevented from doing so.

[g7morris]

Okay thanks to @dannylamb I'm unstuck here.

For starters my test project had a health check on cantaloupe that was crashing things.

I got around the linter by correctly using this instead

#!/command/with-contenv bash
# shellcheck shell=bash
set -e

Now I'm seeing running containers that don't crash with successful Cantaloupe starts.

docker-compose exec cantaloupe bash -lc "ps aux | grep java"
cantalo+    1502  4.3  1.0 7097412 352492 ?      Ssl  18:33   0:12 java -Dcantaloupe.config=/opt/cantaloupe/cantaloupe.properties -Xms3G -Xmx5G -jar /opt/cantaloupe/cantaloupe.jar
root        1569 14.2  0.0   2400  1664 pts/0    Ss+  18:38   0:00 bash -lc ps aux | grep java
root        1577  0.0  0.0   1596   640 pts/0    S+   18:38   0:00 grep java

[g7morris]

I think we just need some folks to review and test when they can. Thanks!

[noahwsmith]

I just updated #310 - this passes Born-Digital's test suite.