docker-compose persistence problem

[EsGeh]

I am not able to persist any data over container restarts. Demonstration:

docker-compose.yaml:

version: '3'

services:

  xmldb:
    image: existdb/existdb:latest
    container_name: xmldb
    volumes:
      - ./res:/eoa/res
      - ./runtime_data/exist_data:/exist-data
    # just for debugging:
    ports:
      - 8443:8443
      - 8080:8080
  ...

Now when i issue these commands ...

$ docker-compose up -d xmldb
Creating network "eoa2-xmldb_default" with the default driver
Creating xmldb ... done
$ docker-compose exec xmldb java -jar start.jar client --no-gui
Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8   -Djava.awt.headless=true   -Dorg.exist.db-connection.cacheSize=256M   -Dorg.exist.db-connection.pool.max=20   -XX:+UnlockExperimentalVMOptions   -XX:+UseCGroupMemoryLimitForHeap   -XX:+UseG1GC   -XX:+UseStringDeduplication   -XX:MaxRAMFraction=1
eXist version 5.0.0-SNAPSHOT (1648d512c), Copyright (C) 2001-2019 The eXist-db Project
eXist-db comes with ABSOLUTELY NO WARRANTY.
This is free software, and you are welcome to redistribute it
under certain conditions; for details read the license file.

Connecting to database...
Connected :-)

type help or ? for help.
exist:/db> mkcol testcreated collection.
exist:/db> ls
crwxr-xr-x      SYSTEM  dba     Thu Jan 31 13:52:10 GMT 2019    system
crwxr-xr-x      SYSTEM  dba     Thu Jan 31 13:52:12 GMT 2019    apps
crwxr-xr-x      admin   dba     Thu Jan 31 13:59:16 GMT 2019    test
exist:/db> quit
quit.
31 Jan 2019 13:59:43,259 [global.BrokerPools.ShutdownHook] INFO  (BrokerPools.java [lambda$static$0]:68) - Executing shutdown thread

no data is saved in my volume:

$  ls -Al  runtime_data/exist_data/
insgesamt 0

also, when restarting the container, all settings are lost ( my collection test doesn't exist) !:

$ docker-compose down
Stopping xmldb ... done
Removing xmldb ... done
Removing network eoa2-xmldb_default
$ docker-compose up -d xmldb
Creating network "eoa2-xmldb_default" with the default driver
Creating xmldb ... done
$ docker-compose exec xmldb java -jar start.jar client --no-gui
Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8   -Djava.awt.headless=true   -Dorg.exist.db-connection.cacheSize=256M   -Dorg.exist.db-connection.pool.max=20   -XX:+UnlockExperimentalVMOptions   -XX:+UseCGroupMemoryLimitForHeap   -XX:+UseG1GC   -XX:+UseStringDeduplication   -XX:MaxRAMFraction=1
eXist version 5.0.0-SNAPSHOT (1648d512c), Copyright (C) 2001-2019 The eXist-db Project
eXist-db comes with ABSOLUTELY NO WARRANTY.
This is free software, and you are welcome to redistribute it
under certain conditions; for details read the license file.

Connecting to database...
Connected :-)

type help or ? for help.
exist:/db> ls
crwxr-xr-x      SYSTEM  dba     Thu Jan 31 14:03:27 GMT 2019    system
crwxr-xr-x      SYSTEM  dba     Thu Jan 31 14:03:28 GMT 2019    apps
exist:/db>

What am I missing here?

[duncdrum]

whats the output of docker volume ls on your system, i.e. does it show the default exist_data volume?

[EsGeh]

This is the output of docker-volume ls :

$ docker volume ls
DRIVER              VOLUME NAME

no volumes are shown. (I cleaned unneeded volumes using docker volume prune before starting the container)

[duncdrum]

i can't do a full debug run of you compose file right now, but the problem seems to be that exist doesn't know about the changed data-dir path. So ii falls back to an internal container path, which since its internal, is ephemeral.

You need to change your set-up so that either the existdb image is properly configured with a different data-dir path, which can then be mounted from a local dir via your compose file. Or tweak your compose file in line with the one in this repo, which is declaring the default data-dir path as a volume inside of the exist via the .env file container, so it becomes persistent .

PS.: You can see where exist thinks its data-dir is, at the start of the exist.log

[EsGeh]

I solved this: It is decisive to first copy the exist config dir and data dir from the container. (Because I think exist fails if /exist/config and /exist/webapp/WEB-INF/data are empty.) My current solution is this:

1. run container from a modified docker-compose file, which doesn't mount any volume /exist/config, or /exist/webapp/WEB-INF/data
2. copy the config dirs /exist/config and /exist/webapp/WEB-INF/data from the container to runtime_data/exist-config resp. runtime_data/exist-data (like described in the README
3. stop the container
4. docker-compose up xmldb to run the container with the actual docker-compose file, which mounts the local dirs runtime_data/exist-config or runtime_data/exist-data to the corresponding paths in the container PLUS the xml files I want to load into exist
5. docker-compose exec xmldb java -jar start.jar client --no-gui into the container and load resources into the database
6. docker-compose down

I hope this helps others