RocksDB Storage option broken since 3.3.10 when starting a new database image

gbjbaaha commented 6 years ago

I believe the Docker image post version 3.3.9 (so 3.3.10 and 3.3.11) is broken when you elect to initialise a new database using storage option "rocksdb".

Here is my test case:-

Base O/S = Ubuntu 18.04 or OSX 10.13.5 Docker version = 18.03.1-ce Docker Compose version = 1.21.1

I am using Docker Compose, but I see the same results if I use the basic "docker run" style command to pull and start the containers.

Create a Docker compose file referencing version 3.3.9 as follows:-

# Docker Composition for test ArangoDB single node cluster.
#
version : '3.3'
services:
  arangodb:
    image: arangodb/arangodb:3.3.9
    environment:
      - ARANGO_ROOT_PASSWORD=password
      - ARANGO_STORAGE_ENGINE=rocksdb
    ports:
      - 8529:8529
    volumes:
      - /Users/me/arangodb/data339/arangodb:/var/lib/arangodb3
      - /Users/me/arangodb/data339/arangoapps:/var/lib/arangodb3-apps
    networks:
      - arangodb
networks:
  arangodb:
    external: true

Ensure that the volume directories mapped do not exist prior to the test. Start up the container with "docker-compose up" This works as expected.

Change the version of ArangoDB to 3.3.10 or 3.3.11. Change the volume directories to point to a new set of empty directories to avoid any confusion. Here is the revised test:-

# Docker Composition for test ArangoDB single node cluster.
#
version : '3.3'
services:
  arangodb:
    image: arangodb/arangodb:3.3.11
    environment:
      - ARANGO_ROOT_PASSWORD=password
      - ARANGO_STORAGE_ENGINE=rocksdb
    ports:
      - 8529:8529
    volumes:
      - /Users/me/arangodb/data3311/arangodb:/var/lib/arangodb3
      - /Users/me/arangodb/data3311/arangoapps:/var/lib/arangodb3-apps
    networks:
      - arangodb
networks:
  arangodb:
    external: true

Start up the container with "docker-compose up" This fails.

Here is output from my working 3.3.9 test:-

docker-compose up
Pulling arangodb (arangodb/arangodb:3.3.9)...
3.3.9: Pulling from arangodb/arangodb
3e17c6eae66c: Already exists
4c7a4cfddc95: Pull complete
9aea8aa9f599: Pull complete
e5ccb1595c6c: Pull complete
188056ffcdae: Pull complete
14b4d1c645d8: Pull complete
5aeb82f8c285: Pull complete
Digest: sha256:7e0769fe46ace61726461c0c156b53a5836117f8f06fdf9d759cfe37e597ee13
Status: Downloaded newer image for arangodb/arangodb:3.3.9
Creating arangodb_arangodb_1 ... done
Attaching to arangodb_arangodb_1
arangodb_1  | choosing Rocksdb storage engine
arangodb_1  | Initializing root user...Hang on...
arangodb_1  | 2018-07-02T09:49:23Z [12] INFO {syscall} file-descriptors (nofiles) hard limit is 1048576, soft limit is 1048576
arangodb_1  | 2018-07-02T09:49:23Z [12] WARNING {memory} /proc/sys/vm/overcommit_memory is set to '1'. It is recommended to set it to a value of 2
arangodb_1  | 2018-07-02T09:49:23Z [12] WARNING {memory} execute 'sudo bash -c "echo 2 > /proc/sys/vm/overcommit_memory"'
arangodb_1  | 2018-07-02T09:49:24Z [12] INFO {startup} Server will now shutdown due to upgrade, database init or admin restoration.
arangodb_1  | Initializing database...Hang on...
arangodb_1  | Database initialized...Starting System...
arangodb_1  | 2018-07-02T09:49:28Z [1] INFO ArangoDB 3.3.9 [linux] 64bit, using jemalloc, build tags/v3.3.9-0-g8b8476c77e, VPack 0.1.30, RocksDB 5.6.0, ICU 58.1, V8 5.7.492.77, OpenSSL 1.1.0f  25 May 2017
arangodb_1  | 2018-07-02T09:49:28Z [1] INFO {authentication} Jwt secret not specified, generating...
arangodb_1  | 2018-07-02T09:49:28Z [1] INFO detected operating system: Linux version 4.9.87-linuxkit-aufs (root@95fa5ec30613) (gcc version 6.4.0 (Alpine 6.4.0) ) #1 SMP Wed Mar 14 15:12:16 UTC 2018
arangodb_1  | 2018-07-02T09:49:28Z [1] INFO using storage engine rocksdb
arangodb_1  | 2018-07-02T09:49:28Z [1] INFO {cluster} Starting up with role SINGLE
arangodb_1  | 2018-07-02T09:49:28Z [1] INFO {syscall} file-descriptors (nofiles) hard limit is 1048576, soft limit is 1048576
arangodb_1  | 2018-07-02T09:49:28Z [1] INFO {authentication} Authentication is turned on (system only), authentication for unix sockets is turned on
arangodb_1  | 2018-07-02T09:49:28Z [1] WARNING {memory} /proc/sys/vm/overcommit_memory is set to '1'. It is recommended to set it to a value of 2
arangodb_1  | 2018-07-02T09:49:28Z [1] WARNING {memory} execute 'sudo bash -c "echo 2 > /proc/sys/vm/overcommit_memory"'
arangodb_1  | 2018-07-02T09:49:28Z [1] INFO using endpoint 'http+tcp://0.0.0.0:8529' for non-encrypted requests
arangodb_1  | 2018-07-02T09:49:28Z [1] INFO Please note that a new bugfix version '3.3.11' is available
arangodb_1  | 2018-07-02T09:49:28Z [1] INFO ArangoDB (version 3.3.9 [linux]) is ready for business. Have fun!

And here is the output from my failing 3.3.11 test:-

docker-compose up
Pulling arangodb (arangodb/arangodb:3.3.11)...
3.3.11: Pulling from arangodb/arangodb
3e17c6eae66c: Pull complete
dc8136269288: Pull complete
e9b0ef755caf: Pull complete
1e1f508032aa: Pull complete
3599dcb0cd3d: Pull complete
9b084f31df5c: Pull complete
89b493964223: Pull complete
Digest: sha256:43e6edb340c98fb47e8bc39a524b8668e75fe19195957af085c9610708290fc4
Status: Downloaded newer image for arangodb/arangodb:3.3.11
Creating arangodb_arangodb_1 ... done
Attaching to arangodb_arangodb_1
arangodb_1  | choosing Rocksdb storage engine
arangodb_1  | Initializing root user...Hang on...
arangodb_1  | 2018-07-02T10:00:06Z [9] INFO {syscall} file-descriptors (nofiles) hard limit is 1048576, soft limit is 1048576
arangodb_1  | 2018-07-02T10:00:06Z [9] WARNING {memory} /proc/sys/vm/overcommit_memory is set to '1'. It is recommended to set it to a value of 0 or 2
arangodb_1  | 2018-07-02T10:00:06Z [9] WARNING {memory} execute 'sudo bash -c "echo 2 > /proc/sys/vm/overcommit_memory"'
arangodb_1  | 2018-07-02T10:00:09Z [9] INFO {startup} Server will now shutdown due to upgrade, database init or admin restoration.
arangodb_1  | Initializing database...Hang on...
arangodb_1  | ArangoDB didn't start correctly during init
arangodb_1  | 2018-07-02T10:00:16Z [52] INFO ArangoDB 3.3.11 [linux] 64bit, using jemalloc, build tags/v3.3.11-0-gbe3795a25a, VPack 0.1.30, RocksDB 5.6.0, ICU 58.1, V8 5.7.492.77, OpenSSL 1.1.0f  25 May 2017
arangodb_1  | 2018-07-02T10:00:16Z [52] INFO {authentication} Jwt secret not specified, generating...
arangodb_1  | 2018-07-02T10:00:16Z [52] INFO detected operating system: Linux version 4.9.87-linuxkit-aufs (root@95fa5ec30613) (gcc version 6.4.0 (Alpine 6.4.0) ) #1 SMP Wed Mar 14 15:12:16 UTC 2018
arangodb_1  | 2018-07-02T10:00:16Z [52] FATAL {startup} content of 'ENGINE' file '/var/lib/arangodb3/ENGINE' and command-line/configuration option value do not match: 'mmfiles' != 'rocksdb'. please validate the command-line/configuration option value of '--server.storage-engine' or use a different database directory if the change is intentional
arangodb_arangodb_1 exited with code 1

dothebart commented 6 years ago

Confirmed! Thanks for reporting.

comigo-github commented 6 years ago

It looks that the problem is with the new use of the --config parameter. In the previous dockers once you use the ARANGO_STORAGE_ENGINE=rocksdb the entrypoint script changed the /etc/arangodb3/arangod.conf directly. But now (in version 3.3.10 and 3.3.11) the entrypoint first copies the arangod.conf file to a temporary location, changes it there, and then pass it as an argument to the arangod command using --config. It seems that if you change it back to the previous implementation it works fine.

neunhoef commented 6 years ago

Please try again now. I have fixed the problem and submitted a PR to the official Docker images which was merged by now. This should be fixed in 3.3.10 and 3.3.11 now as well as in our nightly preview images.

gbjbaaha commented 6 years ago

Just checked Docker Hub (https://hub.docker.com/r/arangodb/arangodb/tags) and 3.3.11 still showing as 7 days ago. Can you confirm that this new image has indeed been pushed up to the hub?

dothebart commented 6 years ago

Hi, If I get that correctly this patch was only applied to _/arangodb at the time being.

gbjbaaha commented 6 years ago

Hi, OK. Do you know when the updated version will get pushed onto Docker Hub? As requested, as soon as this new image is available I will be happy to test it and report back via this issue. Thanks

neunhoef commented 6 years ago

Sorry, I was not aware that ths push is not automatic, therefore this was delayed. I have just updated the images

arangodb/arangodb:3.3.10
arangodb/arangodb:3.3.11
arangodb/arangodb:latest with a fixed version. Unfortunately, arangodb the official one seems to take longer. My PR was merged 3 days ago and the update should be automatic...

gbjbaaha commented 6 years ago

Pulled both 3.3.11 and 3.3.10 tags and re-tested. Happy to report that all now seems good 👍 Many thanks for the quick fix.

arangodb / arangodb-docker

RocksDB Storage option broken since 3.3.10 when starting a new database image #50