search

arangodb / arangodb-dcos

DCOS CLI for ArangoDB
Apache License 2.0
5 stars 2 forks source link

expose framework arguments in options JSON #3

Open mikestaub opened 8 years ago

mikestaub commented 8 years ago

I see here https://github.com/arangodb/arangodb-mesos-framework that I can specify a volume path for the arangodb docker containers to mount for persisting their data using the ARANGODB_VOLUME_PATH variable. (I plan on using a NFS dir for this)

I would like to provide this path in the JSON_FILE when I do: dcos package install --config= arangodb

Otherwise, if the dcos package is uninstalled I will lose all the arangodb data.

Is there another way to achieve this right now?

neunhoef commented 8 years ago

We are in the process of releasing a new version of the ArangoDB Mesos framework and things have changed. Therefore I will have to explain the situation for both versions.

In the version that is currently published via the mesosphere universe repository (Version 0.2.0 of the arangodb subcommand) the documented setup with ARANGODB_VOLUME_PATH works as described to the best of our knowledge. So you can specify, where the ArangoDB instances hold their data via "volume-path".

Having said that, we strongly believe that holding the actual data of ArangoDB on an NFS volume is a very bad idea, both for performance and for reliability of ArangoDB, due to the characteristics of NFS. Other, more modern distributed filesystems like QuoByte (will have a community version soon) might work better, but we still recommend local disks for ArangoDB data. We use memory mapped files and must rely on working and efficient msync() system calls.

For the version that we are in the process of releasing (Version 0.3.0 of the arangodb subcommand, which uses ArangoDB 2.8) the situation is completely different. We are using Mesos' persistent volumes for that version, such that an ArangoDB task that comes back after a restart has instant access to its old data. Furthermore, we have implemented automatic asynchronous replication in the cluster and automatic failover, which both improve resilience and fault tolerance.

However, this means that we have no control any more as to where in the filesystem the data is actually held. The mesos agents assign a "persistent volume" to our tasks and we simply use that. The configuration option ARANGODB_VOLUME_PATH is ignored, except for the coordinator tasks, which are essentially stateless and only need some ephemeral working directory on disk. As far as we know, there is not even a standard way to find out what path is used for the persistent volume. And yes, once you uninstall the ArangoDB service (= dcos package) the data is deleted. This is absolutely intended. You are intentiionally removing ArangoDB from the cluster and by doing so mesos will reclaim the diskspace and reassign it. If you want to backup the data use arangodump in that case.

This sounds harsh, but let me explain the philosophy behind this: The distributed application ArangoDB is responsible to keep your data safe by (internal) replication and automatic failover, as long as you keep it running. This is one of the core ideas of Apache Mesos in particular and cluster operating systems in general. Obviously, regular good old backups are an additional safeguard against catastrophic failures. After all, you want some offsite-backup of everything anyway. This is what arangodump/arangorestore is for.

mikestaub commented 8 years ago

Thanks Max for that timely and detailed response! I guess I will just wait for version 0.3.0 as it sounds like you figured all this stuff out already. Can you expand a bit on how automatic async replication and failover works? I'm assuming that with the default config of 2 db servers all of the documents live on both machines, and if one goes down the coordinator will route all queries to the running machine.