Support different Zookeepers clusters to store state

mesos / elasticsearch

Elasticsearch on Mesos

Apache License 2.0

242 stars 90 forks source link

Support different Zookeepers clusters to store state #231

Open kamaradclimber opened 9 years ago

kamaradclimber commented 9 years ago

In our environment, we want to separate the zookeeper cluster used by mesos masters from the zk used to store schedulers data.

It would be a nice improvement to allow to use separate zookeepers clusters.

frankscholten commented 9 years ago

It's an interesting enhancement but not a priority at the moment. Also requires a lot of testing relative to the feature. However you are welcome to add a PR for this.

frankscholten commented 9 years ago

We have heard from other people that this is an important feature so we will add it to 0.3

frankscholten commented 9 years ago

[x] Add zookeeperFrameworkUrl to ZookeeperCLIParameter
[x] Elasticsearch should use zookeeperFrameworkUrl instead of zookeeperUrl
[x] Add zookeeperFrameworkTimeout
[x] ZookeeperFrameworkUrl should use value of ZookeeperMesosUrl when it is not set
[x] Rename zookeeperUrl in ZookeeperCLIParameter to zookeeperMesosUrl
[x] Check hardcoded params in system test
[x] Add system test

JohnOmernik commented 9 years ago

Based on the configuration page, the /mesos at the end of Zookeeper is hardcoded somewhere. My Mesos cluster uses /mesosha and when I put in zk://node1:5181,node2:5181,node3:5131/mesosha, the configuration overview still has /mesos in the valud MesosZKURL. Shouldn't we take this from the config rather than assuming /mesos?

systeminsightsbuild commented 8 years ago

If heard of this desire as well. What are the arguments for doing this?

philwinder commented 8 years ago

@WalterDalton I'm sorry, I've only just seen this. This is a separate issue and I will open a bug. @systeminsightsbuild It could be said there are two reasons. 1) clarity. You don't have lots of scheduler spam being written to the Mesos ZK. (I don't agree with this point). 2) The more reasonable thought is that you need Mesos to be able to read/write to ZK. By isolating the schedulers to write to another cluster, you mitigate against a scheduler overwhelming ZK with too many reads/writes.

@frankscholten It seems you have ticked lots of boxes on this, and I remember you doing the work. What happened to it? The options proposed above are not in the code?