Updating unicast_hosts.txt without pod restart

[balbirthomas]

Is it possible to have unicast_hosts.txt updated without restarting the pod.

[justinmclark]

I can confirm that unicast_hosts.txt can be updated without restarting the pod. Here are the steps I took:

1. In __init__(), configure the starting self._stored.nodes to only have one hostname, rather than SEED_SIZE. [1]
2. Remove the unicast_hosts.txt string from _config_hash() so pod restarts aren't triggered by changes to the file. [2]
3. Add a unit to the cluster and _on_elasticsearch_unit_joined() will add a new hostname to self._stored.nodes.
4. kubectl exec -n lma <leader-pod> -- cat /usr/share/elasticsearch/config/unicast_hosts.txt and see that the new host has been added to the file without the pod resetting.

One thing to be aware of is that unicast_hosts.txt is only updated in the leader pod and new pods.

[1] Change

self._stored.set_default(nodes=[self._host_name(i)
                                for i in range(SEED_SIZE)])

to

self._stored.set_default(nodes=[self._host_name(0)])

[2] Change

config_string = self._seed_hosts() + self._elasticsearch_config() +\
    self._jvm_config() + self._logging_config() + self._log4j_config()

to

config_string = self._elasticsearch_config() +\
    self._jvm_config() + self._logging_config() + self._log4j_config()