Open scor2k opened 1 year ago
Why not add sleep $((1 + $RANDOM % 360));
to your mysql/reload.sh
command? (obviously you can adjust the 360
seconds to whatever suits your needs. Your certificates will be rotated before they expire so you do not have to update right away when you generate the new ones.
Why not add
sleep $((1 + $RANDOM % 360));
to yourmysql/reload.sh
command? (obviously you can adjust the360
seconds to whatever suits your needs. Your certificates will be rotated before they expire so you do not have to update right away when you generate the new ones.
Thank you for your reply, @komapa. Yes, it's a possible solution, but it's can guarantee nothing. We did the same by adjusting every next node's TTL to one (hour, day), but it also won't protect us in the case of bad luck.
I think you pretty much said this?
command = "consul lock -child-exit-code /some/consul/path/prefix /opt/consul-template/templates/mysql/reload.sh"
Hi!
We use consul-template + Vault PKI to provide SSL certificates for the MySQL Galera cluster. We did some tests with short TTL (15m) for SSL and faced the issue when the Galera cluster crashed because of simultaneous SSL re-generation for all nodes (we send
ALTER INSTANCE RELOAD TLS;
via reload-script each time a new certificate has been done).Also, we faced the same issue with the Apache Kafka cluster (with SSL) but TTL was 7 days. Honestly, it was only once for 1 month, but it has happened.
We applied a fix to shift TTL for 1 day for every next node, it helps to reduce the chance, but it's not a fix.
My question is simple: Any way you have some distributive lock (via Consul) to prevent all instances from updating certificates at the same time?
mysqld config x 3 instatces
consul-template configs x 3 instances
set -eo pipefail
STATUS=0
if [ -f '/opt/mysql/current/bin/mysql' -a -S '/tmp/mysql.sock' ]; then echo "ALTER INSTANCE RELOAD TLS;" | /opt/mysql/current/bin/mysql -u root -p'super-secure-password' STATUS=$? fi
exit $STATUS