Open jbliesener opened 5 years ago
I forgot to mention: a simple systemctl restart zrepl
on the production system brought it back to work.
I triaged the issue, but it's pretty low on my priority list. Implementation concerns:
Wouldn't it be sufficient to check certificate validity BEFORE every outgoing connection and AFTER every incoming connection?
No need to merge anything as long as you run from a single CA. If the CA certificate expires (pretty rare condition), you'd most probably need to restart anyway as it would affect more than just zrepl.
The race condition would be temporary as well and solve automatically on the next retry.
And, yes, it's a low-priority. Just wanted to mention it.
zrepl is running pretty successfully for some time now, replicating two production systems into one backup server. It's a pull/source TLS configuration in which the backup server calls into the production system through TLS.
Today, I ran into an unexpected problem when the backup server complained about an expired TLS certificate on one of the production systems.
When I checked the configuration, I noticed that the Letsencrypt TLS certificate on the production system had expired a month ago, but it already had been successfully renewed through a cron-scheduled task.
So, while the certificate was actually valid, it seems like zrepl continued to use the former expired certificate, that had been deleted from the file system already.
Of course I could include a zrepl restart into my certificate renewal procedure. On the other hand, I find it valid to either document this behavior or to reload the certificate.
For sink jobs, this could happen at the start of the outgoing connection, for source jobs, it could be at the end of an incoming connection.