What this PR does / why we need it:
An etcd watch can fail in the following two ways when an etcd cluster is shutdown
etcd watch is cancelled by the server itself when it tries to gracefully
shutdown by sending a "Cancel" watch response. This is not always
guaranteed. However if etcd server does send this Cancel response it is instantaneous.
The kvdb client creates an etcd session with a timeout of 2 mins. This means
if etcd cluster is shutdown and we don't get the Cancel response, the kvdb
client will wait for 2 mins before shutting the watch.
This change reduces this timeout to 50s. The Portworx run-flat feature expects
all the nodes to conclude that etcd cluster is down within 1 minute. So all the
etcd watches need to error within that time. With this change an kvdb-etcd client watch
is guaranteed to fail within 50s instead of 2 mins if etcd is unreachable.
What this PR does / why we need it: An etcd watch can fail in the following two ways when an etcd cluster is shutdown
etcd watch is cancelled by the server itself when it tries to gracefully shutdown by sending a "Cancel" watch response. This is not always guaranteed. However if etcd server does send this Cancel response it is instantaneous.
The kvdb client creates an etcd session with a timeout of 2 mins. This means if etcd cluster is shutdown and we don't get the Cancel response, the kvdb client will wait for 2 mins before shutting the watch.
This change reduces this timeout to 50s. The Portworx run-flat feature expects all the nodes to conclude that etcd cluster is down within 1 minute. So all the etcd watches need to error within that time. With this change an kvdb-etcd client watch is guaranteed to fail within 50s instead of 2 mins if etcd is unreachable.