During a recent network outage I had a vault-sidekick Pod that was unable to reach vault during a few subsequent lease-renewal attempts, timing out with:
failed to renew the resource: type: gcp, path: gcp/key/service for renewal, error: Put https://vault/v1/sys/leases/renew: dial tcp: i/o timeout
As a result, the lease expired and subsequent renewals failed with:
failed to renew the resource: type: gcp, path: gcp/key/service for renewal, error: Error making API request.
lease not found or lease is not renewable
rescheduling the resource: type: gcp, path: gcp/key/service, channel: 0xdeadbeef
This failure repeated indefinitely until the situation was noticed and the Pod was restarted manually. Ideally, the vault-sidekick Pod would have realized this was a non-recoverable error and exited. It would have then automatically restarted, generated a new lease, and automatically become healthy again.
vault-sidekick v0.3.9 was run with (roughly) the following configuration:
./vault-sidekick -v=3 -vault=https://vault -renew-token -cn=gcp:gcp/key/service:renew=true
During a recent network outage I had a
vault-sidekick
Pod that was unable to reachvault
during a few subsequent lease-renewal attempts, timing out with:As a result, the lease expired and subsequent renewals failed with:
This failure repeated indefinitely until the situation was noticed and the Pod was restarted manually. Ideally, the
vault-sidekick
Pod would have realized this was a non-recoverable error and exited. It would have then automatically restarted, generated a new lease, and automatically become healthy again.vault-sidekick
v0.3.9 was run with (roughly) the following configuration:./vault-sidekick -v=3 -vault=https://vault -renew-token -cn=gcp:gcp/key/service:renew=true