PWX-36873: Add vault cooldowns

zoxpx commented 6 months ago

double "permission denied" REST error will put vault client into 5 minutes cooldown for all REST calls
can disable via VAULT_COOLDOWN_PERIOD:disabled

What this PR does / why we need it:

Turns out it is easy to overwhelm Hasicorp's Vault, and accidentally cause DDOS-attack

As a fix, we're adding REST cooldown for 5 minutes, to all Vault REST client calls

it is triggered by hitting 2 consecutive "permission denied" errors (i.e. PUT<key> ++ RENEW<token> both responded w/ "permission denied")

Which issue(s) this PR fixes (optional) Closes # PWX-36873

Special notes for your reviewer:

zoxpx commented 6 months ago

Note, @adityadani , @CharudathGopal -- looks like the AWS secrets integration test is failing (creds pulled out?)

zoxpx commented 6 months ago

Woops.. forgot to include UT with the original commit (fixed w/ follow-up f9fde71)

zoxpx commented 6 months ago

Ok.. I can't failing AWS Secrets Manager integration test with modifying my tests -- looks like this test has been failing for 7 months.

I'll proceed w/ the squash-commit.

libopenstorage / secrets