search

F5Networks / f5-declarative-onboarding

F5 BIG-IP Declarative Onboarding
Apache License 2.0
58 stars 22 forks source link

Declaration with bad password doesn't fail gracefully #46

Open vsnine opened 5 years ago

vsnine commented 5 years ago

Found that typoing the password of the BIG-IP, which had been changed previously during bootstrap, the declaration takes a long time to abort. Digging into the container logs for restnoded, if bigIpPassword is incorrect, the 401 error is seen but the declaration doesn't immediately fail. The system continues to retry authenticating to the BIG-IP. Making subsequent requests for the job status just continues to display code 202 with message "processing".

Restarting the container is a quick fix to stop the retries.

Fri, 21 Jun 2019 20:34:18 GMT - finest: [f5-declarative-onboarding: restWorker.js] Got onboarding request.
Fri, 21 Jun 2019 20:34:18 GMT - info: [f5-declarative-onboarding: doUtil.js] Platform: CONTAINER
Fri, 21 Jun 2019 20:34:18 GMT - fine: [f5-declarative-onboarding: restWorker.js] Onboard starting for task 9f7e1cac-8fb2-4b20-9684-4da02ba5a8d3
Fri, 21 Jun 2019 20:34:18 GMT - info: [f5-declarative-onboarding: doUtil.js] Platform: CONTAINER
Fri, 21 Jun 2019 20:34:18 GMT - finest: [f5-declarative-onboarding: restWorker.js] done w/ initial accout setup
Fri, 21 Jun 2019 20:34:18 GMT - info: [f5-declarative-onboarding: restWorker.js] This is a BIG-IP
Fri, 21 Jun 2019 20:34:18 GMT - info: [f5-declarative-onboarding: restWorker.js] Waiting for device to be ready.
Fri, 21 Jun 2019 20:34:18 GMT - fine: [f5-declarative-onboarding: restWorker.js] list 10.1.1.2 /shared/echo/available
Fri, 21 Jun 2019 20:34:20 GMT - finest: [f5-declarative-onboarding: restWorker.js] tryUntil: got error {"code":401}
Fri, 21 Jun 2019 20:34:20 GMT - finest: [f5-declarative-onboarding: restWorker.js] typeof err object
Fri, 21 Jun 2019 20:34:20 GMT - finer: [f5-declarative-onboarding: restWorker.js] tryUntil error: <?xml version="1.0" encoding="ISO-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
  "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en">
<head>
<title>Authentication required!</title>
<link rev="made" href="mailto:support@f5.com" />
<style type="text/css"><!--/*--><![CDATA[/*><!--*/
    body { color: #000000; background-color: #FFFFFF; }
    a:link { color: #0000CC; }
    p, address {margin-left: 3em;}
    span {font-size: smaller;}
/*]]>*/--></style>
<style type="text/css"><!--/*--><![CDATA[/*><!--*/
* { width: 400px; font-size: 100%; font-style: normal; }
html { text-align: center; }
body { background: #ffffff; text-align: left; font-family: sans-serif; font-size: 70%; color: #333333; }

a,span { width: auto; }
h1,h2,h3 { margin: 20px 0px 20px 0px; font-weight: bold; }

h1 { padding: 5px; border: 1px solid #999999; background: #eeeeee; color: #000000; font-size: 125%;  }
hr { height: 1px; border: none; border-top: 1px solid #999999; }
img { border: 0px; }
p { width: 350px; margin: 15px 25px 15px 25px; line-height: 135%; }
/*]]>*/--></style>

</head>

<body>
<h1>Authentication required!</h1>
<p>

    This server could not verify that you are authorized to access
    the URL "/mgmt/shared/echo/available".
    You either supplied the wrong credentials (e.g., bad password), or your
    browser doesn't understand how to supply the credentials required.

  </p>
<p>

    In case you are allowed to request the document, please
    check your user-id and password and try again.

</p>

<h2>Error 401</h2>
<address>
  <a href="/">10.1.12</a><br />

  <span>Fri Jun 21 20:34:18 2019<br />
  </span>
</address>
</body>
</html>

 tries left: 0

Shortly after the initial failure it indicates that there are 90 attempts remaining and continues to try with the bad password. Example:

 tries left: 90
Fri, 21 Jun 2019 20:34:20 GMT - finest: [f5-declarative-onboarding: restWorker.js] tryUntil: retryOrReject: numRemaining: 90 , code: 401 , message: tryUntil: max tries reached: <?xml version="1.0" encoding="ISO-8859-1"?> ... removed for brevity ...

If I allow it to continue until it runs out of retries, this is the result.

Fri, 21 Jun 2019 20:52:18 GMT - finer: [f5-declarative-onboarding: restWorker.js] Max tries reached.
Fri, 21 Jun 2019 20:52:18 GMT - info: [f5-declarative-onboarding: restWorker.js] Device initialization failed tryUntil: max tries reached: tryUntil: max tries reached: <?xml version="1.0" encoding="ISO-8859-1"?>
...snip...
Fri, 21 Jun 2019 20:52:18 GMT - severe: [f5-declarative-onboarding: doUtil.js] Error initializing BigIp: tryUntil: max tries reached: tryUntil: max tries reached: <?xml version="1.0" encoding="ISO-8859-1"?>
...snip...
Fri, 21 Jun 2019 20:52:18 GMT - severe: [f5-declarative-onboarding: restWorker.js] Error onboarding: tryUntil: max tries reached: tryUntil: max tries reached: <?xml version="1.0" encoding="ISO-8859-1"?>
...snip...
Fri, 21 Jun 2019 20:52:18 GMT - info: [f5-declarative-onboarding: restWorker.js] Rolling back configuration
Fri, 21 Jun 2019 20:52:18 GMT - severe: [f5-declarative-onboarding: configManager.js] Error getting current config: Cannot read property 'list' of undefined
Fri, 21 Jun 2019 20:52:18 GMT - severe: [f5-declarative-onboarding: restWorker.js] Error rolling back: Cannot read property 'list' of undefined
Fri, 21 Jun 2019 20:52:18 GMT - severe: [f5-declarative-onboarding: restWorker.js] Error rolling back configuration: Cannot read property 'list' of undefined
Fri, 21 Jun 2019 20:52:18 GMT - fine: [f5-declarative-onboarding: restWorker.js] Onboard configuration complete. Saving sys config.
Fri, 21 Jun 2019 20:52:18 GMT - severe: [f5-declarative-onboarding: restWorker.js] Error during onboarding: Cannot read property 'save' of undefined
Fri, 21 Jun 2019 20:52:20 GMT - finest: socket 263 closed
Fri, 21 Jun 2019 20:52:20 GMT - finest: socket 264 opened
Fri, 21 Jun 2019 20:52:25 GMT - finest: socket 264 closed

At this point requesting the task status does return a 500 code along with the error message seen initially.

I feel like this should abort after one or two retries if not immediately.

vsnine commented 5 years ago

This was seen against DO container 1.5.0.

seattlevine commented 5 years ago

Agreed. We generally have to retry on errors in DO because the VE may still be coming up. However, if we get a 401 we can stop trying (or at least shorten the retries from there)

focrensh commented 5 years ago

AT-478 for tracking