`update-integration-configs` should fail fast when `admin_password` is not found

[joshuatcasey]

update-integration-configs should fail fast when admin_password is not found. It's required for CATs, and I suspect WATS and RATS.

We've had a few experiences in the UAA pipelines where update-integration-configs does not find an admin_password for CATs and then updates cats_integration_config.json with admin_password: ''. update-integration-configs shows as "passed" in Concourse status, which indicates no problems. That being said, we do see some output from the Credhub CLI in update-integration-configs that may indicate a problem.

+ check_fast_fails
+ '[' '!' -f integration-configs/concourse/uaa-acceptance-gcp/state/cats_integration_config.json -a '!' -f integration-configs/rats_integration_config.json -a '!' -f integration-configs/wats_integration_config.json ']'
+ setup_bosh_env_vars
+ set +x
/tmp/build/7f3ba2bf/bbl-state/concourse/uaa-acceptance-gcp/state /tmp/build/7f3ba2bf
/tmp/build/7f3ba2bf
+ set +x
Usage:
  credhub [OPTIONS] get [get-OPTIONS]

Get a credential value by name or ID

Application Options:
      --http-timeout=    Http timeout for http-client. Needs to have unit
                         passed in (i.e. 30s, 1m) [$CREDHUB_HTTP_TIMEOUT]
      --version          Version of CLI and targeted CredHub API
      --token            Return your current CredHub authentication token

Help Options:
  -h, --help             Show this help message

[get command options]
      -n, --name=        Name of the credential to retrieve
          --id=          ID of the credential to retrieve
          --versions=    Number of versions of the credential to retrieve
      -j, --output-json  Return response in JSON format
      -q, --quiet        Return value of credential without metadata
      -k, --key=         Return only the specified field of the requested
                         credential
updating CATs integration config file: concourse/uaa-acceptance-gcp/state/cats_integration_config.json...

CATs then fails with error Invalid configuration: 'admin_password' must be provided.

Note that we're still able to retrieve this password via Credhub CLI on our local machine, and it appears from the timestamp that it hasn't updated in a while (version_created_at: "2019-08-16T22:14:20Z"). I'm not sure why update-integration-configs does not retrieve this password.

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/172482424

The labels on this github issue will be updated when the story is started.

[joshuatcasey]

CC: @cloudfoundry/cf-uaa

[davewalter]

Hi @joshuatcasey,

What does credhub find -j -n cf_admin_password return? If it returns multiple credentials, then the xargs command is going to include all of them in the credhub get command, which will result in the error you are seeing. It almost sounds like you have a second deployment of cf-deployment on the same BOSH director.

We have added some sanity checks to the get_password_from_credhub helper function here. Please feel free to try them out to see if it helps you to understand why this is failing.

Regards, Dave and @Birdrock

[joshuatcasey]

We do have multiple passwords returned! There's another deployment that no longer exists but the password is still in Credhub. I'll clean that up and hopefully we don't see this in the future.

[davewalter]

Sounds good @joshuatcasey. I am going to go ahead and close this out.