Creating a Service Broker fails with "Unknown Error"

[belinda-liu]

Describe the bug

When creating a service broker on a cf-for-k8s environment, the command will fail with an "Unknown Error".

To Reproduce*

1. Push some app test-app
2. Run cf create-service-broker 'test-app' <username> <password> <test-app-route>
3. Watch the command error with the following message:

   Creating service broker test-app as admin...
   Job (5fd0d244-5c74-406e-a999-1d608c669461) failed: An unknown error occurred.
   FAILED

Looking at the cf-api-worker logs:

{"timestamp":"2020-10-02T18:35:01.821123094Z","message":"Request failed: 500: {\"error_code\"=>\"UnknownError\", \"description\"=>\"An unknown error occurred.\", \"code\"=>10001, \"test_mode_info\"=>{\"description\"=>\"error response\", \"error_code\"=>\"CF-TargetErro
 r\", \"backtrace\"=>[\"/usr/local/lib/ruby/gems/2.5.0/gems/cf-uaa-lib-3.14.3/lib/uaa/http.rb:123:in `json_parse_reply'\", \"/usr/local/lib/ruby/gems/2.5.0/gems/cf-uaa-lib-3.14.3/lib/uaa/token_issuer.rb:77:in `request_token'\", \"/usr/local/lib/ruby/gems/2.5.0/gems/cf-
 uaa-lib-3.14.3/lib/uaa/token_issuer.rb:267:in `client_credentials_grant'\", \"/cloud_controller_ng/lib/cloud_controller/uaa/uaa_client.rb:34:in `token_info'\", \"/cloud_controller_ng/lib/cloud_controller/uaa/uaa_client.rb:13:in `auth_header'\", \"/cloud_controller_ng/
 lib/cloud_controller/uaa/uaa_client.rb:114:in `scim'\", \"/cloud_controller_ng/lib/cloud_controller/uaa/uaa_client.rb:103:in `block in get'\", \"/cloud_controller_ng/lib/cloud_controller/uaa/uaa_client.rb:107:in `with_cache_retry'\", \"/cloud_controller_ng/lib/cloud_c
 ontroller/uaa/uaa_client.rb:103:in `get'\", \"/cloud_controller_ng/lib/cloud_controller/uaa/uaa_client.rb:27:in `block in get_clients'\", \"/cloud_controller_ng/lib/cloud_controller/uaa/uaa_client.rb:26:in `map'\", \"/cloud_controller_ng/lib/cloud_controller/uaa/uaa_c
 lient.rb:26:in `get_clients'\", \"/cloud_controller_ng/lib/services/sso/uaa/uaa_client_manager.rb:14:in `get_clients'\", \"/cloud_controller_ng/lib/services/sso/dashboard_client_manager.rb:94:in `fetch_clients_from_uaa'\", \"/cloud_controller_ng/lib/services/sso/dashb
 oard_client_manager.rb:37:in `synchronize_clients_with_catalog'\", \"/cloud_controller_ng/app/jobs/v3/services/service_broker_catalog_updater.rb:20:in `refresh'\", \"/cloud_controller_ng/app/jobs/v3/services/synchronize_broker_catalog_job.rb:49:in `perform'\", \"/clou
 d_controller_ng/app/jobs/v3/services/synchronize_broker_catalog_job.rb:13:in `perform'\", \"/cloud_controller_ng/app/jobs/wrapping_job.rb:11:in `perform'\", \"/cloud_controller_ng/app/jobs/wrapping_job.rb:11:in `perform'\", \"/cloud_controller_ng/app/jobs/timeout_job.
 rb:13:in `block in perform'\", \"/usr/local/lib/ruby/2.5.0/timeout.rb:93:in `block in timeout'\"
...
}

Expected behavior

The command should succeed.

Additional context

After chatting with some folks on CAKE (CAPI for k8s), they pointed out that cf-for-k8s does not bootstrap with the cc-service-dashboards UAA client, which is required for this command. This was previously provided in cf-deployment (see https://github.com/cloudfoundry/cf-deployment/blob/3ba20341c7431ace178f8b12d44c470738db1326/cf-deployment.yml#L685-L689).

cf-for-k8s SHA

221ac187883e3ffd4e456705fbbe738f577b2338

Deploy instructions

This was deployed via TAS.

Cluster information

GCP

CLI versions

Fails on both v6 and v7 cf CLI.

1. ytt --version: 0.30.0
2. kapp --version: 0.33.0
3. kubectl version: Client Version: version.Info{Major:"1", Minor:"16+", GitVersion:"v1.16.6-beta.0", GitCommit:"e7f962ba86f4ce7033828210ca3556393c377bcc", GitTreeState:"clean", BuildDate:"2020-01-15T08:26:26Z", GoVersion:"go1.13.5", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.8+vmware.1", GitCommit:"3cbbcf0977af5f3cf455115d060b081f2b8e2329", GitTreeState:"clean", BuildDate:"2020-06-29T22:33:24Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
4. cf version: cf version 7.0.2+17b4eeafd.2020-07-24

Thanks! @belinda-liu && @reid47

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/175102096

The labels on this github issue will be updated when the story is started.

[jspawar]

This will also require some changes on capi-k8s-release to re-introduce config related to that OAuth client (specifically its ID and its client secret) since we recently removed what we presumed to be unused config: https://github.com/cloudfoundry/capi-k8s-release/commit/fe240bd4be20edbb40358e64bba699babe32c3f0

[jamespollard8]

Thanks @belinda-liu for reporting this. We have a related story prioritized/in-flight and we'll circle back on your issue here once we finish that story and have a better idea of the landscape. (Enable at least 1 services suites CATs test)

[jamespollard8]

This now works as long as you've configured your service broker is running with real trusted certs.