`add-gcp-broker` authentication configuration fails

shosti commented 6 years ago

I was following the tutorial to get the service catalog set up, but it failed at the add-gcp-broker step with ErrorGettingAuthCredentials. Here's the output:

[shosti@drogo:~]$ kubectl describe clusterservicebrokers/gcp-broker
Name:         gcp-broker
Namespace:    
Labels:       <none>
Annotations:  kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"servicecatalog.k8s.io/v1beta1","kind":"ClusterServiceBroker","metadata":{"annotations":{},"name":"gcp-broker","namespace":""},"spec":{"a...
API Version:  servicecatalog.k8s.io/v1beta1
Kind:         ClusterServiceBroker
Metadata:
  Creation Timestamp:  2018-05-17T20:04:49Z
  Finalizers:
    kubernetes-incubator/service-catalog
  Generation:        1
  Resource Version:  25
  Self Link:         /apis/servicecatalog.k8s.io/v1beta1/clusterservicebrokers/gcp-broker
  UID:               8e0bf830-5a0d-11e8-98a9-0a580a340013
Spec:
  Auth Info:
    Bearer:
      Secret Ref:
        Name:       gcp-svc-account-secret
        Namespace:  google-oauth
  Relist Behavior:  Duration
  Relist Duration:  15m0s
  Relist Requests:  0
  URL:              https://servicebroker.googleapis.com/v1beta1/projects/myproject/brokers/default
Status:
  Conditions:
    Last Transition Time:  2018-05-17T20:04:49Z
    Message:               Error fetching catalog. Error getting broker auth credentials: secrets "gcp-svc-account-secret" not found
    Reason:                ErrorFetchingCatalog
    Status:                False
    Type:                  Ready
  Reconciled Generation:   0
Events:
  Type     Reason                       Age               From                                Message
  ----     ------                       ----              ----                                -------
  Warning  ErrorGettingAuthCredentials  1m (x26 over 7m)  service-catalog-controller-manager  Error getting broker auth credentials: secrets "gcp-svc-account-secret" not found

Listing the relevant secrets:

[shosti@drogo:~]$ kubectl get secrets -n google-oauth
NAME                       TYPE                                  DATA      AGE
default-token-kvn6d        kubernetes.io/service-account-token   3         14m
google-oauth-token-svfdx   kubernetes.io/service-account-token   3         14m
oauth                      Opaque                                4         14m

(It seems oauth has a secretName of gcp-svc-account-secret--I'm assuming that that's the correct credential.)

More possibly relevant information:

[shosti@drogo:~]$ kubectl version
Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.1", GitCommit:"3a1c9449a956b6026f075fa3134ff92f7d55f812", GitTreeState:"archive", BuildDate:"1970-01-01T00:00:01Z", GoVersion:"go1.9.4", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"9+", GitVersion:"v1.9.7-gke.0", GitCommit:"1883ce4eb0e057cfc2439ebeb9822da0a9d40405", GitTreeState:"clean", BuildDate:"2018-04-19T17:08:34Z", GoVersion:"go1.9.3b4", Compiler:"gc", Platform:"linux/amd64"}

[shosti@drogo:~]$ sc version
sc version 0.1.1 linux/amd64

(sc was built from rev v1.0.0-beta.3)

kibbles-n-bytes commented 6 years ago

@shosti Could you see what the logs for the google-oauth-XXXXXXXX-XXXXX pod in the google-oauth namespace say, or if it even came up successfully? It should have generated the gcp-svc-account-secret, but apparently never did. The expected log output is:

I0502 21:06:33.452666       1 main.go:68] Secret google-oauth/oauth: checking for Service Catalog authentication extension contract...
I0502 21:06:33.538285       1 round_trippers.go:405] GET https://10.51.240.1:443/api/v1/namespaces/google-oauth/secrets?resourceVersion=137839&timeoutSeconds=481&watch=true 200 OK in 85 milliseconds
I0502 21:06:34.383033       1 round_trippers.go:405] GET https://10.51.240.1:443/api/v1/namespaces/google-oauth/secrets/gcp-svc-account-secret 200 OK in 3 milliseconds
I0502 21:06:34.388474       1 round_trippers.go:405] PUT https://10.51.240.1:443/api/v1/namespaces/google-oauth/secrets/gcp-svc-account-secret 200 OK in 4 milliseconds
I0502 21:06:34.389170       1 google_oauth.go:167] Successfully wrote an OAuth access token into secret google-oauth/gcp-svc-account-secret

shosti commented 6 years ago

:man_facepalming: that pod hadn't gotten scheduled because I ran out of space on the cluster. Sorry for bothering you with this!

googlearchive / k8s-service-catalog

`add-gcp-broker` authentication configuration fails #192