search

kubernetes-retired / service-catalog

Consume services in Kubernetes using the Open Service Broker API
https://svc-cat.io
Apache License 2.0
1.05k stars 384 forks source link

Cannot delete servicebinding #2132

Closed nabadger closed 6 years ago

nabadger commented 6 years ago

I've deployed service-catalog from Helm (catalog-0.1.21) and added the gcp-broker (all of this is on minikube).

My aim here is to provisioning/de-provision google-storage buckets.

I can do that using svcat.

I've managed to get the k8s cluster into a state where I could not use svcat to de-provision a bucket.

The key error message I get is 

Description: This request requires client support for asynchronous service operations
» svcat get instances -n test
       NAME        NAMESPACE       CLASS       PLAN                   STATUS                   
+----------------+-----------+---------------+------+-----------------------------------------+
  sc-bucket-test   test        cloud-storage   beta   DeprovisionBlockedByExistingCredentials
» svcat get binding -n test
       NAME        NAMESPACE      INSTANCE      STATUS  
+----------------+-----------+----------------+--------+
  sc-bucket-test   test        sc-bucket-test   Failed  
» svcat describe binding sc-bucket-test -n test
  Name:        sc-bucket-test                                                                                                                                                                                                                                               
  Namespace:   test                                                                                                                                                                                                                                                         
  Status:      Failed - ServiceBroker returned failure; bind operation will not be retried: Status: 422; ErrorMessage: <nil>; Description: This request requires client support for asynchronous service operations.; ResponseError: <nil> @ 2018-06-18 13:35:01 +0000 UTC  
  Secret:      sc-bucket-test                                                                                                                                                                                                                                               
  Instance:    sc-bucket-test                                                                                                                                                                                                                                               

Parameters:
  createServiceAccount: false
  roles:
  - roles/storage.objectCreator
  - roles/storage.objectViewer
  serviceAccount: sc-bucket-sa

If I try to unbind this I get:

» svcat unbind sc-bucket-test -n test -v 10 --logtostderr
I0619 08:57:18.959457    6274 loader.go:357] Config loaded from file /home/nb/.kube/config
I0619 08:57:18.961354    6274 round_trippers.go:386] curl -k -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: svcat/v0.1.21 (linux/amd64) kubernetes/ae56f95" https://192.168.99.100:8443/apis/servicecatalog.k8s.io/v1beta1/namespaces/test/serviceinstances/sc-bucket-test
I0619 08:57:19.085998    6274 round_trippers.go:405] GET https://192.168.99.100:8443/apis/servicecatalog.k8s.io/v1beta1/namespaces/test/serviceinstances/sc-bucket-test 200 OK in 124 milliseconds
I0619 08:57:19.086094    6274 round_trippers.go:411] Response Headers:
I0619 08:57:19.086154    6274 round_trippers.go:414]     Content-Length: 1622
I0619 08:57:19.086178    6274 round_trippers.go:414]     Content-Type: application/json
I0619 08:57:19.086196    6274 round_trippers.go:414]     Date: Tue, 19 Jun 2018 07:57:24 GMT
I0619 08:57:19.086365    6274 request.go:874] Response Body: {"kind":"ServiceInstance","apiVersion":"servicecatalog.k8s.io/v1beta1","metadata":{"name":"sc-bucket-test","namespace":"test","selfLink":"/apis/servicecatalog.k8s.io/v1beta1/namespaces/test/serviceinstances/sc-bucket-test","uid":"1a198ca0-72fc-11e8-aad7-0242ac110006","resourceVersion":"759","generation":3,"creationTimestamp":"2018-06-18T13:32:53Z","deletionTimestamp":"2018-06-19T07:41:47Z","deletionGracePeriodSeconds":0,"finalizers":["kubernetes-incubator/service-catalog"]},"spec":{"clusterServiceClassExternalName":"cloud-storage","clusterServicePlanExternalName":"beta","clusterServiceClassRef":{"name":"e9776b6c-4022-41ec-8b83-7c368ed9c270"},"clusterServicePlanRef":{"name":"c5670b40-4e01-48b8-8acf-d65df714146f"},"parameters":{"bucketId":"sc-bucket-test","location":"EU","storageclass":"MULTI_REGION"},"externalID":"1a198c04-72fc-11e8-aad7-0242ac110006","updateRequests":1},"status":{"conditions":[{"type":"Ready","status":"False","lastTransitionTime":"2018-06-18T13:42:52Z","reason":"DeprovisionBlockedByExistingCredentials","message":"All associated ServiceBindings must be removed before this ServiceInstance can be deleted"}],"asyncOpInProgress":false,"orphanMitigationInProgress":false,"reconciledGeneration":1,"observedGeneration":3,"externalProperties":{"clusterServicePlanExternalName":"beta","clusterServicePlanExternalID":"c5670b40-4e01-48b8-8acf-d65df714146f","parameters":{"bucketId":"sc-bucket-test","location":"EU","storageclass":"MULTI_REGION"},"parameterChecksum":"5939f1ed66b4ea0431823deae67ce08ab9b6f6be7d701dbf9ce267913c7b0866"},"provisionStatus":"Provisioned","deprovisionStatus":"Required"}}
I0619 08:57:19.094986    6274 round_trippers.go:386] curl -k -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: svcat/v0.1.21 (linux/amd64) kubernetes/ae56f95" https://192.168.99.100:8443/apis/servicecatalog.k8s.io/v1beta1/namespaces/test/servicebindings
I0619 08:57:19.098200    6274 round_trippers.go:405] GET https://192.168.99.100:8443/apis/servicecatalog.k8s.io/v1beta1/namespaces/test/servicebindings 200 OK in 3 milliseconds
I0619 08:57:19.098246    6274 round_trippers.go:411] Response Headers:
I0619 08:57:19.098263    6274 round_trippers.go:414]     Content-Type: application/json
I0619 08:57:19.098274    6274 round_trippers.go:414]     Date: Tue, 19 Jun 2018 07:57:24 GMT
I0619 08:57:19.098283    6274 round_trippers.go:414]     Content-Length: 1947
I0619 08:57:19.098497    6274 request.go:874] Response Body: {"kind":"ServiceBindingList","apiVersion":"servicecatalog.k8s.io/v1beta1","metadata":{"selfLink":"/apis/servicecatalog.k8s.io/v1beta1/namespaces/test/servicebindings","resourceVersion":"772"},"items":[{"metadata":{"name":"sc-bucket-test","namespace":"test","selfLink":"/apis/servicecatalog.k8s.io/v1beta1/namespaces/test/servicebindings/sc-bucket-test","uid":"660967b5-72fc-11e8-aad7-0242ac110006","resourceVersion":"766","generation":2,"creationTimestamp":"2018-06-18T13:35:00Z","deletionTimestamp":"2018-06-19T07:45:50Z","deletionGracePeriodSeconds":0,"finalizers":["kubernetes-incubator/service-catalog"]},"spec":{"instanceRef":{"name":"sc-bucket-test"},"parameters":{"createServiceAccount":false,"roles":["roles/storage.objectCreator","roles/storage.objectViewer"],"serviceAccount":"sc-bucket-sa"},"secretName":"sc-bucket-test","externalID":"66093876-72fc-11e8-aad7-0242ac110006"},"status":{"conditions":[{"type":"Ready","status":"Unknown","lastTransitionTime":"2018-06-18T13:36:49Z","reason":"UnbindCallFailed","message":"Error unbinding from ServiceInstance \"test/sc-bucket-test\" of ClusterServiceClass (K8S: \"e9776b6c-4022-41ec-8b83-7c368ed9c270\" ExternalName: \"cloud-storage\") at ClusterServiceBroker \"gcp-broker\": Status: 422; ErrorMessage: \u003cnil\u003e; Description: This request requires client support for asynchronous service operations.; ResponseError: \u003cnil\u003e"},{"type":"Failed","status":"True","lastTransitionTime":"2018-06-18T13:35:01Z","reason":"ServiceBindingReturnedFailure","message":"ServiceBroker returned failure; bind operation will not be retried: Status: 422; ErrorMessage: \u003cnil\u003e; Description: This request requires client support for asynchronous service operations.; ResponseError: \u003cnil\u003e"}],"asyncOpInProgress":false,"currentOperation":"Unbind","reconciledGeneration":1,"operationStartTime":"2018-06-18T13:36:48Z","orphanMitigationInProgress":false,"unbindStatus":"Required"}}]}
I0619 08:57:19.101568    6274 request.go:874] Request Body: {"kind":"DeleteOptions","apiVersion":"servicecatalog.k8s.io/v1beta1"}
I0619 08:57:19.101629    6274 round_trippers.go:386] curl -k -v -XDELETE  -H "Accept: application/json, */*" -H "Content-Type: application/json" -H "User-Agent: svcat/v0.1.21 (linux/amd64) kubernetes/ae56f95" https://192.168.99.100:8443/apis/servicecatalog.k8s.io/v1beta1/namespaces/test/servicebindings/sc-bucket-test
I0619 08:57:19.191054    6274 round_trippers.go:405] DELETE https://192.168.99.100:8443/apis/servicecatalog.k8s.io/v1beta1/namespaces/test/servicebindings/sc-bucket-test 200 OK in 89 milliseconds
I0619 08:57:19.191080    6274 round_trippers.go:411] Response Headers:
I0619 08:57:19.191091    6274 round_trippers.go:414]     Date: Tue, 19 Jun 2018 07:57:24 GMT
I0619 08:57:19.191101    6274 round_trippers.go:414]     Content-Length: 1813
I0619 08:57:19.191110    6274 round_trippers.go:414]     Content-Type: application/json
I0619 08:57:19.191151    6274 request.go:874] Response Body: {"kind":"ServiceBinding","apiVersion":"servicecatalog.k8s.io/v1beta1","metadata":{"name":"sc-bucket-test","namespace":"test","selfLink":"/apis/servicecatalog.k8s.io/v1beta1/namespaces/test/servicebindings/sc-bucket-test","uid":"660967b5-72fc-11e8-aad7-0242ac110006","resourceVersion":"773","generation":2,"creationTimestamp":"2018-06-18T13:35:00Z","deletionTimestamp":"2018-06-19T07:57:24Z","deletionGracePeriodSeconds":0,"finalizers":["kubernetes-incubator/service-catalog"]},"spec":{"instanceRef":{"name":"sc-bucket-test"},"parameters":{"createServiceAccount":false,"roles":["roles/storage.objectCreator","roles/storage.objectViewer"],"serviceAccount":"sc-bucket-sa"},"secretName":"sc-bucket-test","externalID":"66093876-72fc-11e8-aad7-0242ac110006"},"status":{"conditions":[{"type":"Ready","status":"Unknown","lastTransitionTime":"2018-06-18T13:36:49Z","reason":"UnbindCallFailed","message":"Error unbinding from ServiceInstance \"test/sc-bucket-test\" of ClusterServiceClass (K8S: \"e9776b6c-4022-41ec-8b83-7c368ed9c270\" ExternalName: \"cloud-storage\") at ClusterServiceBroker \"gcp-broker\": Status: 422; ErrorMessage: \u003cnil\u003e; Description: This request requires client support for asynchronous service operations.; ResponseError: \u003cnil\u003e"},{"type":"Failed","status":"True","lastTransitionTime":"2018-06-18T13:35:01Z","reason":"ServiceBindingReturnedFailure","message":"ServiceBroker returned failure; bind operation will not be retried: Status: 422; ErrorMessage: \u003cnil\u003e; Description: This request requires client support for asynchronous service operations.; ResponseError: \u003cnil\u003e"}],"asyncOpInProgress":false,"currentOperation":"Unbind","reconciledGeneration":1,"operationStartTime":"2018-06-18T13:36:48Z","orphanMitigationInProgress":false,"unbindStatus":"Required"}}

I'll try to reproduce this to see exactly how I get it into this state.

MHBauer commented 6 years ago

Error message is about async binding support, so here is my guess.

Async binding operations are alpha and need to be specifically enabled.

If installing with helm chart, add --set asyncBindingOperationsEnabled=true, see the helm chart.

nabadger commented 6 years ago

This could well be it.

I've since reinstalled the service-catalog via the GCP helper sc and haven't seen the same issue yet.

I can see that the controller-manager specifies the AsyncBindingOperations=true as a feature-gate when installed this way as well, so that probably is it.

Thanks - I'll close this 👍

MHBauer commented 6 years ago

Very welcome.