cannot update the existing serviceinstance

[lizzzcai]

Hi colleagues,

I have an xsuaa instance created previously, but somehow an error happens in my service instance CR. So I delete my service instance CR and redeploy it. but it is not working. Now I cannot delete this xsuaa instance.

below is the error in btp:

Failed deprovisioning request instanceID: 87288db5-ee08-42ea-934b-d721b51ef9ad, planID: HWEgt9/jJGFjzT8j+/x9pGivrAuKbUMUK8PgWEx3cLY=, serviceID: xsuaa, acceptsIncomplete: true: Status: 500; ErrorMessage: <nil>; Description: Deleting service instance 87288db5-ee08-42ea-934b-d721b51ef9ad (appid aisvc-32094e64-myaifunction!b32291) failed. There still exist clone instances for this master instance ; ResponseError: <nil>

below is the error from my service instance:

  - lastTransitionTime: "2022-03-31T10:05:12Z"
    message: |
      ServiceInstance create failed: request POST https://service-manager.cfapps.sap.hana.ondemand.com/v1/service_instances failed: StatusCode: 502 Body: {"error":"BrokerError","description":"Failed provisioning request context: map[clusterid:96317595-5d49-4286-a940-22d2ebbf9955 crm_customer_id: global_account_id:8e83e691-b5af-5ef5-09e5-3430c50fde01 instance_name:aisvc-32094e64-myaifunction-instance license_type:SAPDEV namespace:mtaicorelizecai-a35742cb origin:kubernetes platform:sapcp region:cf-eu10-canary service_instance_id:dc916858-2576-4ab0-9136-7d788298cd9b signature:vqBlDX09AGhJt+rbJDlDYUeAebrheafqZqa16la0ypgaH3FqYJzidSSg2lbmtDo/c7v99FV5AsCLivGtekSQSvC0m6dJv2UNNgckPlclYjW0CJ3J6qlPUUNQ8pC7n7JTqQ/jqK3ZpQmT7oRx4QNtGjWHX8aVIW6i81yW39jkvedDj97BYMdsYAMNs7uRmhNww/tbSYbIT8BsoDEQr6dXFuSLP7/XcqmDunkeAfHhZrsNC+fD4yAAMg1brb1++h46duJpSk4U6Q1TKLgRqSIKuz3EFG9HUpBNlDEubL+9Fuq7c+klIi6cRZ9I8L9dluU5ddZT25DjlcThd/QPVDN1cQOSIwxcAFxr2QIb6O5ojZbmO4twMZZUMNTYNns0stiZD03yS9EWfnlopo1QrB+2EPvGJqR9f5S2Ovrl9xs2vf8R+VnyK17oirMyUX9Sq1w/NNrDvyuoizvhR4H11rk2ZfCPa4Lhdvnl0BbWBVh4ScU2Ac3ie51EaqWYMuohM7VEmsV+kCbiRsA/8Aid1m2v5InAS1Da4W+VqTG/LFXE7sakKsgPNxF1Fd4ehPMEyT4yv4MZKdJ1yFrYCqqeXVU1+GBG9U8kXLPeohOlehx6Smvk+yiV+jVReiMcB0YTHLPwf7vNnfbHKlC9GyIJEC3oOnyXqgpWntkHIa5+AkDVpmw= subaccount_id:80365308-5046-4afd-845b-a0ec9197e16c subdomain:devsg zone_id:3f0027d5-ec9f-4e45-9756-d29887810cfa], instanceID: dc916858-2576-4ab0-9136-7d788298cd9b, planID: HWEgt9/jJGFjzT8j+/x9pGivrAuKbUMUK8PgWEx3cLY=, serviceID: xsuaa, acceptsIncomplete: true: Status: 500; ErrorMessage: \u003cnil\u003e; Description: org.springframework.cloud.servicebroker.exception.ServiceBrokerException: Service instance with xsappname aisvc-32094e64-myaifunction already exists. Please remove or update it instead.; ResponseError: \u003cnil\u003e"}
    observedGeneration: 1
    reason: CreateFailed

[pavelmaliy]

Hi, How you deleted the CR if you had an error?

[lizzzcai]

Hi, How you deleted the CR if you had an error?

Hi @pavelmaliy , I found the xsuaa instance in the btp is showing deletion failed. then I check my service instance CR which shows above error. I can delete the service binding and CR without blocking. Not sure if you can check it from the backend, now the xsuaa instance still exist in my btp with Deletion Failed status.

[pavelmaliy]

Hi, I am not sure exactly what is the error, but it seems not related to BTP Operator but provision/deprovision issue of xsuaa service. please open internal ticket for that. Thanks

[lizzzcai]

Hi @pavelmaliy , just want to check is it an expected behaviour in btp operator service instance before opening new issue,

When I try to delete an xsuaa ServiceInstance (which has existing clone) created by BTP Operator, I got follow errors,

  status:
    conditions:
    - lastTransitionTime: "2022-04-01T06:29:23Z"
      message: |
        ServiceInstance deletion failed: request DELETE https://service-manager.cfapps.sap.hana.ondemand.com/v1/service_instances/3e3f10d2-3e74-4c0f-93d9-45f757df9dd7 failed: StatusCode: 502 Body: {"error":"BrokerError","description":"Failed deprovisioning request instanceID: 3e3f10d2-3e74-4c0f-93d9-45f757df9dd7, planID: HWEgt9/jJGFjzT8j+/x9pGivrAuKbUMUK8PgWEx3cLY=, serviceID: xsuaa, acceptsIncomplete: true: Status: 500; ErrorMessage: \u003cnil\u003e; Description: Deleting service instance 3e3f10d2-3e74-4c0f-93d9-45f757df9dd7 (appid aisvc-b3df410f-myaifunction-demo!b32291) failed. There still exist clone instances for this master instance ; ResponseError: \u003cnil\u003e"}
      observedGeneration: 2
      reason: DeleteFailed
      status: "False"
      type: Succeeded
    - lastTransitionTime: "2022-04-01T05:28:27Z"
      message: ""
      reason: Provisioned
      status: "True"
      type: Ready
    - lastTransitionTime: "2022-04-01T06:29:23Z"
      message: |
        ServiceInstance deletion failed: request DELETE https://service-manager.cfapps.sap.hana.ondemand.com/v1/service_instances/3e3f10d2-3e74-4c0f-93d9-45f757df9dd7 failed: StatusCode: 502 Body: {"error":"BrokerError","description":"Failed deprovisioning request instanceID: 3e3f10d2-3e74-4c0f-93d9-45f757df9dd7, planID: HWEgt9/jJGFjzT8j+/x9pGivrAuKbUMUK8PgWEx3cLY=, serviceID: xsuaa, acceptsIncomplete: true: Status: 500; ErrorMessage: \u003cnil\u003e; Description: Deleting service instance 3e3f10d2-3e74-4c0f-93d9-45f757df9dd7 (appid aisvc-b3df410f-myaifunction-demo!b32291) failed. There still exist clone instances for this master instance ; ResponseError: \u003cnil\u003e"}
      observedGeneration: 2
      reason: DeleteFailed
      status: "True"
      type: Failed
    instanceID: 3e3f10d2-3e74-4c0f-93d9-45f757df9dd7
    observedGeneration: 2
    ready: "True"
    tags:
    - xsuaa

and seems like this DeleteFailed status cannot be recovered back. And cannot create new service binding as well. this is the error from service binding:

  status:
    conditions:
    - lastTransitionTime: "2022-04-01T07:06:24Z"
      message: service instance 'aisvc-b3df410f-myaifunction-demo-instance' is not
        usable
      observedGeneration: 1
      reason: Blocked
      status: "False"
      type: Succeeded
    - lastTransitionTime: "2022-04-01T07:06:24Z"
      message: ""
      reason: NotProvisioned
      status: "False"
      type: Ready
    observedGeneration: 1
    ready: "False"

Expected behaviour is the DeleteFailed status will be recovered back after some time so that the xsuaa service instance will work as normal.

[pavelmaliy]

Hi, When instance in DeleteFailed state operator will constantly retry to delete the instance. And yes it's expected behavior that when you deleted the instance it considered deleted and not usable even though the instance still exists and has finalizers. Your option is to create binding using smctl (login with BTP operator credentials) create binding and delete the clones, but again it's service specifics.

[lizzzcai]

Hi @pavelmaliy , is there a way to remove the delete state of a service instance? (I tried editing the serviceinstance to remove the deletion timestamp but not working) as in this state, you cannot create any new binding. Issue we are facing: https://github.com/SAP/sap-btp-service-operator/issues/171#issuecomment-1144940381 (this feature is probably helpful as well)

And manually using the smctl with the btp operator credentials seems like cannot create the service binding. I got following error:

❯ smctl bind aisvc-226fb3f9-mango-service-instance aisvc-226fb3f9-mango-service-binding
Error: request POST https://service-manager.cfapps.sap.hana.ondemand.com/v1/service_bindings?async=true failed: StatusCode: 400 Body: {"error":"Bad Request","description":"sapcp operator must provide _clusterid, _namespace and _k8sname labels"}

I looked at the the way that btp operator creates the binding, the parameters are very hard to constructed manually.