capi 1.93.0 v2 api breaking change. orphan mitigation triggered on async provisionning failure

gberche-orange commented 4 years ago

Issue

As a service broker author

in order to ensure compatibility when upgrading capi release 1.97.0
I need stable APIs that avoid breaking change

As a service operator

in order to inspect failed async service provisonning reported by users
I need CC to not trigger orphan mitigation when async provisionning fails

Context

While OSB specifications provide general guidance on when orphan mitigation triggers at https://github.com/openservicebrokerapi/servicebroker/blob/master/spec.md#orphan-mitigation it adds

Note that the Platform MAY allow end users to determine when orphan mitigation occurs, in order to allow them to investigate the cause of any failures.

In capi releases 1.192.0 and older, the orphan mitigation was applying behavior documented at https://docs.google.com/document/d/11iXxAciCIQpCvrnzmGoEqQIbIVxpn6VDYlm_SVuq9TU/edit

Steps to Reproduce

deploy overview broker from https://github.com/mattmcneeney/overview-broker
edit catalog to leave only small plan without associated schema
configure broker with async failure response

cf cs overview-service small gberche3
Creating service instance gberche3 in org osb-cmdb-services-acceptance-tests / space development as gberche...
OK

Create in progress. Use 'cf services' or 'cf service gberche3' to check operation status.
guillaume@guillaume-dev-box:~/code/sec-group-broker-filter$ (master) cf service gberche3
Showing info of service gberche3 in org osb-cmdb-services-acceptance-tests / space development as gberche...

name:             gberche3
service:          overview-service
tags:             
plan:             small
description:      Provides an overview of any service instances and bindings that have been created by a platform.
documentation:    
dashboard:        https://overview-broker.redacted-domain.org/dashboard?time=2020-09-14T15:06:12.514Z
service broker:   overview-broker

Showing status of last operation from service gberche3...

status:    create failed
message:   Operation failed
started:   2020-09-14T15:06:12Z
updated:   2020-09-14T15:06:16Z

There are no bound apps for this service.

There is no upgrade available for this service.

Expected result

No orphan mitigation triggered (i.e. no DELETE sent to broker on async provisionning failure)

Current result

Broker receives DELETE request in absence of user cf delete-service request

[1] Mon Sep 14 2020 15:06:16 GMT+0000

{
  "url": "/v2/service_instances/79a18e0e-84a7-4b62-872b-044ff9c34b08?accepts_incomplete=true&plan_id=f5d9da0f-425d-476b-904e-fc0a347567fb&service_id=3a887103-0359-476e-97c9-37f4564344d5",
  "method": "DELETE",
  "body": {},
  "headers": {
    "host": "overview-broker.redacted-domain.org",
    "user-agent": "HTTPClient/1.0 (2.8.3, ruby 2.5.5 (2019-03-15))",
    "accept": "application/json",
[...]
]  }
}

[2] Mon Sep 14 2020 15:06:16 GMT+0000

{
  "url": "/v2/service_instances/79a18e0e-84a7-4b62-872b-044ff9c34b08/last_operation?plan_id=f5d9da0f-425d-476b-904e-fc0a347567fb&service_id=3a887103-0359-476e-97c9-37f4564344d5",
  "method": "GET",
  "body": {},
  "headers": {
    "host": "overview-broker.redacted-domain.org",
    "user-agent": "HTTPClient/1.0 (2.8.3, ruby 2.5.5 (2019-03-15))",
    "accept": "application/json",
[...]
  }
}

[3] Mon Sep 14 2020 15:06:12 GMT+0000

{
  "url": "/v2/service_instances/79a18e0e-84a7-4b62-872b-044ff9c34b08?accepts_incomplete=true",
  "method": "PUT",
  "body": {
    "service_id": "3a887103-0359-476e-97c9-37f4564344d5",
    "plan_id": "f5d9da0f-425d-476b-904e-fc0a347567fb",
    "organization_guid": "14af188e-b07f-4041-9488-d97bacfcb49c",
    "space_guid": "ff120500-0550-4ce5-abb6-7ad52243c70c",
    "context": {
      "platform": "cloudfoundry",
      "organization_guid": "14af188e-b07f-4041-9488-d97bacfcb49c",
      "space_guid": "ff120500-0550-4ce5-abb6-7ad52243c70c",
      "organization_name": "osb-cmdb-services-acceptance-tests",
      "space_name": "development",
      "instance_name": "gberche3"
    },
    "maintenance_info": {
      "version": "1.0.0"
    }
  },
  "headers": {
    "host": "overview-broker.redacted-domain.org",
    "user-agent": "HTTPClient/1.0 (2.8.3, ruby 2.5.5 (2019-03-15))",
    "content-length": "585",
    "accept": "application/json",
[...]
  }
}

Possible Fix

Restore previous behavior and not trigger OM on failed deprovisionning.

cf-gitbot commented 4 years ago

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/174802358

The labels on this github issue will be updated when the story is started.

gberche-orange commented 4 years ago

There are additionally the following issues with the behavior observed in capi 1.197.0

While orphan migitation seems triggered as a result of a failed async provisionning, CC does not poll the completion of the async deletion. Brokers than rely on the CC to poll their delete completion and receive the operation field face the risk of leaking underlying broker resources (such as vms in the pivotal ODB broker).
when the user asks for a cf delete-service, then a second DELETE request is sent to the broker.

pivotal-marcela-campo commented 4 years ago

Hi @gberche-orange, we will remove the orphan mitigation step for this scenario. We are working on the change. Thank you for raising this with us.

kirederik commented 4 years ago

This issue is now fixed and will be included in the next capi-release. We'll close the issue once it's released

kirederik commented 4 years ago

CAPI 1.99.0 is out 🚢

Closing this issue as it should be fixed. Please reopen if that's not the case!

cloudfoundry / cloud_controller_ng