OSB service instance update support (metadata, plan, params)

[gberche-orange]

Requirements

version update

As a service operator,

• in order to ensure consistency among stemcell, release and bosh deployment manifest updates
• I need to trigger (or control when) a refresh of service instances deployments against deployment model

As a service operator,

• in order to avoid rolling out failure to all service instances upon a model update
• in order to spread load when many service instances need update
• I need to trigger (or control when) a refresh of service instances deployments against deployment model

As a service user,

• in order to control window at which my dedicated instance will undergo a version update (e.g. for controlling performance or downtimes triggered by updates),
• I need to be notified when my service requires update
• And I need to be able to trigger this update using CF cli support in v6.46.0, see #55

Service authors that have built services using the On-Demand Broker want to allow Service Instances to be upgraded individually after a new version of their Service Broker has been deployed.

Now cf services will inform users that an upgrade is available; running cf service provides additional details. To upgrade a service instance, run cf update-service --upgrade. This flag is in experimental stage and may change without notice.

plan and param update

As a service user, I need to be able to update plans and params provisionned in an provisionning request (see #29)

As a service author, in order to handle user update param request, I may need be notified that an update was requested (in order to handle differently an update from an initial provisionning).

As a service author, I may need to have be given previous plan, params, and potentially context (such as service instance name) in order to trigger focussed update on the underlying database upon detecting changes

Example of coab-vars.yml enriched with previous_values

---
epoq:<epoq>.<timestamp> #epoq
deployment_name: "c_aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaa0"
instance_id: "aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaa0"
service_id: "service_definition_id"
plan_id: "plan_guid" # updated value
context:
  platform: "cloudfoundry"
  user_guid: "user_guid1"
  space_guid: "space_guid1"
  organization_guid: "org_guid1"
  name: "updated service instance name"
parameters:
  cacheRatio: 0.9 # updated value
  cacheSizeMb: 20 # updated value
  slowQuery: true # updated value
previous_values:
  plan_id: "previous_plan_guid"

[gberche-orange]

Note: middleware update has been moved to #245

[gberche-orange]

Initial design outline (moved here from the issue body)

Design outline

• Store the desired state into coab-vars.yml (paas-templates), including new params, plans and a a version number (called epoq)
  • Regenerate and overwrite service instances, and push to git
  • watch for the actual state into the manifest.yml produced by COA (in paas-secrets) during update to confirm update completion, ie when actual epoq == desired epoq
  • Additionally, the epoq can be used to perform concurrency control

Steps

• Implement PATCH /v2/service_instances/:instance_id
  • Apply concurrency validation (moved to #220 )
    • Reject request if there is an ongoing update request pending
    • Reject concurrent update requests (OSB states it's broker responsibility to detect concurrency and reject concurrent requests)
  • (Apply input validation: Accept update request without param or plan change (likely to be service instance name update https://github.com/openservicebrokerapi/servicebroker/pull/544) )
  • Override all COAB generated files (i.e. symlinks to model files) including coab-vars.yml file (introduced in #29) containing
    • desired state
    • previous state (obtained from loading in memory content of coab-vars.yml on disk)
  • git push
• Refine GET /v2/service_instances/:instance_id/last_operation endpoint
  • Return status code reflecting whether the update is in progress.
    • Compare D=desired and A=actual state epoq to either return 200 status=succeeded/in progress/failed:
      • If A=D return 200, status=succeded (as per spec)
      • If A<D return 200, status="in progress"
      • If A>D return 200, status="failed", (CC will not trigger orphan mitigation)
• Refine DELETE v2/service_instances/:instance_id endpoint:
  • Reject update halting requests, https://github.com/openservicebrokerapi/servicebroker/pull/581 (COAB has not easy way to stop/cancell a COA concourse job and revert to previous state)

Details of state storage and update:

• Actual state (epoq number) is available in the manifest.yml updated by COA upon successfull deployment.
• Located into bosh manifest global properties section (even if "Deprecated in favor of job level properties and links.")
  • COAB inserts its own bosh operator "insert-coab-epoq-operators.yml" to modify /properties/coab-state/epoq key with value containing epoq integer
• epoq is stored in format so that concurrent git push would conflict preventing concurrent updates.

[gberche-orange]

Current design

• Store the desired state into coab-vars.yml (paas-templates)
  • Regenerate and overwrite service instances, and push to git
  • watch for the actual state (coab_completion_marker ) into the manifest.yml produced by COA (in paas-secrets) during update to confirm update completion, ie when coab-vars == coab_completion_marker

Steps

• [x] Implement PATCH /v2/service_instances/:instance_id
  • [x] Apply input validation (as a workaround for lack of OSB support, see https://github.com/openservicebrokerapi/servicebroker/issues/217) : Accept update request without param or plan change (likely to be service instance name update https://github.com/openservicebrokerapi/servicebroker/pull/544)
    • [x] Provide a way to specify synchronously rejected plan upgrades (usually scale ins), such as with
      • rejected_upgrades_from_to: Map<String: from, Array<String>: to>
  • [x] Override coab-vars.yml file (introduced in #29) containing
    • desired state
    • previous state
      • [x] obtained from OSB request previousValue field: containing initial plan_id and maintenance_info
      • [ ] obtained from loading in memory content of coab-vars.yml on disk: containing initial params and potentially content
  • git push
• [x] Refine GET /v2/service_instances/:instance_id/last_operation endpoint
  • Return status code reflecting whether the update is in progress.
    • Compare coab-vars.ymland coab_completion_marker to either return 200 status=succeeded/in progress/failed (timeout):

[gberche-orange]

Update following exchange with @psycofdj : publicly available Cf-Wall is being deprecated in favor of a closed source cftools suite of microservices designed to integrate CF with Orange IT (CMDB, email alerting, etc). Expressed rationales for deprecating cf-wall:

• set up a cache when querying CF APIs and share this cache among Cf API clients
• support "CF cluster per AZ" pattern instead/in-addition of usual "Multi-az CF cluster"