Generic Extensions (Actions) Proposal

[avade]

Following the discussion in https://github.com/openservicebrokerapi/servicebroker/issues/107 this issue aims to create a proposal for "Service Broker Actions".

---

Problem

As a service broker author, I want to be able to optionally offer operations beyond "provision, update, bind, unbind, delete" so that I can provide new features to users of my service via the service broker API

Examples

• Backup and restore functionality (previous discussion https://github.com/openservicebrokerapi/servicebroker/issues/107)
• Schedule things via the service broker

Workflows

Service clients can:

• Discover the actions available from a service broker
• Trigger a specific action synchronously
• Trigger a specific action asynchronously
• Pass input parameters to the action
• Discover and validate input parameters (input parameter schema)

Service authors can:

• Optionally define actions at service broker registration time

History

https://github.com/openservicebrokerapi/servicebroker/issues/107 https://docs.google.com/document/d/1Y5vwWjvaUIwHI76XU63cAS8xEOJvN69-cNoCQRqLPqU/edit#heading=h.kzfeyua6md59

Next steps

1. Proposal document
2. Review on OSBAPI WG call

[gberche-orange]

Here is a google docs placeholder. I'll try fleshing some ideas there in the coming days. Anyone feel free to start earlier if they have some bandwidth.

@avade do you have pointers to the WG calendar and dialin/minutes details ?

[avade]

@gberche-orange Perfect thanks! - @shalako can add you to the invite.

[shalako]

Sent. Guillaume, consider that dial-in info will change for next week's call. I'll be sending an update to the invite shortly.

[gberche-orange]

thanks @shalako

I've proposed a first draft into https://docs.google.com/document/d/1Dhs4RrvXxyDcmK8LdxbkHcEb_HrpiA4XqVwaIy2sTXg/edit

I've moved the sharing mode to comment so that we can systematically get notifications from suggestions. Let me know the list of people to share with edit role as to accept/reject suggestions collaboratively.

[gberche-orange]

@avade @paulcwarren any feedback on the proposal prior to the WG call today ?

[avade]

I will be taking a look this morning.

On 13 Dec 2016 8:08 a.m., "Guillaume Berche" notifications@github.com wrote:

@avade https://github.com/avade any feedback on the proposal prior to the WG call today ?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/openservicebrokerapi/servicebroker/issues/114#issuecomment-266672890, or mute the thread https://github.com/notifications/unsubscribe-auth/AAeB1yCPnl5BHANSc5_b10wihkeTSqn6ks5rHlJ1gaJpZM4LGvV5 .

[bmelville]

Is there any update on this proposal? It's something we're interested in seeing as well, so would love to see it gain traction.

B.

[avade]

Right now I am focusing on the schema / removing CFisms.

I should get a chance to get stuck into this later next week, but I this feature seems to unlock lots of interesting broker uses cases for Pivotal.

On Tue, Jan 10, 2017 at 6:30 PM, Brendan Melville notifications@github.com wrote:

Is there any update on this proposal? It's something we're interested in seeing as well, so would love to see it gain traction.

B.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/openservicebrokerapi/servicebroker/issues/114#issuecomment-271657279, or mute the thread https://github.com/notifications/unsubscribe-auth/AAeB14J9D1R97ZNgJd-EYWt7Fk1kPhwHks5rQ85JgaJpZM4LGvV5 .

[jmcarp]

I just filed #148 to ask about this more generally, but I was wondering whether this proposal might allow broker-defined actions to be executed on pending service instances.

[gberche-orange]

@jmcarp the proposal doc mentions currency control for broker actions with different approaches. Broker actions are certainly a subresource of a service instance, but we had not yet detailed how/whether consistency would be required between the two.

For instance, if a backup action is on going, can the parent service instance be updated or deleted, or similarly to service bindings should all broker actions complete before the parent service instance can be deleted.

Having more details on your use cases for broker actions would help refine the proposal, including during long-lasting service instance create/update operations.

[metayd]

LGTM Any Update? Thanks

[duglin]

general direction looks good. Couple of things

• I'd like to discuss this at the f2f in a few weeks
• in prep for that, could we get a version of the spec with the proposed edits?
• we'll probably want to include the ability to describe the parameters to the actions via schema
• we may want to consider defining an "Actions Profile" to define some common action names so that we don't get "backup" vs "back-up" - if that's a problem, not 100% sure yet.

[petereberlein]

The proposal document actually contains two options: actions as parameters in a PATCH request and actions modelled as resources. I would strongly prefer the latter one as it is more transparent and also better extensible for long running actions (delete them, get a status update, etc.)

[avade]

I would like to revisit the conversation about supporting B&R as a first class API concept. It is still Pivotals main use case for actions.

We can still progress with actions as a concept, but I would prioritise solving the B&R case first.

[petereberlein]

Addressing a concrete use case (like B&R) will help detailing the concept, but it should be generic enough to add further actions later on without the need of extending/changing this concept.

So maybe we can just use B&R as a concrete example for actions in general?

[avade]

The approach I was thinking was to put both solutions (actions and making B&R first class in the API) then we can compare and contrast.

[duglin]

We'd prefer to not overload the current APIs so the 2nd approach (dedicated resources) feels cleaner. One other bit of feedback I got was that the actions should not be linked to the plans - we'd prefer if there wasn't a tight coupling between plans and actions. Also, since the list of available actions may vary based on the state of the instance it would be nicer to support GET /v2/service_instances/:instance_id/actions instead.

Are people comfortable with making a statement about the type of actions that should be included here? Meaning, they should be limited to admin type of actions that act upon the service instance. I don't think we want people to take actions that should be invoked by the end-user via the connection info in the binding and move them to be actions performed by the platform on their behalf. I view this as the diff between action "on" the service vs actions "with" service service.

[mattmcneeney]

In the CF world, many of these actions are likely to be long-running, so we would like to support asynchronous calls, with the ability to fetch the current status of any actions and cancelling a pending action. I'd also like to see a branch (or multiple) with the proposed changes so we can discuss the proposed implementation.

[gberche-orange]

I started detailing how a backup use-case could like with the 2nd approach. I called them "snapshots" to reflect that they are triggered by application developers at the given point in time (as to avoid confusion with service operators scheduled backups intended for data loss/durability)

In terms of UX, the end users would see the following list of actions for a given data service instance (say a RDBMS service) inferred from the broker catalog response, along with associated input parameters and response format (ommited here)

• create-snapshot
• restore-snapshot
• list-snapshots
• delete-snaphot

A possible REST calls flow would then be:

Requesting a snapshot (async)
curl -X PUT /v2/service_instances/:instance_id/actions_instances/1 -d "actions=create-snapshot&parameters={name=snapshot_A}"
202 ACCEPTED

Checking snapshot completion: in progress
curl -X GET /v2/service_instances/:instance_id/actions_instances/1
200 OK
state="in progress"
description="creating snapshot  10%"

Checking snaphot completion: completed
curl -X GET /v2/service_instances/:instance_id/actions_instances/1
200 OK
state="suceeded"
description="snapshot  id 1 created successfully."
resource={name=snapshot_A, size=100MB, download-url=...}

Requesting a new snapshot:

curl -X PUT /v2/service_instances/:instance_id/actions_instances/2 -d "actions=create-snapshot&parameters={name=snapshot_B}"
202 ACCEPTED

Checking snaphot completion: in progress (stalled)
curl -X GET /v2/service_instances/:instance_id/actions_instances/2
200 OK
state="in progress"
description="creating snapshot  10%"

Cancelling a pending snaphot operation:
curl -X DELETE /v2/service_instances/:instance_id/actions_instances/2
200 OK

Requesting lists of snapshots (synchronous)

curl -X PUT /v2/service_instances/:instance_id/actions_instances/3 -d "action=list-snapshots"
201 CREATED

curl -X GET /v2/service_instances/:instance_id/actions_instances/3
state="suceeded"
description="list of snapshots successfully completed."
resource=[{name=snapshot_A, size=100MB, download-url=...} , ]

Restore a given snapshot in time:

curl -X PUT /v2/service_instances/:instance_id/actions_instances/4 -d "action=restore-snapshot&parameters={name=snapshot_A}"
202 ACCEPTED

Checking restore completion: in progress
curl -X GET /v2/service_instances/:instance_id/actions_instances/4
200 OK
state="in progress"
description="restoring snapshot  10%"

Checking restore completion: completed
curl -X GET /v2/service_instances/:instance_id/actions_instances/4
200 OK
state="suceeded"
description="snapshot  id 1 created successfully."
resource={size=100MB,download-url=...}

Delete a given snapshot
curl -X PUT /v2/service_instances/:instance_id/actions_instances/4 -d "action=delete-snapshot&parameters={name=snapshot_A}"
200 OK

I started a draft branch detailing the schema part enabling discovery of the actions: https://github.com/avade/servicebroker/compare/method-attached-schemas...orange-cloudfoundry:issue114-from-165?expand=1 including the input parameters and the response This is based on avade:method-attached-schemas in #165. I did not start the work on detailing the REST endpoints in the specs.

Pending issues:

• how long should completed action resources available for query ? should the spec describe this ? Brokers could need to implement a kind of garbage collection on completed actions.
• is there a need/use-cases for update/delete of an action resource ?

[petereberlein]

The semantics of deleting an action resource would be something like a cancel - this is already useful for the backup/restore use case. Updating is less obvious as a typical implementation would not allow updating action parameters during action execution. But who knows what people will use actions for in the future? So we might not specify update for now but reserve it for when a good use case comes up.

[mattmcneeney]

There is an ongoing conversation in the OSBAPI Slack about the state that brokers do/should keep (this has implications on the async bindings and fetching input parameters features). There is support for stateless brokers; it keeps them simple and easy to implement. If brokers were required to keep the results of many actions, perform garbage collection, etc, then this could make them much more complex to develop.

However, with async create, update and delete (optional), brokers do already have to maintain some state (remembering the last operation result).

Regarding @gberche-orange's proposal (thanks Guillaume!);

• I like the extensibility of actions and the ability to define schemas
• Is there some inconsistency with existing endpoints if the platform is specifying the action_instances ID? With the existing async calls, the broker returns an operation string to the platform which is used to check the status of the operation via the last_operation endpoint.
• To me it makes sense for PUT to create a new action, GET to fetch the status and DELETE to cancel it. I'm not sure we should handle other HTTP verbs for now until we have a strong use case.

Before we dive too deep into the implementation, please could everyone share their current use cases for actions? For CF, the key ones we want to solve asap are backup and restore. What other actions are people finding a need for brokers to support?

[gberche-orange]

Before we dive too deep into the implementation, please could everyone share their current use cases for actions?

The proposal document "problem" section details some use-cases such as:

• stop/start a service instance as to reduce resource consumption without losing data hold by the service. For data services, a stop may trigger an archival of the data to cold storage (such as AWS glacier) and a release of hot/warm computing resources associated to the service instance (storage, RAM, …), a start may fetch data from cold storage and provision computing resources.
• configure the service, such as enable/disable verbose logging.

I refined the problem section with additional details, benefits of actions and non-goals section. Feel free to make suggestions to the google doc or ask for edit permissions.

I would like to revisit the conversation about supporting B&R as a first class API concept. It is still Pivotals main use case for actions. We can still progress with actions as a concept, but I would prioritise solving the B&R case first.

It would be useful to alternative see 1st class support for snapshot/restore in OSB to compare both approaches.

Potential pros for 1st class support for snapshot/restore in OSB w.r.t. broker actions:

• could allow for more friendly user experience (e.g. display a list of available snapshots available to restore from and prompt user to select one) than what simple generic broker actions are likely to enable while keeping complexity in OSB API low.

Potential cons for 1st class support for snapshot/restore in OSB w.r.t. broker actions:

• common B&R denominator among data services is too small, and that a normative B&R be too simplistic to be effective for some services. Would need extensibility to be built-in.

[nilebox]

Any updates on this issue?

I'd like to discuss this at the f2f in a few weeks

I see the following action in the F2F notes: Define a generic “actions” mechanism - Alex L, Doug @avade @duglin do you have any updates or further details on this?

Before we dive too deep into the implementation, please could everyone share their current use cases for actions?

Our use cases are very similar to the ones already listed: we need a support of extensible set of Operations actions supported by service brokers (backup/restore or anything else service broker specific). So generic mechanism of defining actions would be useful for us.

[vaikas]

As discussed on the weekly call, here's a draft proposal for making actions be more generic. https://docs.google.com/document/d/1fVBzqi6OK-lyfjDKADSk35hZD4YUqM2AMtoLOgrDDCE/edit

[rhodie27]

As part of my AI from the F2F the attached zip contains an example swagger for generic backup & restore actions, (there are 4 formats, so pick your poison).

It not perfect, but I think it will help to continue the conversation. Things that still need refinement:

1. async representation (easier in openapi 3.0 which isn't fully supported yet, I just added bare bones)
2. security / credential management
3. restore point retention / deletion
4. scheduling (not part of this swagger, but I will upload another with a scheduler action)
5. non-success http status codes
6. better swagger formatting

Next steps:

1. review with osb team
2. incorporate feedback
3. build a sample UI from an OSB API that serves this swagger on request

bandr-swagger.zip

[shpandrak]

Hi, Regarding the backup and restore actions discussion. While restoring service data to production is an important use case. The more frequent use case is actually restoring service instance backup to a new service instance (lets call it repurpose). In most cases, even when a restore to production is needed, it is common to first repurpose the backup to a new instance in order to perform validation or having both current and restored instances side by side. Current suggestions I have seen treat backups as a sub entity limited to a specific service instance scope. Need to make sure we support the repurpose use case for the API to be useful. One option could be to allow creating a blank instance and then running restore but it is problematic since it prevents implementations to do optimizations (e.g. aws rds can restore from snapshot). Seems like backups behave as a top level entity. Anyways, the backup and restore natives/custom actions is a long expected feature - looking forward for seeing it in production 👍

[nilebox]

@rhodie27 hi Michael, any updates on the progress of this issue?

[rhodie27]

@nilebox, sure. I will have an updated swagger, a server implementation of that swagger and an example UI for actions posted in a couple of days.

[mogul]

If anyone's still looking for sample actions outside of B&R, one of the frequent requests we get is "I'd like a way to invalidate the cache please" for https://github.com/18F/cf-cdn-service-broker. (It seems like they often want to do it for a specific object, not the whole thing.)

[vaikas]

@rhodie27 have you had a chance to make progress on this?

[rhodie27]

waiting on legal @vaikas-google. Let's discuss on the call tomorrow.

[rhodie27]

Apologies for the delay, I should not have any more roadblocks going forward.

I put a demo server & ui @ https://github.com/rhodie27/osb-actions-poc to prove out the concept. Please download, build and use to see everything in action and then ask any questions.

In general, everythings looks really good from our point of view, but we can discuss in more detail at the next call and/or F2F.

The next step(s) would be to:

1. Draft the language to get discovery_url in the spec
2. Figure out how/where we want to handle/store the 'suggested' action swaggers, (like backup/restore, schedule, etc.)
3. Draft/finalize said set of suggested actions.

Hopefully we can dedicate a good chunk of time for this at the F2F.

[nilebox]

@rhodie27 thanks for the proof of concept, something concrete is much easier to digest.

We are currently working on the support for actions in brokers internally, so this proposal is relevant to us. However, we are focused on supporting custom actions as first-class citizens, supported in the same way as regular backup/restore.

Why do we need completely separate endpoints for each of the predefined actions (backup/restore, start/stop/pause)? Why can't we just make them part of the generic actions API in the similar fashion to service_instances and service_bindings? i.e. have a /service_instances/{instance_id}/service_actions/{action_id} where you specify the action type in the body similar to the way we currently specify service/plan for instance, for example.

Even if we do want to have distinct sub-URLs for backups/restores, it's probably good to still have an /service_actions/ prefix for all actions just to allow for support of other kinds of features for instances in the future. Actually, we already do have binding and last_operation endpoints on instances, i.e. service_instances/{instance_id}/service_bindings and service_instances/{instance_id}/last_operation.

Also comments on the spec:

• What about support for asynchronous execution? Are they supported for backups (there is a GET endpoint on a single backup)? For restores there is no GET endpoint, but they could also take significant time (and we should probably be able to reliably retrieve the status of restore operation even if we lost the connection). Same for start/restart/stop - how do we know what's the current state of the instance and if the operation succeeded?
• REST endpoints are not compliant with the current OSB spec, e.g.:
  • for instance provisioning: PUT is for provisioning instance (with client-side uuid provided), PATCH for updating it;
  • in actions spec, POST is used for creating backups (no client-side uuid provided, server-side id generated instead), PUT for updating backups
• What does updating a backup mean? Is it really needed? "Action" sounds like something imperative/immutable which should not support updates (I understand that there's a distinction between purely imperative action like restart and backup which produces a database dump, but still - trying to be generic)
• Listing backups does sound like something useful, will we be able to support similar semantics for custom actions?
• swagger keyword should be replaced with open-api since this is the correct name of the standard (see https://swagger.io/specification/)

Hopefully we can dedicate a good chunk of time for this at the F2F.

Unfortunately, I won't be able to participate in it in person, but would be happy to connect remotely if you can setup a Zoom/Skype/whatever call.

[nilebox]

More comments:

• in addition to above, the problem with backups is that it mixes the persistent entity (a database dump) with operation (backup creation/update/delete). It might make sense to have separate GET endpoints for them, e.g. backups/{backup_id} vs backups/{backup_id}/last_operation like we have for instances.
• maybe we even should address this issue differently, if we want to support such distinction in a generic way (make imperative actions and persistent entities completely separate), since currently we have both imperative actions (start/stop/restart, restore) and persistent backups at the same level with very different REST semantics.
• there is a resume action missing (a counterpart for pause, similar to start/stop)

[rhodie27]

@nilebox thanks for the feedback, let me see if I can address all of these comments.

Why do we need completely separate endpoints for each of the predefined actions (backup/restore, start/stop/pause)? Why can't we just make them part of the generic actions API in the similar fashion to service_instances and service_bindings? i.e. have a /service_instances/{instance_id}/service_actions/{action_id} where you specify the action type in the body similar to the way we currently specify service/plan for instance, for example.

1. In the proposal from @vaikas-google, the service provider can really do whatever they want with the returned discovery_url swagger specification. It's owned by them, but we want to provide examples of popular actions. For our examples, we can discuss this, but I think it really depends on the action in question.
2. The backup/restore and stop/start/pause/restart swaggers are just examples to prove out the concept. We need to sit down and really plan them out if we want to own these in addition to the osb spec.

What about support for asynchronous execution?

Yup, I just didn't add it yet into the example. The question I have for the group is would we want async to look like we have it in the V2 spec, or something like the callbacks feature that swagger pushes.

REST endpoints are not compliant with the current OSB spec

Understood. I took an older version of an OSB swagger spec just to get this done quicker. I'll update when we get the OSB swagger updated to the current spec version.

What does updating a backup mean? Is it really needed?

From my pov it is needed if you want to do things like move backup files around, update the expiration times, etc. Is it mandatory? No, but we can talk about this more as we formalize the example.

Listing backups does sound like something useful, will we be able to support similar semantics for custom actions?

Listing backups is mandatory, (imo), for a backup/restore usecase. Sorry, I don't understand the latter part of your question; can you expand on that?

swagger keyword should be replaced with open-api

Will fix. Thanks.

Unfortunately, I won't be able to participate in it in person, but would be happy to connect remotely if you can setup a Zoom/Skype/whatever call.

We will have zoom or something for sure. We can also setup something to talk outside of the F2F.

in addition to above, the problem with backups is that it mixes the persistent entity (a database dump) with operation (backup creation/update/delete). It might make sense to have separate GET endpoints for them, e.g. backups/{backup_id} vs backups/{backup_id}/last_operation like we have for instances.

Yeah we could probably do that. Imo, the backup resource is the action of requesting a backup, (who, what, when, where, etc). The mechanism to do the actual backup of the service instance is just a property of that request.

maybe we even should address this issue differently, if we want to support such distinction in a generic way (make imperative actions and persistent entities completely separate), since currently we have both imperative actions (start/stop/restart, restore) and persistent backups at the same level with very different REST semantics.

I think this is what we get with generic actions; the service providers will be able to provide whatever they want in their swaggers. I guess we could try to come up with a standard way to format their apis, in general.

there is a resume action missing (a counterpart for pause, similar to start/stop)

Yeah, I spent no time on the start/stop/resume/pause stuff... just provide another discovery_url as an example. As part of the next steps above I want to start making these swagger specs more 'official'.

[rhodie27]

Also, one additional thing to think about is that the actions URLs do not have to be part of the OSBAPI server; they just happen to be in the example I provided. An example of this could be an Azure service broker sending backup & restore actions to their native api.

[nilebox]

Imo, the backup resource is the action of requesting a backup, (who, what, when, where, etc). The mechanism to do the actual backup of the service instance is just a property of that request.

Agreed, but then if you want to manage database dumps, you need separate actions like create-backup and delete-backup instead of PUT and DELETE on a "backup" resource, and have GET create-backup and GET delete-backup. Then you will be able to tell who and when created and deleted database dump (or even tried to create/delete but failed). With a "backup" resource you don't have such granular history of actions, only the current state of the resource.

The question I have for the group is would we want async to look like we have it in the V2 spec, or something like the callbacks feature that swagger pushes.

The issue with the current async support in OSB is that there are no mandatory GET endpoints (for retrieving results of performed provisioning/binding). That's why in the async binding PR (https://github.com/openservicebrokerapi/servicebroker/pull/334) GET endpoints become mandatory.

[nilebox]

@rhodie27 @vaikas-google I would like to discuss an alternative proposal for generic actions API we have designed internally. The most important difference from what was proposed there is that marketplace can support standard and custom operations in exactly the same way, and both can be fully supported by CloudFoundry / Service Catalog without any special code for handling specific types.

The basic idea is that:

• OSB declares generic endpoints for operations (both synchronous and asynchronous) supported by the marketplace, let's call it "conventional API". It also declares common schemas for request/response bits (still allowing for arbitrary input/output payload). This allows the platform to understand the semantics of "action execution", "async action status polling" and so on, in a generic way.
• There is a broker endpoint for declaring a list of supported operations in the OpenAPI (Swagger) format. If the URLs of the operations in the OpenAPI spec are compatible with the "conventional API", and the schemas are based on the common schemas mentioned above, the marketplace provides support for their execution. If not, these types of operations can be still potentially "proxied" by the platform, but without understanding semantics of particular GET/POST/etc request (we can discuss if we actually want to support this extra flexibility).

See draft OpenAPI (Swagger) spec: https://swaggerhub.com/apis/Atlassian/osb-actions/0.1-alpha, it has examples of requests and responses. And PDF document describing the design in more details with examples OSB-Ops.pdf (I can transform it into markdown and publish on GitHub, if you like the idea in general).

I can also implement an example of the broker's implementation of this spec with support for backup/restore and start/stop/restart, i.e. the same "popular" operations currently being evaluated, so we can compare them with the proof-of-concept made by @rhodie27.

If there's an interest in this proposal, we can discuss it next week at the SIG call and/or at the F2F.

[rhodie27]

A couple of quick comments before I dig in further:

1. If it wasn't clear, the generic actions proposal from @vaikas-google is the google doc above with the changes to the osb api for discovery_url support. Not the swagger I created; that's just an example. The PoC I did was to see if building a dynamic (and performant) UI off of a discovered swagger doc was acceptable. I think that's proved out, so maybe you don't need to do the same here, but others might disagree.
2. Your swagger proposal seems more in line with the "actions a first class citizen" debate; maybe we need to rehash that? The problem I see is that we can't make broker authors follow the external action guidelines; we can only recommend that they do. That's why I figured offering up explicit resources made more sense. If we do this as a first class citizen your way is more appropriate.
3. The biggest thing I'm interested in is standardization of names and properties. I don't want to see one broker implement "backups" another "snapshots" and then another "mysqldumps". I think your swagger still leaves room for that.
4. Just wondering, how would you handle scheduling jobs in your proposal? E.g. I want to run a backup on a service instance every 6 hours?
5. This looks really nice. Good job.

[nilebox]

Thanks for the quick feedback!

the generic actions proposal from @vaikas-google is the google doc above with the changes to the osb api for discovery_url support

yeah, but my impression was that there is a huge gap between just discovery_url (which essentially lets platform only act as a "proxy" which can only validate the request without understanding semantics and standartization, as far as I understand) and supporting "actions as a first-class citizen". I will probably spend some time on re-reading the google doc and check if I missed something.

Your swagger proposal seems more in line with the "actions a first class citizen" debate; maybe we need to rehash that?

Yep, it is. Maybe we do need to rehash that :)

we can't make broker authors follow the external action guidelines; we can only recommend that they do

Not sure what do you mean by "external guidelines" there, and what's the problem with supporting that. It's possible wrap any specifics into the standardized API, why not? Instance provisioning is also cloud platform specific, yet we were able to define a standardized generic API for that.

I don't want to see one broker implement "backups" another "snapshots" and then another "mysqldumps". I think your swagger still leaves room for that.

Yes, you're right, this concern is not addressed in my swagger. Maybe just written guidelines with convention on the naming in existing brokers is enough? Also, I'm not sure if you can standardize properties. SQS doesn't support backups at all, and some other resources could have more specific terms like "snapshots", "replication" or whatever else rather than just "backups". Even MySQL could support "native dumps" as well as "SQL logs" or even export to XML. I think this flexibility is good. OSB services are not just merely different types of databases, they can be anything.

But I agree that it's nice to call same things with the same names, no doubt. How do you achieve that with your example or @vaikas-google 's proposal?

Just wondering, how would you handle scheduling jobs in your proposal? E.g. I want to run a backup on a service instance every 6 hours?

That's an interesting use case, I haven't thought about it. Several options come to my mind:

• "Do nothing": keep scheduling out of scope of the broker - leave it to external service to invoke platform/broker when the backup should be performed
• Define the schedule_backup action. The only problem with this approach is that it's not trivial how we'll check the history of all performed backups or a list of all available database dumps (backup results). One of the options is for schedule_backup to return a custom URL in the response, with the web UI or REST endpoint where user can go and fetch this information.

[ash2k]

Define the schedule_backup action. The only problem with this approach is that it's not trivial how we'll check the history of all performed backups or a list of all available database dumps (backup results).

It could be an action list_backups that returns a list of all existing backups. Whether they were created via an OSB API call or not does not matter.

But I agree, I'd leave this out of scope of the spec.

[nilebox]

@rhodie27 sorry for the noise, I feel dumb now since I totally misunderstood the purpose of your proof-of-concept work and completely missed the auto-generated UI part. I was confused by very different operation implementations (backup as a complex resource, restart as a simple imperative action), and had an impression that the goal is to standardize API for popular actions.

Now I understand that the purpose of this exercise is an evaluation of building UI based on arbitrary swagger spec supplied by the broker. I do think that async operation handling, as well as handling errors is a must for evaluation though.

This approach is very similar to what we've tried to achieve with our proposal. Basically, our proposal is an extension of yours, it also requires swagger spec for supported operations, with an extra (opt-out) standardization for supporting async jobs, status response and error outputs which will help us with the tooling to support these features. So yes, you're absolutely right that this proposal brings back the "actions a first class citizen" debate, but with the flexibility and opt-out.

With such standardization, CF or Service Catalog (or a separate service) will be able to not only initiate the asynchronous action (and then force user to use a separate action for manual polling), but will also be able to poll for status until the operation succeeds / fails, and display messages and errors with a nice UI or logging messages / output.

By "opt-out" I mean that if a particular action is hard to wrap into a standardized API (as you mentioned above), it can be exposed directly with an arbitrary contract. This will mean that we will need to utilize your UI for supporting such flexibility, but with less help than with the standardized API. Polling for asynchronous action can be a pain for example, with no indication of the progress in the UI, no smart error formatting and so on.

The biggest thing I'm interested in is standardization of names and properties.

I still fail to see how do you achieve that with just discovery_url approach though.

[nilebox]

The PoC I did was to see if building a dynamic (and performant) UI off of a discovered swagger doc was acceptable. I think that's proved out

Not sure if I follow, so I would like to ask for some clarification there. For completely dynamic UI there should be nothing about specific actions (backup, restore etc) in the UI, right? It should be only based on the supplied swagger spec and nothing else. However in the osb.js file I see that there is a switch-case over hardcoded supported actions, and even specific methods for modifying backups (osb.js#L274). So if I supply some random swagger, the UI won't work I suppose. Was it just a temporary quick hack, and could it be rewritten to a completely generic code?

[vaikas]

as far as naming mismatches, that's the point of adheres_to. There should be an agreed upon format for these, but imho they don't belong to OSB, they could be defined as part of another set of people.

[rhodie27]

Was it just a temporary quick hack, and could it be rewritten to a completely generic code?

Yes, quick hack, could definitely be made better; I'll wait until we actually decide on a format for all of this.

And yes, w/ regards to names & properties I was hoping something could be defined outside of the spec, but maybe in the same org. Either way, it would needed, IMO, with both of these specs. And on that note, is this something we should discuss in detail on the call on Tuesday or should we setup a separate / more specific call before the F2F?

[vaikas]

Yeah, in my head and I thought we talked about this in the last f2f, the adheres_to would be a pointer to a spec defining the semantics of the supported actions. There's no need to have one, but defining use case specific specs (for example backup) seems like it's best left to folks who are subject matter experts and the OSB spec should make it possible to support various use cases but not be involved in defining those semantics. By utilizing adheres_to there should be no issues with the naming conflicts or differing interpretations of what actions mean. If I understand it correctly, it seems like the one issue being described here is the calling semantics of actions and in Nail proposal this is the generic invocation mechanism for asynchronous jobs and being able to poll them for status as well as canceling. We chatted about this a bit yesterday offline and I do think it would be beneficial for us to agree on the goals and design parameters before heading down the path too far. I had made a design decision that I think might be at odds here:

• Broker should not be necessarily the one implementing the actions. Hence there's no additional endpoints required on the broker. Thinking here was that a broker could implement management functions but would allow for offloading these

On Wed, Nov 15, 2017 at 7:57 PM, Michael Rhodes notifications@github.com wrote:

Was it just a temporary quick hack, and could it be rewritten to a completely generic code?

Yes, quick hack. It should work with any similar swagger files, regardless of the backup/restore function names.

And yes, w/ regards to names & properties I was hoping something could be defined outside of the spec, but maybe in the same org. Either way, it would needed to be in, IMO, with both of these specs. And on that note, is this something we should discuss in detail on the call on Tuesday or should we setup a separate / more specific call before the F2F?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/openservicebrokerapi/servicebroker/issues/114#issuecomment-344808812, or mute the thread https://github.com/notifications/unsubscribe-auth/AKwedGpgYbQ7tSs-xMWrdErDVblnEaqNks5s27KcgaJpZM4LGvV5 .

[vaikas]

Yeah, in my head and I thought we talked about this in the last f2f, the adheres_to would be a pointer to a spec defining the semantics of the supported actions. There's no need to have one, but defining use case specific specs (for example backup) seems like it's best left to folks who are subject matter experts and the OSB spec should make it possible to support various use cases but not be involved in defining those semantics. By utilizing adheres_to there should be no issues with the naming conflicts or differing interpretations of what actions mean. If I understand it correctly, it seems like the one issue being described here is the calling semantics of actions and in Nail proposal this is the generic invocation mechanism for asynchronous jobs and being able to poll them for status as well as canceling. We chatted about this a bit yesterday offline and I do think it would be beneficial for us to agree on the goals and design parameters before heading down the path too far.

On Wed, Nov 15, 2017 at 7:57 PM, Michael Rhodes notifications@github.com wrote:

Was it just a temporary quick hack, and could it be rewritten to a completely generic code?

Yes, quick hack. It should work with any similar swagger files, regardless of the backup/restore function names.

And yes, w/ regards to names & properties I was hoping something could be defined outside of the spec, but maybe in the same org. Either way, it would needed to be in, IMO, with both of these specs. And on that note, is this something we should discuss in detail on the call on Tuesday or should we setup a separate / more specific call before the F2F?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/openservicebrokerapi/servicebroker/issues/114#issuecomment-344808812, or mute the thread https://github.com/notifications/unsubscribe-auth/AKwedGpgYbQ7tSs-xMWrdErDVblnEaqNks5s27KcgaJpZM4LGvV5 .

[rhodie27]

Let's deep-dive into the topic on Tuesdays call and if we need more conversation maybe we can do it prior to the F2F.

[vaikas]

SGTM.

On Thu, Nov 16, 2017 at 10:30 AM, Michael Rhodes notifications@github.com wrote:

Let's deep-dive into the topic on Tuesdays call and if we need more conversation maybe we can do it prior to the F2F.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/openservicebrokerapi/servicebroker/issues/114#issuecomment-345014986, or mute the thread https://github.com/notifications/unsubscribe-auth/AKwedGcdQ3OKyhpdls_n30RtVcjTOTDwks5s3H9UgaJpZM4LGvV5 .

[shashankmjain]

I feel, the OSB Actions API should also allow creation of constructs needed for services like (LoadBalancer Endpoints, Security Groups,Network Policies). Currently they are achieved as part of the service provisioning calls, but I feel the concepts like Security Groups, Service endpoints should be decoupled from the service provisioning itself.