Provide cf create-services command with manifest and vars-file

[TheFonz2017]

What's the user value of this feature request? CF users often need to create a set of service instances, before they push their applications referencing the services. Today, CF users need to create shell scripts to do so, or install custom CLI plugins which do or not do the job well.

With the proposed solution, users can

• describe the service instances their application(s) need(s) in a single services-manifest.yml file,
• describe them with variables that will be substituted from a service-manifest-variables.yml file by the CLI plugin at deploy time and
• do not have write shell scripts for the creation of service instances anymore.

Since the same variables file format and variable substitution syntax is used as for application manifest.yml files (used by cf push) the same variables file (and thus variable values) that is used to substitute variables in an application manifest.yml can be used to deploy / create the service instances.

This will also help in dev-ops scenarios, where service instances might have to be named differently and there should be a uniform mechanism of managing automated service instance creation.

Who is the functionality for?

• Application and service developers
• Dev Ops personnel

How often will this functionality be used by the user? Often, as automated service instance creation for a variety of services is a pre-requisite for even small-scale Cloud applications.

Who else is affected by the change? None that we can think of.

Is your feature request related to a problem? Please describe. The problem we are facing is the following: there is a clean and consistent standard way of pushing and configuring multiple applications using a manifest.yml and vars-file but there is no standard automated and way of describing and creating several service instances at once (and using a vars-file). There exists a 3rd party CLI plugin (CreateServicesPush), but this does not cleanly separate service creation from pushing. Apart from separating bulk service creation and bulk application push into separate commands, there should be a standard way of doing this without the need for 3rd party plugins.

Describe the solution you'd like We would like to see a new command in the CLI, cf create-services that can be called like this:

• cf create-services - reads a file called services-manifest.yml and creates the service instances described therein.

• cf create-services -f another-services-manifest.yml - reads the specified file and creates the service instances described therein.

• cf create-services --vars-file manifest-variables.yml - reads a file called services-manifest.yml containing variable keys (specified in standard CF ((varkey))-syntax), substitutes them with their values as specified in the given vars-file and creates the service instances.

The services-manifest.yml file should have a structure similar to this one:

---
create-services:
# Database Service.
- name:   "((db-prefix))-database-container"
  broker: "postgres"
  plan:   "lite"
  parameters: > {
     # JSON-based parameters, usually passed with -c option to cf create-service
  }

# Message Bus Service
- name:   "((bus-prefix))-database-container"
  broker: "rabbitMq"
  plan:   "enterprise"
  tags: ["t1", "t2", "t3"]

The variable substitution file should have the same format and syntax as is already used for cf push. As you can see, parameters and tags should be supported as well.

If service creation is asynchronous, there should be an option to --wait until all service instances have been created.

Describe alternatives you've considered Alternatives are to install a 3rd party plugin, which does not support vars-files and does not cleanly separate the service instance creation from the application push. Another alternative is write shell scripts that call cf create-service multiple times. However, these scripts are platform dependent and can be written in a variety of ways. For dev ops purposes a standardized, clean and concise approach is required that does not readily exist today.

Additional context None.

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/160753443

The labels on this github issue will be updated when the story is started.

[abbyachau]

/cc @mattmcneeney @williammartin

[mattmcneeney]

Thanks @TheFonz2017 for opening this issue. This is something we have been asked about before, and your input is appreciated.

Since some service brokers only support synchronous operations, and some only support asynchronous operations, this won't be a trivial feature for us to implement.

I have one question: are you also interested in defining service bindings in a manifest file too?

[TheFonz2017]

Thanks for your answer @mattmcneeney!

To your question, not sure I got your point, but I think the service bindings should be specified in the standard cf push manifest.yml as is already the case.

The services-manifest.yml is solely intended for the specification of the service instances that should be created and by which name, plan and tags. It is crucial to be able to use placeholder variables and the same placeholders could be used inside of a standard (cf push) manifest.yml. Having a vars-file.yml that resolves the placeholders to actual values, that same vars-file could then be passed to the cf create-services and the cf push command.

To your point about the different service-broker behaviours - what if we let the users who write the services-manifest.yml decide that. They should know best, about the service brokers they use. The service-manifest.yml could get another (optional) property:

---
create-services:
# Database Service.
- name:   "((db-prefix))-database-container"
  broker:  "postgres"
  plan:     "lite"
  wait:      true     # specified by user for asynchronous service brokers
  parameters: > {
     # JSON-based parameters, usually passed with -c option to cf create-service
  }

Notice the "wait" property that can be given for asynchronous service brokers (that would replace the --wait for cf create-services). It would be an error to specify this property for synchronous service brokers.

What are your thoughts on that solution? That should simplify it, I hope.

[mattmcneeney]

Thanks @TheFonz2017. I'm not sure the user we're talking about here would necessarily know if a service broker supports asynchronous operations or not. As per the Open Service Broker API spec, CC can give the service broker the choice to perform an operation asynchronously, but it is fundamentally up to the service broker.

You're right that you can specify service bindings today in application manifests, but that only works for brokers that perform the bind operation synchronously. No logic exists for handling asynchronous binding operations today. When we looked at adding support for that, we estimated it to be a substantial amount of work, and adding support for asynchronous provisioning (which is more common than asynchronous binding) would likely also be substantial.

Regardless, we will certainly look into this again and see what we can do!

[TheFonz2017]

Thank you @mattmcneeney, that's good to know. By "the user" I meant the person writing the services-manifest.yml. If I am the developer of an application that requires a certain set of service instances, I should usually know how the service brokers behind these instances behave. Therefore I should know if a broker behaves asynchronously and that I need to add a wait: true to the service instance definition in services-manifest.yml.

Think of my proposal as of an easier way to do several cf create-service calls being able to use variable substitution. Instead of having to write a (platform-specific) script that calls one cf create-service at a time and does not easily support variable substitution, the approach above could be a simple (for the beginning) standard solution . I guess, internally, you could simply turn it into a bunch of cf create-service calls yourself.

Just like with manifest.yml and it not being able to support asynchronous bindings, the proposal above does not necessarily handle all the possible cases. The majority of cases would be a good start. :) Being able to use manifest.yml files is super helpful and a great asset to have. Even if it is (according to your words) not perfect yet.

Anyway, your consideration is much appreciated! Cheers!

[mattmcneeney]

If I am the developer of an application that requires a certain set of service instances, I should usually know how the service brokers behind these instances behave.

I'm not sure this is always the case. Many development teams use service instances from independent software vendors, and I don't think they would know whether or not those service brokers are going to behave asynchronously or not. I think whatever solution we could provide for this should handle the case where developers are not sure what will happen. This is really an implementation detail which I think we should hide from them.

Thanks for the further info. We will keep you updated on this!

[TheFonz2017]

Thanks @mattmcneeney.

As a final comment (and then I will really shut up ;)) just consider the development process. Even if development teams use third party services, they will (hopefully) sooner or later test against them and then code against them. They will (and should) understand how the service (broker) works they are making use of, and whether it behaves asynchronously or synchronously. They will then be able to write the services-manifest.yml file accordingly.

The thing that could be an issue, is if the 3rd party service broker changes its behaviour (e.g. from synchronous to asynchronous). But if that happens it is a change in API, i.e. a change of the service broker's interface. This should never happen as it would break backwards compatibility. If it happens anyway it would be a mistake of the 3rd party service broker. Catering for that from a (CF) platform point of view is virtually impossible, and maybe should not even be tried.

Cheers!

[TheFonz2017]

One more comment, I just tested the 3rd Party Plugin Create-Services-Push. They support asynchronous and synchronous broker behaviour transparently. May be worth having a look at.

[macleanpinto]

@mattmcneeney, Is this feature out yet?

[mattmcneeney]

Sorry @macleanpinto - I'm no longer working on this. I'd suggest reaching out to someone else in the CLI team.

[FelisiaM]

Hi @macleanpinto

This feature is not planned or implemented yet.

Currently, we are focused on making sure the next major version of the CLI communicates to CAPI v3 for services commands. Given that and the fact that there is a plugin available, I don't think we will be able to implement this any time soon.

Thanks, Felisia

[github-actions[bot]]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed.