How to apply structuring to configuration

[marcusholl]

We have use cases where several properties denotes one entity, lets e.g. for interacting with a backend.

In this case we have the following options for modelling these properties.

• use flat properties containing a namespace in the key.

  myBackendEndpointURL: 'https://example.org/myBackend'
  myBackendCredentialsID: 'theCreds'
  myBackendOpts: '-Djavax.net.ssl.trustStore:/my/trust/store'

• use nesting for structuring the properties

  myBackend:
    endpointUrl: 'https://example.org/myBackend'
    credentialsId: 'theCreds'
    opts: '-Djavax.net.ssl.trustStore:/my/trust/store'    

• <space for more options>

We should collect some pros and cons in order to make a decission which approach to follow.

[daniel-kurzynski]

In our discussion we basically came up the following two approaches. For the team internal discussion I tried to illustrate it using an example with two steps: cloudFoundryDeploy to deploy to cloud foundry and cloudPlatformDeploy which deploys either to neo or cloud foundry as an example for steps using a mixture of environments.

Name-spacing using Maps

Example Configuration

general:
  cloudFoundry:
    apiUrl: api.sap.com

steps:
  cloudFoundryDeploy:
    apiUrl: trial.api.sap.com 
    credentialsId: cf-cloud

Example Usage

For steps which are specific for a certain environment, no name-spacing is needed:

cloudFoundryDeploy(script:script, apiUrl: 'test.api.sap.com')

As an alternative, we could also provide the environment specific parameters as a map:

cloudFoundryDeploy(script: script, cloudFoundry:[apiUrl: 'test.api.sap.com'] )

For steps which are using multiple environments we could provide the environment specific parameters as maps:

cloudPlatformDeploy(script: script, cloudFoundry:[apiUrl: 'trial2.api.sap.com'], neo: [apiUrl: 'neo2.sap.com'])

Name-spacing using Prefixes

Example Configuration

general:
  cfApiUrl: api.sap.com

steps:
  cloudFoundryDeploy:
    cfApiUrl: api.sap.com
    cfCredentialsId: cf-cloud

Example Usage

For steps working with one or multiple environments we would use prefixes:

cloudFoundryDeploy(script:script, cfApiUrl: 'api.sap.com')
cloudPlatformDeploy(script: script: neoApi: 'neo.api.sap.com', cfApi: 'api.sap.com')

Arguments:

In this sections, I tried to collect the arguments from yesterday and from our discussion today in the team

Pro Maps

Cleaner Configuration

Prefixing is unclean. Following a consistent approach, we have to put prefixes everywhere, where it belongs to an environment. Prefixing, especially using abbreviations, makes the names harder to read. More or less all parameters for steps such as cloudFoundryDeploy would be prefixed. Furthermore, there can be clashes, if abbreviations can be interpreted to map to multiple environments.

Deeper Structures

Also for one environment, there can be deep structures. Example:

general:
  cloudFoundry:
    environmentVariables:
      destinations:
        ErpQueryEndpoint: 'https://s4hana.sap.com'

Do we also flatten them in a prefix approach?

Usability

Using maps, options which belong to the same environment are grouped together. With prefixing they are distributed over the general section.

Migration Effort (Legacy)

In this repository and also for the users of the SAP S/4HANA Cloud SDK pipeline, we do not use prefixes or only partially (Should be consistent in the future). How do we migrate our external customers if we change the APIs in the configuration and the method signatures.

Contra Maps

Migration Effort (Legacy)

The internal piper version already relies on prefixing. Thus, there are migration efforts to change the configuration structure and pipelines in the internal projects. Also around 60 steps need to be migrated.

Usability

Using the prefixing approach, the it might be easier to identify the place where to configure something. It is always cfApi where as for the maps approach is it either cloudFoundry->api or just api in the parameters. However, it could also be cloudFoundry->api for the parameters. Thus, it would also be consistent here.

Anyway a documentation should explain where to configure it. Only problem might be, that users do not read the documentation carefully.

[fwilhe]

Considering the comment by @daniel-kurzynski, I think prefixes are a bad solution. YAML as a format supports deep nesting, and I think we should make use of that. As every engineering solution, this has tradeoffs, which Daniel already pointed out, but my impression is quite clear that we should embrace the YAML format.

[marcusholl]

Outcome of the discussion is:

Parameters belonging into one scope should be provided preferably in the configuration structured as a map like this:

myBackend:
  endpoint: 'https://example.org/myContext'
  credentialsID = 'ZYX'

rather than as a flat properties like list with having each key prefixed.

A pipeline step gets invoked with the map, e.g.

myPipelineStep script: this, myBackend: [endpoint: 'https://example.org/myContext', ...]

[marcusholl]

With PR #219 I switch the changeManagment related steps to a nested config. In case somebody would like to check how this looks like: review that PR.

[marcusholl]

... we are using nested configs in the meantime