Installation: Functionality definition, characteristics and scenarios

[miguelsorianod]

The installation part of the 3scale-operator is meant to deploy a functional AMP platform.

The operator should also be able to "maintain" the platform configuration respecting the contents defined in the Operator and will provide some configurability options to the users.

The initial idea I have is to have a single Controller that will manage the AMP platform. Doing this has the consequence that all changes to the AMP platform will go inside the same reconciliation loop.

In case the installation operator also should be able to deploy apicast standalone I would create another different CRD.

CRDs

ThreeScale: Represents a 3scale AMP deployment ApicastStandalone (in case we want operator to manage this): Represents a 3scale Apicast Standalone deployment

Each CRD will deploy all the elements that form it.

A "standard" ThreeScale AMP deployment will deploy what's currently defined in the AMP template

Requirements

• The operator should be able to deploy a standard AMP platform with sane defaults and allow some customization scenarios
• The initial desired customization scenarios once the standard AMP platform is working is having the same functionality as the evaluation, ha, and s3 AMP templates.
• Ideally it should be able to react to changes in the CRD/s definition/s with the minimum possible downtime or no downtime (if possible), making those changes effective in the deployed resources.

Possible desired functionalities

• Ability to have an HA scenario where application pods have an increased redundancy in replicas and critical databases are externalized
• Ability to have an Evaluation scenario where resource limits and requests are removed
• There will be an option to choose the main system database software, being the allowed values: "mysql", "oracle", being "mysql" by default
• There will be an option to choose the shared storage class for system, being automatically selected by default
• There will be an option to select whether the images should be obtained from source code or from DockerImages, being the default option from DockerImages
• There will be an option to select whether we want resource limits or not (evaluation version)
• There will be an option to select whether we want critical-databases externally or not (HA version)

Possible AMP CRD representations

Here are some rough ideas on what an AMP CRD might look:

Specifying the scenario names as a key of the 'spec' of the CRD. Each scenario will have its own options. There will also be "general" options:

apiVersion: amp.3scale.net/v1alpha1
kind: ThreeScale
metadata:
  name: ApiManagementPlatform
spec:
  ampVersion: # this would control the AMP images in case Docker Images are used (also source code tags???) 
  includeResourceLimits:
  ha:
    externalDatabaseConnectionDefinitions:
      system:
      apicast:
      backend:
      zync:
  evaluation:
    <scenarioOptions>
  <otherPossibleScenarios>

Another possible scenario is defining keys for each "subsystem" on the AMP CRD:

apiVersion: amp.3scale.net/v1alpha1
kind: ThreeScale
metadata:
  name: ApiManagementPlatform
spec:
  ampVersion: # this would control the AMP images in case Docker Images are used (also source code tags???) 
  includeResourceLimits:
  apicast:
  backend:
  system:
    sharedStorageClass:
    mainDatabaseType:
    applicationsPodReplicas: # does not modify database pods
    wildcardDomain:
    wildcardPolicy:
  zync:
  imageOrigin:
    docker:
      <options>
    sourceCode:
      <options>
    allowInsecureTransport:

By looking at the previous ways of organizing the CRDs I see several levels of configurability that can exist (some of them might not exist depending on how we decide to organize the CRDs):

• GlobalModifiers
• ScenarioModifiers
• SubsystemModifiers
• ComponentModifiers

Current open questions

• How are we going to manage the Operator at versioning level? and the 3scale releases with it?
• What's the desired way we want to structure the CRD without being too much complex but flexible enough to add new future functionality?
• Will apicast standalone have a separate operator different than the "installation" operator? notice that here I just talked about
• How are we going to manage the customizability of the images used in the infrastructure? The options are by adding customization options in the CRDs or at versioning level
• At wich level do we want to apply modifications? Global level, Subsystem level, Component level? We have to be careful with this because too much configurability would be hard to maintain whereas too few configurability will not be prepared to new needs
• When specifying sensitive data, now parameters cannot be used because we don't have templates, how would we handle the specification of secrets?
  • Directly on the CRD definition
  • Referencing a previously existing Secret that has to be created permanently and before deploying the Operator
• Is it worth to explore the option of having a CRD for each subsystem in AMP instead of a single CRD? (having a 'system' CRD, a 'backend' CRD, etc...). This would make them independent between them but would require some coordination
• How are we going to manage boot dependencies between elements? Will we control it through the Operator via fields in the status map?
• How are we going to manage changes that imply a redeployment of the resource? There might be service loss or data loss depending on the change.
• Are there some changes that might not make sense with Operator? For example, what should be done if a standard deployment is already deployed with the operator and it is changed to an HA deployment? In the current context this means that the databases are externalized, thus the data would be lost (unless migrated by the user, which should take care it is consistent, up-to-date, etc...)
• Are we going to convert all the current template Parameters to CRD fields?
• How are we going to manage deletions/addition of elements when scenarios/configuration change?

First steps

• Migrate and try to adapt the current code we have to autogenerate OpenShift templates in this repository (or in a standalone repo?) and make the needed refactors to be able to reuse it for the installation operator. We need to maintain the ability of being able to keep generating templates from the data model because operator and templates will coexist for a time.
• Deploy a standard AMP platform using the installation operator. As a first step it will not maintain changes. It will just perform an initial deployment if the elements do not previously exist.

[miguelsorianod]

Sample scenarios for AMP CRD (I)

We write here some isolated scenarios that we might have for the AMP installation CRD (not the Apicast standalone one). In this sample scenarios, each one is independent of the others so the structure might be incompatible between them. This should help us to know what available possibilities are to organize CRDs

Empty AMP CRD (Standard AMP scenario without customizations)

Scenario where a standard AMP deploy is desired, without any desire for configurability

apiVersion: amp.3scale.net/v1alpha1
kind: AMP
metadata:
  name: ApiManagementPlatform
spec:
status:

The kind would be AMP, and apiVersion would be amp.3scale.net/v1alpha1. Having an "empty" AMP CRD would deploy a default AMP version of the product and would deploy a "standard" AMP scenario. This is, what's currently defined in the AMP template

A standard scenario basically deploys the following subsystems:

• Apicast (staging and production)
• Backend
• System
• Zync

The resources that are currently deployed in the standard template are:

configmap/apicast-environment
configmap/backend-environment
configmap/mysql-extra-conf
configmap/mysql-main-conf
configmap/redis-config
configmap/smtp
configmap/system
configmap/system-environment
deploymentconfig.apps.openshift.io/apicast-production
deploymentconfig.apps.openshift.io/apicast-staging
deploymentconfig.apps.openshift.io/apicast-wildcard-router
deploymentconfig.apps.openshift.io/backend-cron
deploymentconfig.apps.openshift.io/backend-listener
deploymentconfig.apps.openshift.io/backend-redis
deploymentconfig.apps.openshift.io/backend-worker
deploymentconfig.apps.openshift.io/system-app
deploymentconfig.apps.openshift.io/system-memcache
deploymentconfig.apps.openshift.io/system-mysql
deploymentconfig.apps.openshift.io/system-redis
deploymentconfig.apps.openshift.io/system-sidekiq
deploymentconfig.apps.openshift.io/system-sphinx
deploymentconfig.apps.openshift.io/zync
deploymentconfig.apps.openshift.io/zync-database
imagestream.image.openshift.io/amp-apicast
imagestream.image.openshift.io/amp-backend
imagestream.image.openshift.io/amp-system
imagestream.image.openshift.io/amp-wildcard-router
imagestream.image.openshift.io/amp-zync
imagestream.image.openshift.io/postgresql
persistentvolumeclaim/backend-redis-storage
persistentvolumeclaim/mysql-storage
persistentvolumeclaim/system-redis-storage
persistentvolumeclaim/system-storage
route.route.openshift.io/api-apicast-production
route.route.openshift.io/api-apicast-staging
route.route.openshift.io/apicast-wildcard-router
route.route.openshift.io/backend
route.route.openshift.io/system-developer
route.route.openshift.io/system-master
route.route.openshift.io/system-provider-admin
secret/apicast-redis
secret/backend-internal-api
secret/backend-listener
secret/backend-redis
secret/system-app
secret/system-database
secret/system-events-hook
secret/system-master-apicast
secret/system-memcache
secret/system-recaptcha
secret/system-redis
secret/system-seed
secret/zync
service/apicast-production
service/apicast-staging
service/apicast-wildcard-router
service/backend-listener
service/backend-redis
service/system-developer
service/system-master
service/system-memcache
service/system-mysql
service/system-provider
service/system-redis
service/system-sphinx
service/zync
service/zync-database

Secrets would be automatically created and passwords in secrets would be automatically generated too. The secrets would be automatically added to the spec section of the deployed CRD (TODO: to be defined where the secrets would appear).

In case some secret was already created before the deploy of the operator then the behaviour would be to create the fields that do not exist in them. The existing secrets would not be recreated nor its existing fields.

ImageStreams would gather images from a default docker registry. In this scenario it is assumed that the images in the docker registry would previously exist before the deploy

The elements to deploy would be all the elements that form a standard AMP deployment ()

For Apicast standalone the kind would be Apicast and apiVersion would be apicast.3scale.net/v1alpha1. This is, we would separate the products by ApiVersion and not by Kind. A product might have more than one CRD (more than one Kind)

Versionable AMP CRD (Standard AMP scenario with configurable AMP version)

Scenario where a standard AMP deploy is desired, having the ability to change the AMP version

apiVersion: amp.3scale.net/v1alpha1
kind: AMP
metadata:
  name: ApiManagementPlatform
spec:
  version: <version-string>
status:

The version field would control the AMP release to deploy (maybe better to name it release to not cause confusion??). An AMP release number would NOT have any relationship with the docker image version numbers that would form that release.

The version field would control:

• What images are deployed (might have different numbers)
• What components are deployed
• Would perform an ordered redeploy of components

Changing the version field would trigger redeploy of components in an ordered way

TODO there are lots of scenarios based on this that shoul be tackled (upgrade, downgrade, upgrade with breaking changes, upgrade without breaking changes, ...)

A different image than the images specified in a specific AMP standard release is desired

Having a specific AMP version specified in CRD, for some reason, one or more images wants to be overriden. For example, to test a specific image for a subsystem, development images, etc...

The idea is that if the image field is specified then it would override the images originally specified in the version field

Alternative 1

Have a centralized map named images where the Image URLs can be overriden:

apiVersion: amp.3scale.net/v1alpha1
kind: AMP
metadata:
  name: ApiManagementPlatform
spec:
  version: <version-string>
  images:
    system: <image-url-string>
    apicast: <image-url-string>
    backend: <image-url-string>
    memcached: <image-url-string>
    postgresql: <image-url-string>
    mysql: <image-url-string>
status:

Characteristics:

• Images centralized in a map key
• The structure might be inconsistent with having a source on a BuildConfig

Having this has the consequence of having a CRD structured based on grouping concepts in generic maps instead of having subsystem maps (system, backend, ... sections)

Alternative 2

Have different maps one for each subsystem where the Image URLS can be overriden:

apiVersion: amp.3scale.net/v1alpha1
kind: AMP
metadata:
  name: ApiManagementPlatform
spec:
  version: <version-string>
  apicast:
    image: <image-url-string>
  backend:
    image: <image-url-string>
  memcached:
    image: <image-url-string>
  mysql/oracle:
    image: <image-url-string>
  postgresql:
    image: <image-url-string>
  redis:
    image: <image-url-string>
  router:
    image: <image-url-string>
  system: 
    image: <image-url-string>
  zync:
    image: <image-url-string>
status:

Characteristics:

• There is one map per subsystem
• The structure might be inconsistent with having a source on a BuildConfig
• Our current generator source code is not thought with this in mind
• Databases are considered subsystems in itself separated from the AMP subsystems

Alternative 3

apiVersion: amp.3scale.net/v1alpha1
kind: AMP
metadata:
  name: ApiManagementPlatform
spec:
  version: <version-string>
  apicast:
    image: <image-url-string>
  backend:
    redis-image: <image-url-string>
    backend-image: <image-url-string>
  memcached:
    image: <image-url-string>
  router:
    image: <image-url-string>
  system:
    redis-image: <image-url-string>
    mysql-image/oracle-image: <image-url-string>
    memcached-image: <image-url-string>
  zync:
    image: <image-url-string>
    postgresql-image: <image-url-string>
status:

Characteristics:

• There is one map per subsystem
• The structure might be inconsistent with having a source on a BuildConfig
• Our current generator source code is not thought with this in mind
• Database image fields are specified inside the AMP subsystems fields. This has the consequence of having the same database type image field in different subsystems, leading to possible repetition. For example, the case of 'redis-image', which is repeated in several subsystems. This would be a change respect to the current template behaviour, where the same database image is used between all the subsystems

A BuildConfig source is desired for the images instead of a Docker registry

Currently the AMP template has ImageStreams, and each ImageStream has two tags:

• latest
• <version>

When builds from source code are desired what is done is that a BuildConfig is created for the ImageStream and the BuildConfig is configured to output its results to the tag latest of the corresponding ImageStream.

The idea is that Operator would not control this. If that is desired we will require the user of the operator to manually configure the BuildConfig.

The problem with this approach is that the Operator logic might encounter unexpected changes on the DeploymentConfig status when the BuildConfigs are created due to this would trigger a redeploy of the DeploymentConfigs.

No resource limits/requests are desired

In some situations like a low-resource environment or local development environment it might be desired to not have any kind of resource requirements.

This is what the 'evaluation' version of the templates does.

Alternative 1

Having a field that controls whether to have limits or not for all components:

apiVersion: amp.3scale.net/v1alpha1
kind: AMP
metadata:
spec:
  disable-resource-limits: <boolean> # false by default
status:

Characteristics:

• Simpler to manage
• It matches what we currently offer on the templates
• Coarse granularity

Alternative 2

Allow each subsystem to control whether to have limits or not:

apiVersion: amp.3scale.net/v1alpha1
kind: AMP
metadata:
  name: ApiManagementPlatform
spec:
  version: <version-string>
  apicast:
    disable-resource-limits: <boolean> # false by default
  backend:
    disable-resource-limits: <boolean> # false by default
  memcached:
    disable-resource-limits: <boolean> # false by default
  router:
    disable-resource-limits: <boolean> # false by default
  system:
    disable-resource-limits: <boolean> # false by default
  zync:
    disable-resource-limits: <boolean> # false by default
status:

Characteristics:

• More complex to manage
• Fine granularity

Shared storage for system is desired to be in S3

It is desired to use S3 as system's shared storage.

Alternative 1

apiVersion: amp.3scale.net/v1alpha1
kind: AMP
metadata:
  name: ApiManagementPlatform
spec:
  system-shared-storage: #Only one of the two fields below can be written
    pvc:
      size:
    s3:
      secretReference:
status:

Alternative 2

apiVersion: amp.3scale.net/v1alpha1
kind: AMP
metadata:
  name: ApiManagementPlatform
spec:
  apicast:
  ...
  system:
    shared-storage:
      pvc:
        size:
      s3:
        secretReference:
status:

Externalized critical databases are desired

It is desired to have highly-available databases for the critical databases, externally to the OpenShift cluster, because OpenShift currently does not have productized versions of the databases.

Alternative 1

Have a section of the CRD of database locations

apiVersion: amp.3scale.net/v1alpha1
kind: AMP
metadata:
  name: ApiManagementPlatform
spec:
  database-locations:
    system-redis:
      secretReference:
    system-mysql:
      secretReference:
    backend-redis:
      secretReference:
status:

This assumes the secrets are previously created by the user and NOT by the operator

Alternative 2

Have subsystems sections where on each subsystem database location information can be set

apiVersion: amp.3scale.net/v1alpha1
kind: AMP
metadata:
  name: ApiManagementPlatform
spec:
  apicast:
    ...
  backend:
    redis:
      secretReference:
    ...
  system:
    mysql:
      secretReference:
    redis:
      secretReference:
    ...
  ...
status:

This assumes the secrets are previously created by the user and NOT by the operator

Alternative 3

Have scenario sections where each scenario has a set of configurable options

apiVersion: amp.3scale.net/v1alpha1
kind: AMP
metadata:
  name: ApiManagementPlatform
spec:
  scenario:
    ha:
      database-locations: #another alternative is to have subsystems inside the 'ha' scenario
        system:
        redis:
        backend:
        ...
status:

Some scenarios might be incompatible between them and can be difficult to the user to know which ones are compatible or incompatible.

This assumes the secrets are previously created by the user and NOT by the operator

More redundancy of application pods is desired

The same alternative than in the previous scenario might appear, changing database-locations by replicas

Summary

After looking a few examples it seems that there are the following ways to organize the CRD:

• Have top-level sections where each section controls the configuration for all subsystems
• Have top-level sections where each section has elements for each of the subsystems
• Have top-level sections one for each subsystem, considering databases as separate subsystems than AMP subsystems
• Have top-level sections being this top-level sections agroupations of different scenario-specific options
• Have top-level sections one for each subsystem, having databases inside AMP subsystems
• A mix of the previous ones. For example, having top-level scenarios and inside each scenario have all subsystems defined and then define options for those subsystems.

There are the following tradeoffs depending on how are organized:

• Finer vs Coarser granularity
• Simple configuration vs Complex configuration
• Flexibility vs Manageability

TODO In this text we have not analyzed the side effects of changing the values on each of the scenarios