feat/publishing-strategies-api

[roivaz]

Adds new PublishingStrategies API to unify how workloads are exposed.

Design

Any workload that exposes an endpoint that can be consumed by clients (either in-cluster or out-of-cluster clients) exposes the api field publishingStrategies, which allows a user to configure how the endpoint/s is/are exposed. This is called a "publishing strategy". All workload endpoints have a default publishing strategy, which right now can be one of two types: "Simple" or "Marin3rSidecar". Other design decisions:

• All workload endpoints are identified by a name (i.e.: HTTP, Proxy, etc).
• When a user specifies configuration through the workload's custom resource, the user defined configuration is merged with the defaults. This is the default mode of operation for the API, but there is also a 'replace' mode.
• By default, specifying configuration for a non-defined service is an error unless the option create: true is set to declare the intent of creating a new endpoint.
• The 'replace' mode can be used to avoid merging with the default configuration. In the replace mode, the user must specify all fields. Replace mode still requires create: true when adding a new endpoint.
• Replace mode can be used to duplicate an endpoint's strategy, declaring for example both a Simple and a Marin3rStrategy for the same endpoint, effectively causing the endpoint to be exposed by two different strategies at the same time. The user can then control the dns to decide how to route traffic to each strategy.

Simple strategy

The service is exposed through a k8s Service resource, which can be a ClusterIP Service or a LoadBalancer Service (either NLB or ELB).

publishingStrategies:
  mode: Merge
  endpoints:
    - name: HTTP
      # strategy to use
      strategy: Simple
      simple:
        # overrides the default service name
        serviceName: service-name
        # the type of service: ClusterIP/ELB/NLB
        serviceType: NLB
        # configures the external-dns annotation in the Service resource
        externalDnsHostnames:
          - example.com
        # AWS network load balancer configuration in the form of Service annotations
        # Check API definition for a comprehensive list of options
        networkLoadBalancerConfig:
          proxyProtocol: true
        # overrides the default list of ports for the Service resource
        servicePorts:
          - name: http
            protocol: TCP
            port: 80
            targetPort: echo-api-http
          - name: https
            protocol: TCP
            port: 443
            targetPort: echo-api-https

Marin3rStrategy

publishingStrategies:
  endpoints:
    - name: HTTP
      # strategy to use
      strategy: Marin3rSidecar
      # configuration options for the Marin3rSidecar strategy
      marin3rSidecar:
        # envoy dynamic configurations
        dynamicConfigs:
          http:
            generatorVersion: v1
            listenerHttp:
              allowHeadersWithUnderscores: true
              enableHttp2: false
              port: 38080
              proxyProtocol: true
              routeConfigName: router_external
        # override envoy image
        envoyImage: "envoyproxy/envoy:v1.22.11"
        # configures the external-dns annotation in the Service resource
        externalDnsHostnames:
          - example.com
        # AWS network load balancer configuration in the form of Service annotations
        # Check API definition for a comprehensive list of options
        networkLoadBalancerConfig:
          proxyProtocol: true
        # envoy node-id. If not specified the workload name is used
        nodeID: backend-listener
        # envoy sidecar port definitions
        ports:
          - name: http
            port: 15080
          - name: http-internal
            port: 15081
          - name: https
            port: 15443
          - name: metrics
            port: 15000
        # envoy sidecar resource requirements
        resources: {}
        # overrides the default Service name
        serviceName: service-name
        # overrides the default list of ports for the Service resource
        servicePorts:
          - name: http
            port: 80
            protocol: TCP
            targetPort: http
          - name: https
            port: 443
            protocol: TCP
            targetPort: https
        # the type of service: ClusterIP/ELB/NLB
        serviceType: NLB
        # configures Marin3r's shutdown manager lifecycle hooks
        shtdnmgrExtraLifecycleHooks:
          - backend-listener

Upgrade

Each controller will upgrade it's the custom resources to migrate from the old API fields to the new publishingStrategies field. As default Service names have changed, the controllers will check if the Services already exists and keep the old names to avoid recreation of Service, which most certainly would lead to loss of Service during the upgrade. This has been tested in dev but still requires thorough testing in the staging environment.

[roivaz]

/kind feature /priority important-longterm /assign

[raelga]

/lgtm

Outstanding work, congrats @roivaz

[3scale-robot]

LGTM label has been added.

Git tree hash: be46faffb136d92ea398c0812ac2f0204a4f14c4

[roivaz]

/approve

[3scale-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: roivaz

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files: - ~~[OWNERS](https://github.com/3scale-ops/saas-operator/blob/main/OWNERS)~~ [roivaz] Approvers can indicate their approval by writing `/approve` in a comment Approvers can cancel approval by writing `/approve cancel` in a comment