Implement manual load balancing strategy

[donovanmuller]

As per the supported load balancing strategies in the initial design a manual strategy should be implemented to ensure the guarantees stated:

Manual - Eligibility is manually specified as to which cluster(s) are eligible. If there are no available Pods in the specified clusters, then no cluster Ingress node IPs will be resolved and the client will get a NXDOMAIN response

Scenario 1:

• Given 2 separate Kubernetes clusters, X, and Y
• Each cluster has a healthy Deployment with a backend Service called app and that backend service exposed with a Gslb resource on all 2 clusters as:

apiVersion: ohmyglb.absa.oss/v1beta1
kind: Gslb
metadata:
  name: app-gslb
  namespace: test-gslb
spec:
  ingress:
    rules:
      - host: app.cloud.example.com
        http:
          paths:
            - backend:
                serviceName: app
                servicePort: http
              path: /
  strategy: manual 
    clusters:
      - cluster-x

• Each cluster has one worker node that accepts Ingress traffic. The worker node in each cluster has the following name and IP:

cluster-x-worker-1: 10.0.1.10
cluster-y-worker-1: 10.1.1.11

When issuing the following command, curl -v http://app.cloud.example.com, I would expect the IP's resolved to reflect as follows (if this command was executed 3 times consecutively):

$ curl -v http://app.cloud.example.com # execution 1
*   Trying 10.0.1.10...
...

$ curl -v http://app.cloud.example.com # execution 2
*   Trying 10.0.1.10...
...

$ curl -v http://app.cloud.example.com # execution 3
*   Trying 10.0.1.10...
...

The resolved node IP's will always be those of cluster X. Even if cluster X has no healthy Deployments and cluster Y does, the NXDOMAIN response will be returned regardless. This strategy allows full manual control over which cluster's Ingress node IPs can be resolved.

NOTE:

• The design of the specification around how to indicate the manual cluster preference as described in this issue is solely for the purpose of describing the scenario. It should not be considered a design.
• The scenario where there are more than 1 clusters added to the list of manually included clusters introduces the question around how (in terms of "load balancing") the Ingress node IPs of those clusters get resolved. Would they use the default round robin strategy, if any strategy at all?

[donovanmuller]

The usefulness of this strategy should be considered before implementing. Are their valid scenarios where this feature would be valuable given the questions it raises:

The scenario where there are more than 1 clusters added to the list of manually included clusters introduces the question around how (in terms of "load balancing") the Ingress node IPs of those clusters get resolved. Would they use the default round robin strategy, if any strategy at all?

and given 3 clusters, X, Y and Z and given a Gslb resource on all 3 clusters of:

apiVersion: ohmyglb.absa.oss/v1beta1
kind: Gslb
metadata:
  name: app-gslb
  namespace: test-gslb
spec:
  ingress:
    rules:
      - host: app.cloud.example.com
        http:
          paths:
            - backend:
                serviceName: app
                servicePort: http
              path: /
  strategy: manual 
    clusters:
      - cluster-x
      - cluster-y

then you could achieve the same outcome (if you did not want cluster Z to be included in resolving Ingress node IPs) by just not creating the Gslb resource on cluster Z (or deleting it if it was created before). I.e. Only have Gslb resources on cluster X and Y using the other strategies.