Custom upstream load-balancing service to return the upstream host

[AmitKatyal-Sophos]

Title: Custom upstream load-balancing service

Description: We have a requirement to select the upstream host based on the custom load balancing parameters. With the current Envoy proxy load balancing support, it is only possible by updating the Envoy configuration.

While going through the envoy load balancer handling, I came across overrideHostToSelect, which is returned when the http router filter is invoked, and envoy proxy sends the request to the overridden host.

/**

• Returns the host the load balancer should select directly. If the expected host exists and     the host can be selected directly, the load balancer can bypass the load balancing algorithm.     and return the corresponding host directly.    */   virtual absl::optional overrideHostToSelect() const PURE;

Proposed Solution: Add a new "custom load balancer" http filter that will be invoked before the router filter. The "custom load balancer" http filter configuration will expect the "customer LB" service endpoint. The custom LB service will return the upstream host. The HTTP filter in turn will override the upstream host with the host returned by the "custom LB service". If the "custom LB" service doesn't return any hosts, then the existing cluster LB algorithm will be kicked in to select the appropriate host.

[phlax]

cc @wbpcode

[wbpcode]

Sounds good to me to use a external custom lb service to determine the final target host. If there is a more complete proposal would be great.

[AmitKatyal-Sophos]

Thanks @wbpcode, I'll try to come up with the proposal document.

[AmitKatyal-Sophos]

Problem

The existing envoy proxy supports custom load balancing by dynamically updating the envoy configuration, which will require frequent config updates and adds latency before the new configuration is in effect.

Custom Load Balancer

A custom load balancer is an http filter that overrides the upstream host based on the upstream host selected by the custom LB service. This override host will eventually overwrite the load balancing result. As each envoy client can have different load balancing requirements, this filter allows much better control of upstream custom load balancing without updating the envoy configuration.

Note

As upstream host load balancing is fully managed by the envoy client, it is the envoy client's responsibility to ensure the requests are evenly distributed among upstream hosts.

Overview

The custom LB HTTP filter allows requests to be routed to the upstream host returned by the "custom LB service". Here, "custom LB service" is an upstream cluster to the custom LB http filter and will be provided by the envoy clients who would want to have full control over the LB algorithm. The custom LB service will have the ability to select the upstream host based on their requirements and will return the selected upstream host to the custom LB http filter. The custom load balancer http filter, in turn, will override the upstream host, and eventually the request will get routed to the upstream host selected by the custom LB service.

Configuration

This filter should be configured with the type URL type.googleapis.com/envoy.extensions.filters.http.custom_loadbalancer.v3.CustomLoadBalancer

filters:

name: envoy.filters.http.custom_loadbalancer typed_config: "@type": type.googleapis.com/envoy.extensions.filters.http.custom_loadbalancer.v3.CustomLoadBalancer stat_prefix: custom_loadbalancer grpc_service: envoy_grpc: cluster_name: custom-loadbalancer

clusters:

name: custom-loadbalancer type: static typed_extension_protocol_options: envoy.extensions.upstreams.http.v3.HttpProtocolOptions: "@type": type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions explicit_http_config: http2_protocol_options: {} load_assignment: cluster_name: custom-loadbalancer endpoints: lb_endpoints: endpoint: address: socket_address: address: 127.0.0.1 port_value: 8080

How it works

While processing the request, the custom LB http filter will request custom LB service for the upstream host. The upstream host returned by the custom LB service will be used to insure the final load balancing results. If no upstream host is returned by the custom LB service, then existing or default Envoy load balancing handling will be used to select the upstream host.

[AmitKatyal-Sophos]

@wbpcode I have referred to the existing stateful session documentation and tried to come up with details. Please review and provide your feedback. Thanks.

[AmitKatyal-Sophos]

Any suggestions/comments ?. Thanks

[AmitKatyal-Sophos]

@wbpcode, Any suggestions/comment ?. I would be happy to contribute. Thanks.

[wbpcode]

It's good to me overall. But still unclear the more specific design, like the grpc definition.

And because there is no maintainer sponsor, so I think this filter should be added as contrib filter first. I think you can try to create a draft PR first?

[AmitKatyal-Sophos]

@wbpcode, thanks for the suggestions! I'll try to update the grpc definition. sure, let me work on the draft PR as contrib filter first. Will keep you posted.

[holooooo]

Hello guys, is there any news? We have a similar situation that need to custom loadbalancing. Is there something I can do?