Proposal for client improvements

[anskarl]

The current implementation of DruidClient (Druid v2.1.1) uses the connection-level client-side API (outgoingConnectionHttp and outgoingConnectionHttps). There are several limitations with the current implementation, most important ones are described below:

• Lack of an API for custom client implementation
• Cannot support multiple Druid query hosts (Brokers)
• Cannot control the number of connections to Druid
• Connections are per requested query and are not cached or reused

I would like to propose an internal API change and an additional implementation of client with advanced features:

1. The API changes affect only the internals of the library and will not change the API usage of projects using Scruid. The only changes that are visible to other projects are related with the configuration file (application.conf) and can be migrated easily.
2. Implementation of an additional client, named as DruidAdvancedHttpClient, to address the aforementioned issues.
3. The original implementation of the Druid client will remain in the project and can be selected from the configuration (perhaps remain as the default client option).

Main features of the Advanced Http client

1. Uses Akka HTTP to manage cached connection-pool to Druid host, therefore we can control the number of active connections to Druid, as well as to reuse the connections.
2. Uses a queue to issue requests to the connection-pool and receive responses when they are available (based on Akka HTTP - Using the host-level API with a queue). By having a queue in front of a cached connection-pool, we can decide what to do when the queue overflows (Backpressure, Fail, DropHead, etc., for details see OverflowStrategy in Akka documentation)
3. Can connect to more than one Druid Broker nodes and load balance the requests. In that case, the client creates a separate cached connection-pool per Broker and adds a Balancer in front of them.

Proposed changes

• Two main implementations of clients: (1) DruidHttpClient with connection-level client-side API and (2) DruidAdvancedHttpClient with cached connection-pool, load-balancer and queue. DruidHttpClient is simply a refactored version of DruidClient (Druid v2.1.1).
• Provide a base client API which is represented by a trait in order to easily change from DruidHttpClient to DruidAdvancedHttpClient, or to implement a custom client of our own.
• DruidConfig (controlled by reference.conf or application.conf files) accepts a sequence of druid hosts along with their port number. For example single host is defined as hosts="127.0.0.1:8082", while multiple hosts are defined in a comma separated format hosts=10.10.0.1:8082,10.10.0.2:8082.

• DruidConfig invokes (using reflection) the desired implementation of a client. For example, client-backend = "ing.wbaa.druid.client.DruidAdvancedHttpClient" for DruidAdvancedHttpClient, client-backend = "ing.wbaa.druid.client.DruidHttpClient" for standard DruidHttpClient and client-backend = "my.package.name.to.SomeCustomDruidClient" for any other custom client that implements the DruidClient trait.

  • Any druid client can be configured separately, see for example the configuration of DruidAdvancedHttpClient below:

  client-backend = "ing.wbaa.druid.client.DruidAdvancedHttpClient"
  
  client-config = {
  
   druid-advanced-http-client ={
  
       queue-size = 32768
       queue-overflow-strategy = "Backpressure"
       request-retries-per-host = 10
       retry-delay = 10 ms
  
       host-connection-pool = {
           max-connections = 32
           min-connections = 0
           max-open-requests = 128
           max-connection-lifetime = 15 min
       }
   }
  
  }

Tests

To demonstrate and test the new implementation I also added the following:

• Docker compose for testing to help to test the advanced client, in terms of load-balancing and tolerance over random failures.
  1. Runs Druid binded to multiple ports, in order to emulate multiple Brokers
  2. Uses ClusterFk Chaos Proxy as an unreliable HTTP proxy in order to simulate random failures.
• Spec test for load-balancing
• Performance tests of DruidAdvancedHttpClient compared to DruidHttpClient

The testing environment can be setup as below:

docker-compose run --rm wait_for_druid && docker-compose up -d chaos_proxies

It is a work in progress, but all the proposed features are complete (I would like to write docs and do some code cleanups). The complete implementation of can be found in https://github.com/anskarl/scruid/commit/6685179dbc9b197cb329b593ef8663a13604b15d

I am looking forward to your comments :)

[krisgeus]

Thanks for the efforts. Great to have support for HA Druid added. Awaiting the PR so we can review the code changes

[Fokko]

Thanks @anskarl

Looks very good and thank you for the extensive proposal. I'm also fine with merging DruidHttpClient and DruidAdvancedHttpClient. This will make maintenance easier, the only issue here will be to come up with some general purpose configuration.

[anskarl]

Compared to the initial POC, I have performed the following changes:

• Code refactorings to DruidAdvancedHttpClient for readability
• Updated docker-compose for tests. Contains three types of broker nodes: (1) healthy ones, (2) one with random delays and (3) a node that always responds with internal server error (HTTP code 500)
• For simplicity, spec test of load balancing is moved into DruidAdvancedHttpClientSpec
• The parameters request-retries-per-host and retry-delay are renamed to query-retries and query-retry-delay, respectively
• Updated configuration of CircleCI, in order to use docker-compose for tests
• Added a configuration builder for DruidAdvancedHttpClient, in order to programmatically define the parameters of the client
• Added documentation about Druid Clients (features and configuration)
• Added tests for configuration overrides
• Adjusted timeouts in unit tests

I will make a PR to review the changes.

I am not sure if it is ideal to keep only one client (at least for the next release). Regarding the configuration, I think that it may need some additional changes. For example hosts parameter may move into client configuration and therefore only the DruidAdvancedHttpClient will accept multiple hosts, while the current one will accept only a single host parameter.

[anskarl]

I am closing the issue, since the corresponding PR has been merged.