Presto Router to support custom SPI plugin scheduler to route query

[saravanan19]

Traditional scheduling algorithms like RoundRobin, WeightedRoundRobin doesn't work in all scenarios. It leads to poor cluster utilization and performance. There user may want to control the scheduling based on lot of factors like data source attached, cluster resource status, cluster size, historical performance, and other parameters. Providing SPI plugin to load custom schedulers would be beneficial to improve the overall performance and utilization of the clusters.

Expected Behavior or Use Case

Provide a way for the user to load custom scheduler SPI plugins to route query.

Presto Component, Service, or Connector

presto-router

Possible Implementation

• Introduce interfaces such as Scheduler and SchedulerFactory in presto-spi. a) Scheduler provides methods Optional<URI> getDestination(String user, String query); and void setCandidates(List<URI> candidates); b) SchedulerFactory provides methods Scheduler create(); and String getName(); c) Plugin interface to add additional method Iterable<SchedulerFactory> getSchedulerFactories();.

• In router-config.json file scheduler should allow new type of scheduler. Something like CustomScheduler. Also, an additional field to provide the name of the Scheduler to use from the plugin. The plugin can support multiple types of Scheduler.

• Add default Optional<URI> getDestination(String user, String query) method to the scheduler interface.

• Create a plugin manager RouterPluginManager to load plugins. This will load the Scheduler plugin from `etc/plugins/scheduler

• Create a SchedulerManager class to load the scheduler from the list of schedulers provided by plugin, based on the name of the Scheduler from router-config.json.

• Create a new Scheduler under package com.facebook.presto.router.scheduler; called CustomScheduler.java that initializes with SchedulerManager obj. CustomScheduler implements com.facebook.presto.router.scheduler.Scheduler. CustomScheduler will act as a wrapper to use the scheduler that comes from plugin. getDestination will invoke schedulerManager.getScheduler().getDestination(user, query);

Example Screenshots (if appropriate):

Context

• To increase the cluster utilization & performance.
• To reduce query failures.
• Granular level of control over where to route the query.

[tdcmeehan]

@saravanan19 this seems like a pretty reasonable request. Would you like to work on this?