Closed prithvip closed 2 months ago
cc @tdcmeehan @rschlussel @arhimondr
@prithvip thank you for writing this up. Do you mind creating an RFC for it? For large changes, we want to start writing detailed RFCs so we know 1) the reasons why we made certain decisions and 2) we have easily accessible offline documentation that explains the thinking that went into these decisions.
@tdcmeehan No problem, I've created an RFC PR here: https://github.com/prestodb/rfcs/pull/23
Motivation
Currently, each Presto cluster owns its own resource management and queue. The resource manager of each cluster decides which query in the cluster’s queue should be executed next, based on logic described in the cluster’s resource group configuration.
This status quo of cluster-level queueing works fine to manage one or a few clusters, but quickly becomes problematic when managing a deployment of tens or hundreds of Presto clusters. Some issues include:
High-level Proposal
We propose to add support in Presto for an optional queueing service that sits upstream of multiple Presto clusters. This queueing service will serve as the global entry-point for all queries from the client and will implement the QueuedStatement protocol. This queueing service is responsible for:
The internal details of this queueing service are still under development, and this document focuses on the changes required to Presto for such a service to work.
Diagram
Design Constraints
High-level Work Items
Because we want this to be transparent to Presto clients, the front-end of the queueing service will reuse the existing functionality of classes in the presto-dispatcher package such as QueuedStatementResource, DispatchManager, DispatchQuery, etc. The high-level work items are: