search

prestodb / rfcs

Presto RFCs
Apache License 2.0
0 stars 22 forks source link

Add RFC for Multi-Cluster Resource Management #23

Open prithvip opened 3 months ago

prithvip commented 3 months ago

A proposal for multi-cluster resource management

prithvip commented 3 months ago

cc @tdcmeehan @rschlussel @arhimondr @NikhilCollooru

prithvip commented 3 months ago

@tdcmeehan

Can you add some details on how authentication and authorization will work between the queueing service and the individual coordinators?

The queueing service will authenticate and, as a result of the authentication, will create an Identity object, same as how the coordinator does it today. This Identity object will be serialized and passed as a HTTP header to the coordinator's QueuedStatementResource. The coordinator will have a trusted relationship with the queueing service, and do authentication to make sure the request is being received from the queueing service (configured trusted principal). This trusted relationship is similar to how Presto Proxy works today with coordinator. There is no change to authorization and authorization requests will happen at the coordinator (during plan time).

Can you go into more detail on what sort of changes would be needed in the client protocol (QueuedStatementResource)?

There will be no changes required to any client. QueuedStatementResource will be extended with a new POST endpoint, as "/v1/statement/{queryId}?slug=SLUG". Two headers will be added: 1) forward the serialized Identity object 2) forward query state timings such as waitingForPrerequisitiesTime, queuedTime, etc. DispatchManager will require some minor modifications to accept a externally-minted queryId, slug, and pre-existing state timings. This is the high level overview, but of course, more detailed discussion can happen on the PR.