apache / kyuubi

Apache Kyuubi is a distributed and multi-tenant gateway to provide serverless SQL on data warehouses and lakehouses.
https://kyuubi.apache.org/
Apache License 2.0
2.11k stars 916 forks source link

[Umbrella] Support to reconnect the JDBC connection #4426

Open turboFei opened 1 year ago

turboFei commented 1 year ago

Code of Conduct

Search before asking

Describe the proposal

  1. in engine side, do not terminate the session life cycle once receive the close session request, support fireoff, and wait the current statements finished(configurable)
  2. when the connection to kyuubi gateway disconnect, close the session to engine(engine will hold the session lify cycle with timeout)
  3. client holds the engine refId and sessionhandle, client can reopen the session to any kyuubi instance and connect to the original engine with engine refId.
  4. when client get the status of operation with opHandle, if the operation not found in gateway, get the status from engine side as fallback
  5. kyuubi client can persist the session information to external storage and can recover with that.

Task list

TBD

Are you willing to submit PR?

turboFei commented 1 year ago

cc @lightning-L

turboFei commented 1 year ago

cc @yaooqinn @ulysses-you @pan3793 @cfmcgrady @bowenliang123

RAHUL-KESHERVANI commented 6 months ago

hey can we please get this solved . I am trying to fetch data using incremental collect as described here https://kyuubi.readthedocs.io/en/master/deployment/spark/incremental_collection.html and facing issue when client tries to query 10gb data and reconnect to the server @lightning-L @turboFei

let me know if i can contribute to the code in any way