search

databricks / databricks-sql-go

Golang database/sql driver for Databricks SQL.
Apache License 2.0
34 stars 38 forks source link

Added heartbeat to cloud download #173

Open rcypher-databricks opened 9 months ago

rcypher-databricks commented 9 months ago

While downloading cloud files a heartbeat will query operation status at regular intervals to keep the connection alive. Implemented heartbeat type which takes an instance of driver.Pinger and calls it at a regular interval. Implemented statusGetter type which implements driver.Pinger by calling GetOperationStatus() Updated NewBatchLoader() to take an argument of type driver.Pinger and updated batchLoader to instantiate a heartbeat instance using the pinger. Updated NewRows() to create an instance of statusGetter and updated the various constructor functions to take a diver.Pinger argument. Added interface fetcher.Overwatch. NewConcurrentFetcher() now takes an Overwatch argument. An instance of Overwatch will be started/stopped in conjunction with fetcher start/stop.

yunbodeng-db commented 7 months ago

Core review: can you be triple sure that we're not leaking go routines?

Feature review: why is this needed? how can the session timeout while we're downloading results?

During cloud fetch, the client downloads cloud files without the involvement of DBSQL or cluster so the server may shut down due to inactivity. Heartbeat thread would keep it alive.

rcypher-databricks commented 7 months ago

But as I said before. The operation/session are open until the client reads the entire result set or manually closes. So the server shouldn't be shutting down due to inactivity.