starburstdata / metabase-driver

Starburst Metabase driver
Apache License 2.0
65 stars 10 forks source link

Remove prepared statements for sync #48

Closed andrewdibiasio6 closed 2 years ago

andrewdibiasio6 commented 2 years ago

Overview

During sync, current logic uses clojure's functional programming JDBC wrapper library to call the native Trino JDBC driver. This clojure wrapper lib forces us to use prepared statements and there is no way to disable that from this lib. The issue with this is:

  1. each sync query results in 2 service calls to Trino resulting in double the amount of network calls, see example:
PREPARE statement5 FROM SHOW TABLES FROM "tpch"."sf10000"
---
EXECUTE statement5
  1. they are useless prepared statments since they actually contain no values to add parameters for, they can't even be reused.

Solution

Remove prepared statements by:

Tests

I tried to preserve all the major reduce logic. I manually tested the sync locally, and all sync logic is currently tested via the existing unit tests. If sync broke, then most tests would fail.

Manual Tests

I ran sync on TPCH catalog, all schemas, all tables on insights cluster. Below is the query count of the sync for the three drivers: