search

relferreira / metabase-sparksql-databricks-driver

GNU Affero General Public License v3.0
31 stars 32 forks source link

Update to 50.23 and implement Brigad OAuth Databricks Driver Changes #40

Open AlexVialaBellander opened 2 months ago

AlexVialaBellander commented 2 months ago

This PR implements the OAuth changes implemented by shrodingers at Brigad.

So the credits are a bit complicated, but originally, this driver was developed by Fernando Goncalves and Rajesh Kumar Ravi. Their original repository is no longer around. However, github user relferreira, kindly updates it here. However, his solution does not allow for OAuth Secrets, which was something solved by shrodingers at Brigad.

Thus, this work is a combination of two somewhat actively maintained repositories. All that I do is to merge the two solutions and update the driver to work with Metabase 0.50.23.

Best here is if we could separate the two drivers and have this become the central repo for databricks metabase.

sos3k commented 2 months ago

is it possible to optimise the usage of sql serverless warehouses to do not perform select 1 and describe table actions so frequently? that makes the cost of using serverless warehouses significantly high. Do you have any estimate when this PR will be merged into the master? we are having issues with postgres database constraints when trying to sync databricks UC tables in metabase due to the constraints violation

shrodingers commented 2 months ago

Oh just realized this PR was up for 3 weeks ! Would gladly help here if needed, would be nice to see that merged in teh main repo ! (and to eventually backport some updates made in the fork we did) For the usage @sos3k , we disabled auto-refreshes and caching by metabase, it is not tied to the driver but the way metabase handles schema updates, it can totally be disabled or set to only run on daily (we had the same cost issue on first deploy :) )

AlexVialaBellander commented 2 months ago

main on my repo works but just bumping to 0.50.26 doesnt, see here: https://github.com/AlexVialaBellander/metabase-sparksql-databricks-driver/pull/1

borft commented 2 months ago

main on my repo works but just bumping to 0.50.26 doesnt, see here: AlexVialaBellander#1

Hmm, you are right,my bad, same problem here. It works fine for 0.50.25 though. (