Update to DuckDB v0.7.0

[sikmike]

Hi, is there any chance that you could update the jdbc driver to v0.7?

Thank you!

[AlexR2D2]

Hi, there is some problems with loading DuckDB lib 0.7.0, maybe we should wait until 0.7.1 bug fix release...

Caused by: java.lang.UnsatisfiedLinkError: Can't load library: /var/folders/py/qykynng122z96qvmmgvfjc8r0000gn/T/libduckdb_java2792657405913840957.so
    at java.base/java.lang.ClassLoader.loadLibrary(ClassLoader.java:2633)
    at java.base/java.lang.Runtime.load0(Runtime.java:768)
    at java.base/java.lang.System.load(System.java:1837)
    at org.duckdb.DuckDBNative.<clinit>(DuckDBNative.java:52)
    at org.duckdb.DuckDBDatabase.<init>(DuckDBDatabase.java:28)
    at org.duckdb.DuckDBDriver.connect(DuckDBDriver.java:38)
    at metabase.plugins.jdbc_proxy$proxy_driver$reify__61480.connect(jdbc_proxy.clj:33)

and

Caused by: java.lang.NoClassDefFoundError: Could not initialize class org.duckdb.DuckDBNative
    at org.duckdb.DuckDBDatabase.<init>(DuckDBDatabase.java:28)
    at org.duckdb.DuckDBDriver.connect(DuckDBDriver.java:38)
    at metabase.plugins.jdbc_proxy$proxy_driver$reify__61480.connect(jdbc_proxy.clj:33)

[nicosuave]

0.7.1 is out by the way! @AlexR2D2

[AlexR2D2]

@nicosuave Yes, I know. I already tried a couple of times to move to 0.7.1 but DuckDB driver initialisation filed. So, i need time to debug this. Meanwhile, from time to time i debugging another issue with dates and Metabase variables. Also i tried to move into latest version of Metabase but so far it doesn't work It seems there is some build issues with new ver of Metabase

[nicosuave]

Sounds like you are on top of it 💪. Just giving you the heads up in case you hadn't seen!

[lucmartinon]

Hello @AlexR2D2 ,

First of all, thanks a ton for the work here, adding duckdb support to Metabase will be extremely useful for us.

I tried a very naive attempt at upgrading to v7.1 and to metabase v46.2, by simply upgrading the versions numbers in the deps.edn file and following the build instructions. However I get an error that seems likely unrelated

$ clojure -X:build :project-dir ""$(pwd)"" 
Unreadable arg: "/xxxx/metabase_duckdb_driver/metabase/modules/drivers"

Any idea where that could come from, or if this has a chance to work at all?

Also, another question: we run metabase in docker so I tried the build_docker_image.sh script. But the script could never finish, it was always hanging up at some point, without a clear patern. However I was able to replicate what ChrisH explained here: https://discourse.metabase.com/t/duckdb-support/20092/12: building an image from the simple Dockerfile he shared, which uses open-jdk, and then using this image to run metabase. Isn't this much simpler than the build_docker_image.sh script?

Sorry if that is a stupid question, I am not very advanced in image building (or closure builds haha). Thanks a lot again!

[AlexR2D2]

Hello @lucmartinon,

I uploaded the new version of driver with DuckDB 0.7.1. Also I removed all build docker instruction and add simple Dockerfile example as you proposed. I think this is much more better, thanks! :)

If you want to build the plugin by yourself this is how to do it now:

How to build

• Clone metabase repository git clone --depth 1 --branch v1.46.2 https://github.com/metabase/metabase.git

• Create duckdb driver directory in the cloned metabase sourcecode mkdir -p metabase/modules/drivers/duckdb

• Copy duckdb source code into created dir cp -rf metabase_duckdb_driver/* metabase/modules/drivers/duckdb/

• Add duckdb record to the deps file metabase/modules/drivers/deps.edn File sholud looks like this:

  ...
  metabase/sqlserver          {:local/root "sqlserver"}
  metabase/vertica            {:local/root "vertica"}
  metabase/duckdb             {:local/root "duckdb"}}}  <- add this

• Build duckdb driver

  cd metabase
  clojure -X:build:drivers:build/driver :driver :duckdb

• jar file of duckdb driver will be generated here metabase/resources/modules/duckdb.metabase-driver.jar cp metabase/resources/modules/duckdb.metabase-driver.jar <somewhere>

[lucmartinon]

Hey @AlexR2D2 thanks a lot for the update here! I just tried it and I still get the error "IO Error: Trying to read a database file with version number 43, but we can only read version 39." when adding my DuckDb database based on a file generated from the python lib 7.1

I had a very quick look and could it be that you just forgot to update the duckdb version number here:

https://github.com/AlexR2D2/metabase_duckdb_driver/blob/main/deps.edn

it still says 6.1!

[lucmartinon]

Then I tried to build it per you instructions.

I htink there is a } to many on your line metabase/duckdb {:local/root "duckdb"}}}

but even after removing this, I get an error when building:

Could not locate metabase/driver/duckdb__init.class, metabase/driver/duckdb.clj or metabase/driver/duckdb.cljc on classpath.

Although I did copy the files to metabase/modules/drivers/duckdb/, so I am not sure why they aren't found.

[AlexR2D2]

@lucmartinon you got the error while building driver because you forgot to patch the metabase file metabase/modules/drivers/deps.edn

you should add metabase/duckdb {:local/root "duckdb"} to the deps in this file, so the end of file should looks something like this

  metabase/sqlserver          {:local/root "sqlserver"}
  metabase/vertica            {:local/root "vertica"}
  metabase/duckdb             {:local/root "duckdb"}}}  <- add this

i had the same error until i patched this file

[AlexR2D2]

update the latest release, so now it should be 0.7.1 :)

[lucmartinon]

Thanks! I am still getting an error: When I use the newly released jar directly, I get 'java.nio.ByteBuffer org.duckdb.DuckDBNative.duckdb_jdbc_startup(byte[], boolean, java.util.Properties)' when I try to add my duckdb file in memory

the logs provide a bit of additional info:

at org.duckdb.DuckDBNative.duckdb_jdbc_startup(Native Method)
at org.duckdb.DuckDBDatabase.<init>(DuckDBDatabase.java:28)

I also tried to build it from scratch (indeed I had not patched the edn file correctly), but I get a new error:

 "jakarta/servlet/http/HttpServletRequest has been compiled by a more recent version of the Java Runtime (class file version 55.0), this version of the Java Runtime only recognizes class file versions up to 52.0",
 :phase :execution}
    Step "Compile clojure source files" failed with error "Syntax error macroexpanding at (ring/adapter/jetty9/common.clj:1:1)."

I tried to update my JRE to the latest, but that didn't solve it.

[AlexR2D2]

@lucmartinon i have updated release one more time :D, this time i included org.duckdb/duckdb_jdbc in the build from Maven repository (not build from my own). org.duckdb/duckdb_jdbc from Maven repository will not work for macOS of Intel arch (this is why i build my own version of duckdb_jdbc)

I am use the VSCode dev container to build the plugin, here is the link https://github.com/microsoft/vscode-dev-containers/tree/v0.241.1/containers/clojure So, i create the proj dir, then copy the .devcontainer to this proj dir and also clone metabase_duckdb_driver to this dir, run container via VSCode and etc

[lucmartinon]

This time it's working like a charm! Thanks a ton for the many updates and the swift replies!

[lucmartinon]

Hello again,

Obviously now that this work, the next question is:

Would you have time to release a new version for the freshly released duckdb 8.0 ?

I know I'm asking a lot!

thanks again :)

[marclamberti]

+1 please 🙏

[AlexR2D2]

Hi, the plugin with duckdb 0.8.0 released. Maybe a duckdb 0.8.1 bug fix will be released soon.