search

AlexR2D2 / metabase_duckdb_driver

Metabase DuckDB Driver shipped as 3rd party plugin
Apache License 2.0
70 stars 21 forks source link

Could not initialize class org.duckdb.DuckDBNative #1

Open tomsej opened 1 year ago

tomsej commented 1 year ago

Hi, thank you for this wonderful driver. Unfortunately, I am unable to install it into the official Docker image. I have tried following Dockerfile:

FROM metabase/metabase:v0.43.0

RUN apk add libc6-compat

ADD https://github.com/AlexR2D2/metabase_duckdb_driver/releases/download/0.1.1/duckdb.metabase-driver.jar /plugins/
RUN chmod 744 /plugins/duckdb.metabase-driver.jar

But when I try to create a DuckDB database connection, I get the following error:

2022-09-22 20:20:00,334 ERROR api.database :: Cannot connect to Database
java.lang.Exception: Could not initialize class org.duckdb.DuckDBNative
    at metabase.driver.util$can_connect_with_details_QMARK_.invokeStatic(util.clj:55)
    at metabase.driver.util$can_connect_with_details_QMARK_.doInvoke(util.clj:39)
    at clojure.lang.RestFn.invoke(RestFn.java:442)
    at metabase.api.database$test_database_connection.invokeStatic(database.clj:514)
    at metabase.api.database$test_database_connection.doInvoke(database.clj:503)
    at clojure.lang.RestFn.invoke(RestFn.java:425)
    at metabase.api.database$fn__74666$test_connection_details__74671$fn__74672.invoke(database.clj:563)
    at metabase.api.database$fn__74666$test_connection_details__74671.invoke(database.clj:544)
    at metabase.api.database$fn__74698.invokeStatic(database.clj:580)
    at metabase.api.database$fn__74698.invoke(database.clj:566)
    at compojure.core$wrap_response$fn__28029.invoke(core.clj:160)
    at compojure.core$wrap_route_middleware$fn__28013.invoke(core.clj:132)
    at compojure.core$wrap_route_info$fn__28018.invoke(core.clj:139)
    at compojure.core$wrap_route_matches$fn__28022.invoke(core.clj:151)
    at compojure.core$routes$fn__28041$f__28042.invoke(core.clj:198)
    at compojure.core$routes$fn__28041.invoke(core.clj:200)
    at metabase.server.middleware.auth$enforce_authentication$fn__66669.invoke(auth.clj:14)
    at compojure.core$routes$fn__28041$f__28042.invoke(core.clj:198)
    at compojure.core$routes$fn__28041.invoke(core.clj:200)
    at compojure.core$make_context$handler__28069.invoke(core.clj:289)
    at compojure.core$make_context$fn__28073.invoke(core.clj:299)
    at compojure.core$routes$fn__28041$f__28042.invoke(core.clj:198)
    at compojure.core$routes$fn__28041$f__28042$respond_SINGLEQUOTE___28043.invoke(core.clj:197)
    at compojure.core$make_context$fn__28073.invoke(core.clj:300)
    at compojure.core$routes$fn__28041$f__28042.invoke(core.clj:198)
    at compojure.core$routes$fn__28041$f__28042$respond_SINGLEQUOTE___28043.invoke(core.clj:197)
    at compojure.core$make_context$fn__28073.invoke(core.clj:300)
    at compojure.core$routes$fn__28041$f__28042.invoke(core.clj:198)
    at compojure.core$routes$fn__28041$f__28042$respond_SINGLEQUOTE___28043.invoke(core.clj:197)
    at compojure.core$make_context$fn__28073.invoke(core.clj:300)
    at compojure.core$routes$fn__28041$f__28042.invoke(core.clj:198)
    at compojure.core$routes$fn__28041$f__28042$respond_SINGLEQUOTE___28043.invoke(core.clj:197)
    at compojure.core$make_context$fn__28073.invoke(core.clj:300)
    at compojure.core$routes$fn__28041$f__28042.invoke(core.clj:198)
    at compojure.core$routes$fn__28041$f__28042$respond_SINGLEQUOTE___28043.invoke(core.clj:197)
    at compojure.core$make_context$fn__28073.invoke(core.clj:300)
    at compojure.core$routes$fn__28041$f__28042.invoke(core.clj:198)
    at compojure.core$routes$fn__28041$f__28042$respond_SINGLEQUOTE___28043.invoke(core.clj:197)
    at compojure.core$make_context$fn__28073.invoke(core.clj:300)
    at compojure.core$routes$fn__28041$f__28042.invoke(core.clj:198)
    at compojure.core$routes$fn__28041$f__28042$respond_SINGLEQUOTE___28043.invoke(core.clj:197)
    at compojure.core$make_context$fn__28073.invoke(core.clj:300)
    at compojure.core$routes$fn__28041$f__28042.invoke(core.clj:198)
    at compojure.core$routes$fn__28041$f__28042$respond_SINGLEQUOTE___28043.invoke(core.clj:197)
    at metabase.api.routes$fn__78001$fn__78004.invoke(routes.clj:59)
    at compojure.core$routes$fn__28041$f__28042.invoke(core.clj:198)
    at compojure.core$routes$fn__28041.invoke(core.clj:200)
    at clojure.lang.AFn.applyToHelper(AFn.java:160)
    at clojure.lang.AFn.applyTo(AFn.java:144)
    at clojure.core$apply.invokeStatic(core.clj:667)
    at clojure.core$apply.invoke(core.clj:662)
    at metabase.server.routes$fn__78149$fn__78150.doInvoke(routes.clj:57)
    at clojure.lang.RestFn.invoke(RestFn.java:436)
    at compojure.core$routes$fn__28041$f__28042.invoke(core.clj:198)
    at compojure.core$routes$fn__28041.invoke(core.clj:200)
    at compojure.core$make_context$handler__28069.invoke(core.clj:289)
    at compojure.core$make_context$fn__28073.invoke(core.clj:299)
    at compojure.core$routes$fn__28041$f__28042.invoke(core.clj:198)
    at compojure.core$routes$fn__28041$f__28042$respond_SINGLEQUOTE___28043.invoke(core.clj:197)
    at compojure.core$wrap_route_matches$fn__28022.invoke(core.clj:153)
    at compojure.core$routes$fn__28041$f__28042.invoke(core.clj:198)
    at compojure.core$routes$fn__28041$f__28042$respond_SINGLEQUOTE___28043.invoke(core.clj:197)
    at compojure.core$wrap_route_matches$fn__28022.invoke(core.clj:153)
    at compojure.core$routes$fn__28041$f__28042.invoke(core.clj:198)
    at compojure.core$routes$fn__28041$f__28042$respond_SINGLEQUOTE___28043.invoke(core.clj:197)
    at compojure.core$wrap_route_matches$fn__28022.invoke(core.clj:153)
    at compojure.core$routes$fn__28041$f__28042.invoke(core.clj:198)
    at compojure.core$routes$fn__28041$f__28042$respond_SINGLEQUOTE___28043.invoke(core.clj:197)
    at metabase.server.routes$fn__78137$fn__78139.invoke(routes.clj:41)
    at compojure.core$routes$fn__28041$f__28042.invoke(core.clj:198)
    at compojure.core$routes$fn__28041.invoke(core.clj:200)
    at metabase.server.middleware.exceptions$catch_uncaught_exceptions$fn__74985.invoke(exceptions.clj:98)
    at metabase.server.middleware.exceptions$catch_api_exceptions$fn__74982.invoke(exceptions.clj:86)
    at metabase.server.middleware.log$log_api_call$fn__78619$fn__78620$fn__78621.invoke(log.clj:211)
    at metabase.driver.sql_jdbc.execute.diagnostic$do_with_diagnostic_info.invokeStatic(diagnostic.clj:15)
    at metabase.driver.sql_jdbc.execute.diagnostic$do_with_diagnostic_info.invoke(diagnostic.clj:9)
    at metabase.server.middleware.log$log_api_call$fn__78619$fn__78620.invoke(log.clj:203)
    at toucan.db$_do_with_call_counting.invokeStatic(db.clj:216)
    at toucan.db$_do_with_call_counting.invoke(db.clj:209)
    at metabase.server.middleware.log$log_api_call$fn__78619.invoke(log.clj:202)
    at metabase.server.middleware.browser_cookie$ensure_browser_id_cookie$fn__81845.invoke(browser_cookie.clj:38)
    at metabase.server.middleware.security$add_security_headers$fn__60355.invoke(security.clj:148)
    at metabase.server.middleware.json$wrap_json_body$fn__80994.invoke(json.clj:63)
    at metabase.server.middleware.json$wrap_streamed_json_response$fn__81012.invoke(json.clj:99)
    at metabase.server.middleware.offset_paging$handle_paging$fn__60379.invoke(offset_paging.clj:42)
    at ring.middleware.keyword_params$wrap_keyword_params$fn__82112.invoke(keyword_params.clj:55)
    at ring.middleware.params$wrap_params$fn__82131.invoke(params.clj:77)
    at metabase.server.middleware.misc$maybe_set_site_url$fn__35170.invoke(misc.clj:59)
    at metabase.server.middleware.session$bind_current_user$fn__46635$fn__46636.invoke(session.clj:291)
    at metabase.server.middleware.session$do_with_current_user.invokeStatic(session.clj:270)
    at metabase.server.middleware.session$do_with_current_user.invoke(session.clj:259)
    at metabase.server.middleware.session$bind_current_user$fn__46635.invoke(session.clj:290)
    at metabase.server.middleware.session$wrap_current_user_info$fn__46617.invoke(session.clj:240)
    at metabase.server.middleware.session$wrap_session_id$fn__46601.invoke(session.clj:173)
    at metabase.server.middleware.auth$wrap_api_key$fn__66677.invoke(auth.clj:27)
    at ring.middleware.cookies$wrap_cookies$fn__82032.invoke(cookies.clj:216)
    at metabase.server.middleware.misc$add_content_type$fn__35153.invoke(misc.clj:27)
    at metabase.server.middleware.misc$disable_streaming_buffering$fn__35178.invoke(misc.clj:76)
    at ring.middleware.gzip$wrap_gzip$fn__82074.invoke(gzip.clj:86)
    at metabase.server.middleware.misc$bind_request$fn__35181.invoke(misc.clj:93)
    at metabase.server.middleware.ssl$redirect_to_https_middleware$fn__81861.invoke(ssl.clj:38)
    at metabase.server$async_proxy_handler$fn__78391.invoke(server.clj:73)
    at metabase.server.proxy$org.eclipse.jetty.server.handler.AbstractHandler$ff19274a.handle(Unknown Source)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
    at org.eclipse.jetty.server.Server.handle(Server.java:516)
    at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:400)
    at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:645)
    at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:392)
    at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
    at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
    at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
    at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
    at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
    at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.lang.NoClassDefFoundError: Could not initialize class org.duckdb.DuckDBNative
    at org.duckdb.DuckDBDatabase.<init>(DuckDBDatabase.java:22)
    at org.duckdb.DuckDBDriver.connect(DuckDBDriver.java:35)
    at metabase.plugins.jdbc_proxy$proxy_driver$reify__60163.connect(jdbc_proxy.clj:33)
    at java.sql/java.sql.DriverManager.getConnection(Unknown Source)
    at java.sql/java.sql.DriverManager.getConnection(Unknown Source)
    at clojure.java.jdbc$get_driver_connection.invokeStatic(jdbc.clj:271)
    at clojure.java.jdbc$get_driver_connection.invoke(jdbc.clj:250)
    at clojure.java.jdbc$get_connection.invokeStatic(jdbc.clj:411)
    at clojure.java.jdbc$get_connection.invoke(jdbc.clj:274)
    at clojure.java.jdbc$db_query_with_resultset_STAR_.invokeStatic(jdbc.clj:1111)
    at clojure.java.jdbc$db_query_with_resultset_STAR_.invoke(jdbc.clj:1093)
    at clojure.java.jdbc$query.invokeStatic(jdbc.clj:1182)
    at clojure.java.jdbc$query.invoke(jdbc.clj:1144)
    at clojure.java.jdbc$query.invokeStatic(jdbc.clj:1160)
    at clojure.java.jdbc$query.invoke(jdbc.clj:1144)
    at metabase.driver.sql_jdbc.connection$can_connect_with_spec_QMARK_.invokeStatic(connection.clj:264)
    at metabase.driver.sql_jdbc.connection$can_connect_with_spec_QMARK_.invoke(connection.clj:261)
    at metabase.driver.sql_jdbc.connection$can_connect_QMARK_.invokeStatic(connection.clj:272)
    at metabase.driver.sql_jdbc.connection$can_connect_QMARK_.invoke(connection.clj:268)
    at metabase.driver.sql_jdbc$fn__83386.invokeStatic(sql_jdbc.clj:35)
    at metabase.driver.sql_jdbc$fn__83386.invoke(sql_jdbc.clj:33)
    at clojure.lang.MultiFn.invoke(MultiFn.java:234)
    at metabase.driver.util$can_connect_with_details_QMARK_$fn__32661.invoke(util.clj:51)
    at metabase.util$do_with_timeout$fn__6203.invoke(util.clj:353)
    at clojure.core$binding_conveyor_fn$fn__5772.invoke(core.clj:2034)
    at clojure.lang.AFn.call(AFn.java:18)
    at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    ... 1 more
2022-09-22 20:20:00,349 DEBUG middleware.log :: POST /api/database 400 7.6 ms (0 DB calls)
{:valid false,
 :dbname "Could not initialize class org.duckdb.DuckDBNative",
 :message "Could not initialize class org.duckdb.DuckDBNative"}

Not sure if the problem is the Alpine base image of Metabase.

AlexR2D2 commented 1 year ago

Hello, thanks for feedback, i am working on issue from time to time :)

AlexR2D2 commented 1 year ago

Yes, the problem in the Alpine image of Metabase. The DuckDB embeds the icu lib. ICU requeres the glibc pthread lib (not the musl pthread lib installed in the Alpine by default). I tried to install different versions of the glibc library to the Metabase Alpine image but there is some issues with anyway.

Until 2.32-r0 version of the glibc, there is the std::system_error in the icu std::call_once call. I this this is the same error as in https://github.com/microsoft/ClearScript/issues/331.

Starting from 2.33-r0 version of glibc there is "Error relocating /lib/ld-linux-x86-64.so.2: unsupported relocation type 37" error.

So, instead of digging into all of this issues i just created a Ubuntu based image of Metabase and added DuckDB plugin. Please see Docker section of README.md

peralmq commented 1 year ago

@AlexR2D2 is the script still working for you? For me it is stuck. Or.. how long should it take? I've been waiting for 2 hours and it seems stalled on the below line => [builder 4/4] RUN INTERACTIVE=false CI=true MB_EDITION=oss bin/bu 6773.7s.

And I really appreciate that the build script was created in the first place. I'm on a Mac M1 if that matters. But am using the export DOCKER_DEFAULT_PLATFORM=linux/amd64 environment variable. 


 => => #       Create backend artifact /home/circleci/resources/i18n/tr.edn from /home/circleci/locales/tr.po                                                 
[+] Building 2859.0s (10/13)                                                    => CACHED [runner 1/4] FROM docker.io/library/eclipse-temurin:11-jre@sh  0.0s [+] Building 6777.0s (10/13)
 => [internal] load build definition from Dockerfile                      0.0s  => => transferring dockerfile: 1.91kB                                    0.0s  => [internal] load .dockerignore                                         0.0s
 => => transferring context: 274B                                         0.0s  => [internal] load metadata for docker.io/library/eclipse-temurin:11-jr  1.2s  => [internal] load metadata for docker.io/metabase/ci:java-11-clj-1.11.  1.2s
 => [internal] load build context                                         1.3sf => => transferring context: 111.86MB                                     1.3s  => [builder 1/4] FROM docker.io/metabase/ci:java-11-clj-1.11.0.1100.04-  0.0s
 => CACHED [runner 1/4] FROM docker.io/library/eclipse-temurin:11-jre@sh  0.0s
 => [runner 2/4] RUN apt-get update &&     apt-get install -y bash font  94.8s
 => CACHED [builder 2/4] WORKDIR /home/circleci                           0.0s
 => [builder 3/4] COPY --chown=circleci . .                               0.7s
 => [builder 4/4] RUN INTERACTIVE=false CI=true MB_EDITION=oss bin/bu  6773.7s
 => => #           Wrote 3602 messages.
 => => #       Create backend artifact /home/circleci/resources/i18n/tr.edn fr
 => => #         Create directory /home/circleci/resources/i18n if it does not
 => => #           /home/circleci/resources/i18n already exists.
 => => #         Delete /home/circleci/resources/i18n/tr.edn if exists
 => => #           Don't need to delete /home/circleci/resources/i18n/tr.edn,
AlexR2D2 commented 1 year ago

@peralmq yet it still works in my mac. It takes around 20 minutes to build from scratch (2,5 GHz Quad-Core Intel Core i7, 16 GB 1600 MHz DDR3). But there were a couple of times it hanging too.

I pushed my image to docker hub, so you can try the follow:

  1. Change the file content metabase_duckdb_driver/docker/Dockerfile to look like this: 
    
FROM elmidster/ubuntu_metabase:v1.44.6

COPY target/duckdb.metabase-driver.jar /plugins/ RUN chmod 744 /plugins/duckdb.metabase-driver.jar


2. Download plugin into target dir
```bash
metabase_duckdb_driver $ mkdir target
metabase_duckdb_driver $ wget -P target https://github.com/AlexR2D2/metabase_duckdb_driver/releases/download/0.1.6/duckdb.metabase-driver.jar
  1. Build image 
    metabase_duckdb_driver $ docker build -t metabase_duckdb -f docker/Dockerfile .
peralmq commented 1 year ago

Thanks so much for publishing it to DockerHub, it works! 🙏

I skipped the separate downloading of the plugin and just did it all in the Dockerfile

FROM elmidster/ubuntu_metabase:v1.44.6

ADD https://github.com/AlexR2D2/metabase_duckdb_driver/releases/download/0.1.6/duckdb.metabase-driver.jar /plugins/
RUN chmod 744 /plugins/duckdb.metabase-driver.jar

yet it still works in my mac. It takes around 20 minutes to build from scratch (2,5 GHz Quad-Core Intel Core i7, 16 GB 1600 MHz DDR3). But there were a couple of times it hanging too.

OK, I tried and retried to build it on this M1 Pro and it always stalled on the same place.. must be something with the CPU architecture, I find it to be a source of difficulty still when building docker images.

gianluca-valentini commented 3 months ago

Hi, any news about this? Hi, I have the same problem using DuckDB on Alpine 3.10. java.lang.NoClassDefFoundError: Could not initialize class org.duckdb.DuckDBNative at org.duckdb.DuckDBConnection.newConnection(DuckDBConnection.java:48) ~[duckdb_jdbc-0.10.1.jar:?] at org.duckdb.DuckDBDriver.connect(DuckDBDriver.java:41) ~[duckdb_jdbc-0.10.1.jar:?] at java.sql.DriverManager.getConnection(DriverManager.java:677) ~[java.sql:?] at java.sql.DriverManager.getConnection(DriverManager.java:189) ~[java.sql:?]

Is there a workaround to fix it? I'm using DuckDB 0.10.1 in memory The code that create the error is

ByteBuffer nativeReference =
            DuckDBNative.duckdb_jdbc_startup(db_dir.getBytes(StandardCharsets.UTF_8), readOnly, properties);

and after this, I cannot understand what happen.

UPDATE the problem is present when I run jar file using java -cp I hope that it can help

AlexR2D2 commented 3 months ago

Hi, did you try the native DuckDB binary in Alpine?

gianluca-valentini commented 3 months ago

Hi @AlexR2D2

Thank you for reaching out. I haven't tried the native DuckDB binary in Alpine yet. Could you provide me with more details on how to proceed?

Currently, I have a pod based on Alpine that needs to utilize DuckDB. The pod is launched using the command "java -cp ". Could you please guide me on how to test the DuckDB binary within this setup?

Thanks a lot!

Gianluca

AlexR2D2 commented 3 months ago

The idea is to check if the native DuckDB binary works in Alpine or not. You can check this using docker image of Alpine. Most likely the DuckDB will not work there. Because DuckDB embeds the icu lib requeres the glibc pthread lib (not the musl pthread lib installed in the Alpine by default). If so you can't get metabase_duckdb_driver working in Alpine

gianluca-valentini commented 3 months ago

@AlexR2D2 thanks a lot. It does not work on Alpine. I hope that DuckDB will fix it. Changing pod SO with UBUNTU, it works fine

Thanks a lot for your help Gianluca