search

Baoqi / metabase-dremio-driver

Dremio driver for Metabase BI
Eclipse Public License 2.0
50 stars 10 forks source link

Driver update for dremio 18.1.0 and metabase 0.41.0 #3

Closed urba1n closed 2 years ago

urba1n commented 3 years ago

Do you plan to update the driver to support latest version of dremio and metabase ?

Baoqi commented 3 years ago

Do you plan to update the driver to support latest version of dremio and metabase ?

Hi, @urba1n

There seems no incompatible changes for metabase-dremio-driver to work with newer version of dremio or metabse. So, current version should works in metabase 0.41.0 and dremio 18.1.0. (I tested on metabase 0.41.0)

Did you encounter some issues when run current version of metabase-dremio-driver?

urba1n commented 3 years ago

Hi @Baoqi,

I get some warnings and an error related to usage of timezone (see below). For the rest, everything seems to work correctly.

metabase_1 | WARNING: An illegal reflective access operation has occurred metabase_1 | WARNING: Illegal reflective access by cdjd.org.apache.calcite.avatica.com.google.protobuf.UnsafeUtil (file:/plugins/dremio.metabase-driver.jar) to field java.nio.Buffer.address metabase_1 | WARNING: Please consider reporting this to the maintainers of cdjd.org.apache.calcite.avatica.com.google.protobuf.UnsafeUtil metabase_1 | WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations metabase_1 | WARNING: All illegal access operations will be denied in a future release metabase_1 | 2021-10-11 08:21:35,299 WARN metrics.Metrics :: Removing old metric since name matched newly registered metric. Metric name: USER-send-durations-ms metabase_1 | 2021-10-11 08:21:37,288 ERROR sql-jdbc.execute :: Failed to set timezone 'Europe/Zurich' for :dremio database metabase_1 | java.sql.SQLException: PARSE ERROR: Failure parsing the query. metabase_1 | metabase_1 | SQL Query SET SESSION TIMEZONE TO 'Europe/Zurich'; metabase_1 | startLine 1 metabase_1 | startColumn 13 metabase_1 | endLine 1 metabase_1 | endColumn 20 metabase_1 | SQL Query SET SESSION TIMEZONE TO 'Europe/Zurich'; metabase_1 | metabase_1 | [Error Id: b6b5f954-d858-4909-9037-beb38c4861de on 23976a898bc4:31010] metabase_1 | metabase_1 | (org.apache.calcite.sql.parser.SqlParseException) Encountered "TIMEZONE" at line 1, column 13. metabase_1 | Was expecting one of: metabase_1 | "=" ... metabase_1 | "." ... metabase_1 | "[" ... metabase_1 | metabase_1 | com.dremio.exec.planner.sql.parser.impl.ParserImpl.convertException():341 metabase_1 | com.dremio.exec.planner.sql.parser.impl.ParserImpl.normalizeException():134 metabase_1 | org.apache.calcite.sql.parser.SqlParser.handleException():130 metabase_1 | org.apache.calcite.sql.parser.SqlParser.parseStmtList():185 metabase_1 | com.dremio.exec.planner.sql.SqlConverter.parseMultipleStatementsImpl():172 metabase_1 | com.dremio.exec.planner.sql.SqlConverter.parseSingleStatementImpl():187 metabase_1 | com.dremio.exec.planner.sql.SqlConverter.parse():203 metabase_1 | com.dremio.exec.planner.sql.handlers.commands.CommandCreator.getSqlCommand():289 metabase_1 | com.dremio.exec.planner.sql.handlers.commands.CommandCreator.toCommand():240 metabase_1 | com.dremio.exec.work.foreman.AttemptManager.plan():471 metabase_1 | com.dremio.exec.work.foreman.AttemptManager.lambda$run$1():378 metabase_1 | com.dremio.service.commandpool.ReleasableBoundCommandPool.lambda$getWrappedCommand$3():138 metabase_1 | com.dremio.service.commandpool.CommandWrapper.run():62 metabase_1 | com.dremio.context.RequestContext.run():95 metabase_1 | com.dremio.common.concurrent.ContextMigratingExecutorService.lambda$decorate$3():199 metabase_1 | com.dremio.common.concurrent.ContextMigratingExecutorService$ComparableRunnable.run():180 metabase_1 | java.util.concurrent.Executors$RunnableAdapter.call():511 metabase_1 | java.util.concurrent.FutureTask.run():266 metabase_1 | java.util.concurrent.ThreadPoolExecutor.runWorker():1149 metabase_1 | java.util.concurrent.ThreadPoolExecutor$Worker.run():624 metabase_1 | java.lang.Thread.run():748 metabase_1 | Caused By (com.dremio.exec.planner.sql.parser.impl.ParseException) Encountered "TIMEZONE" at line 1, column 13. metabase_1 | Was expecting one of: metabase_1 | "=" ... metabase_1 | "." ... metabase_1 | "[" ... metabase_1 | metabase_1 | com.dremio.exec.planner.sql.parser.impl.ParserImpl.generateParseException():30874 metabase_1 | com.dremio.exec.planner.sql.parser.impl.ParserImpl.jj_consume_token():30682 metabase_1 | com.dremio.exec.planner.sql.parser.impl.ParserImpl.SqlSetOption():5850 metabase_1 | com.dremio.exec.planner.sql.parser.impl.ParserImpl.SqlStmt():915 metabase_1 | com.dremio.exec.planner.sql.parser.impl.ParserImpl.SqlStmtList():833 metabase_1 | com.dremio.exec.planner.sql.parser.impl.ParserImpl.parseSqlStmtList():185 metabase_1 | com.dremio.exec.planner.sql.parser.impl.ParserWithCompoundIdConverter.parseSqlStmtList():73 metabase_1 | org.apache.calcite.sql.parser.SqlParser.parseStmtList():183 metabase_1 | com.dremio.exec.planner.sql.SqlConverter.parseMultipleStatementsImpl():172 metabase_1 | com.dremio.exec.planner.sql.SqlConverter.parseSingleStatementImpl():187 metabase_1 | com.dremio.exec.planner.sql.SqlConverter.parse():203 metabase_1 | com.dremio.exec.planner.sql.handlers.commands.CommandCreator.getSqlCommand():289 metabase_1 | com.dremio.exec.planner.sql.handlers.commands.CommandCreator.toCommand():240 metabase_1 | com.dremio.exec.work.foreman.AttemptManager.plan():471 metabase_1 | com.dremio.exec.work.foreman.AttemptManager.lambda$run$1():378 metabase_1 | com.dremio.service.commandpool.ReleasableBoundCommandPool.lambda$getWrappedCommand$3():138 metabase_1 | com.dremio.service.commandpool.CommandWrapper.run():62 metabase_1 | com.dremio.context.RequestContext.run():95 metabase_1 | com.dremio.common.concurrent.ContextMigratingExecutorService.lambda$decorate$3():199 metabase_1 | com.dremio.common.concurrent.ContextMigratingExecutorService$ComparableRunnable.run():180 metabase_1 | java.util.concurrent.Executors$RunnableAdapter.call():511 metabase_1 | java.util.concurrent.FutureTask.run():266 metabase_1 | java.util.concurrent.ThreadPoolExecutor.runWorker():1149 metabase_1 | java.util.concurrent.ThreadPoolExecutor$Worker.run():624 metabase_1 | java.lang.Thread.run():748 metabase_1 | metabase_1 | at com.dremio.jdbc.impl.DremioCursor.nextRowInternally(DremioCursor.java:542) metabase_1 | at com.dremio.jdbc.impl.DremioCursor.loadInitialSchema(DremioCursor.java:628) metabase_1 | at com.dremio.jdbc.impl.DremioResultSetImpl.execute(DremioResultSetImpl.java:1883) metabase_1 | at com.dremio.jdbc.impl.DremioResultSetImpl.execute(DremioResultSetImpl.java:59) metabase_1 | at cdjd.org.apache.calcite.avatica.AvaticaConnection$1.execute(AvaticaConnection.java:667) metabase_1 | at com.dremio.jdbc.impl.DremioMetaImpl.prepareAndExecute(DremioMetaImpl.java:184) metabase_1 | at cdjd.org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:675) metabase_1 | at com.dremio.jdbc.impl.DremioConnectionImpl.prepareAndExecuteInternal(DremioConnectionImpl.java:128) metabase_1 | at cdjd.org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:156) metabase_1 | at cdjd.org.apache.calcite.avatica.AvaticaStatement.execute(AvaticaStatement.java:217) metabase_1 | at com.dremio.jdbc.impl.DremioStatementImpl.execute(DremioStatementImpl.java:97) metabase_1 | at com.mchange.v2.c3p0.impl.NewProxyStatement.execute(NewProxyStatement.java:75) metabase_1 | at metabase.driver.sql_jdbc.execute$set_time_zone_if_supported_BANG_.invokeStatic(execute.clj:161) metabase_1 | at metabase.driver.sql_jdbc.execute$set_time_zone_if_supported_BANG_.invoke(execute.clj:146) metabase_1 | at metabase.driver.sql_jdbc.execute$fn__52143.invokeStatic(execute.clj:193) metabase_1 | at metabase.driver.sql_jdbc.execute$fn__52143.invoke(execute.clj:188) metabase_1 | at clojure.lang.MultiFn.invoke(MultiFn.java:239) metabase_1 | at metabase.driver.sql_jdbc.execute$execute_reducible_query.invokeStatic(execute.clj:475) metabase_1 | at metabase.driver.sql_jdbc.execute$execute_reducible_query.invoke(execute.clj:463) metabase_1 | at metabase.driver.sql_jdbc.execute$execute_reducible_query.invokeStatic(execute.clj:472) metabase_1 | at metabase.driver.sql_jdbc.execute$execute_reducible_query.invoke(execute.clj:463) metabase_1 | at metabase.driver.sql_jdbc$fn__85052.invokeStatic(sql_jdbc.clj:54) metabase_1 | at metabase.driver.sql_jdbc$fn__85052.invoke(sql_jdbc.clj:52) metabase_1 | at metabase.driver.dremio$fn__446.invokeStatic(dremio.clj:117) metabase_1 | at metabase.driver.dremio$fn__446.invoke(dremio.clj:107) metabase_1 | at clojure.lang.MultiFn.invoke(MultiFn.java:244) metabase_1 | at metabase.query_processor.context$executef.invokeStatic(context.clj:59) metabase_1 | at metabase.query_processor.context$executef.invoke(context.clj:48) metabase_1 | at metabase.query_processor.context.default$default_runf.invokeStatic(default.clj:68) metabase_1 | at metabase.query_processor.context.default$default_runf.invoke(default.clj:66) metabase_1 | at metabase.query_processor.context$runf.invokeStatic(context.clj:45) metabase_1 | at metabase.query_processor.context$runf.invoke(context.clj:39) metabase_1 | at metabase.query_processor.reducible$pivot.invokeStatic(reducible.clj:34) metabase_1 | at metabase.query_processor.reducible$pivot.invoke(reducible.clj:31) metabase_1 | at metabase.query_processor.middleware.mbql_to_native$mbql__GT_native$fn__49071.invoke(mbql_to_native.clj:25) metabase_1 | at metabase.query_processor.middleware.check_features$check_features$fn__49558.invoke(check_features.clj:39) metabase_1 | at metabase.query_processor.middleware.limit$limit$fn__47444.invoke(limit.clj:37) metabase_1 | at metabase.query_processor.middleware.cache$maybe_return_cached_results$fn__49932.invoke(cache.clj:204) metabase_1 | at metabase.query_processor.middleware.optimize_temporal_filters$optimize_temporal_filters$fn__50925.invoke(optimize_temporal_filters.clj:204) metabase_1 | at metabase.query_processor.middleware.validate_temporal_bucketing$validate_temporal_bucketing$fn__50969.invoke(validate_temporal_bucketing.clj:50) metabase_1 | at metabase.query_processor.middleware.auto_parse_filter_values$auto_parse_filter_values$fn__49128.invoke(auto_parse_filter_values.clj:43) metabase_1 | at metabase.query_processor.middleware.wrap_value_literals$wrap_value_literals$fn__39230.invoke(wrap_value_literals.clj:161) metabase_1 | at metabase.query_processor.middleware.annotate$add_column_info$fn__43661.invoke(annotate.clj:615) metabase_1 | at metabase.query_processor.middleware.permissions$check_query_permissions$fn__45599.invoke(permissions.clj:108) metabase_1 | at metabase.query_processor.middleware.pre_alias_aggregations$pre_alias_aggregations$fn__50109.invoke(pre_alias_aggregations.clj:40) metabase_1 | at metabase.query_processor.middleware.cumulative_aggregations$handle_cumulative_aggregations$fn__45997.invoke(cumulative_aggregations.clj:60) metabase_1 | at metabase.query_processor.middleware.visualization_settings$update_viz_settings$fn__45935.invoke(visualization_settings.clj:63) metabase_1 | at metabase.query_processor.middleware.resolve_joined_fields$resolve_joined_fields$fn__47042.invoke(resolve_joined_fields.clj:102) metabase_1 | at metabase.query_processor.middleware.resolve_joins$resolve_joins$fn__50695.invoke(resolve_joins.clj:171) metabase_1 | at metabase.query_processor.middleware.add_implicit_joins$add_implicit_joins$fn__50231.invoke(add_implicit_joins.clj:190) metabase_1 | at metabase.query_processor.middleware.large_int_id$convert_id_to_string$fn__47061.invoke(large_int_id.clj:59) metabase_1 | at metabase.query_processor.middleware.format_rows$format_rows$fn__50283.invoke(format_rows.clj:74) metabase_1 | at metabase.query_processor.middleware.add_default_temporal_unit$add_default_temporal_unit$fn__46433.invoke(add_default_temporal_unit.clj:23) metabase_1 | at metabase.query_processor.middleware.desugar$desugar$fn__45908.invoke(desugar.clj:21) metabase_1 | at metabase.query_processor.middleware.binning$update_binning_strategy$fn__39130.invoke(binning.clj:229) metabase_1 | at metabase.query_processor.middleware.resolve_fields$resolve_fields$fn__45242.invoke(resolve_fields.clj:34) metabase_1 | at metabase.query_processor.middleware.add_dimension_projections$add_remapping$fn__49483.invoke(add_dimension_projections.clj:314) metabase_1 | at metabase.query_processor.middleware.add_implicit_clauses$add_implicit_clauses$fn__49781.invoke(add_implicit_clauses.clj:147) metabase_1 | at metabase.query_processor.middleware.upgrade_field_literals$upgrade_field_literals$fn__47429.invoke(upgrade_field_literals.clj:40) metabase_1 | at metabase.query_processor.middleware.add_source_metadata$add_source_metadata_for_source_queries$fn__46810.invoke(add_source_metadata.clj:123) metabase_1 | at metabase.query_processor.middleware.reconcile_breakout_and_order_by_bucketing$reconcile_breakout_and_order_by_bucketing$fn__50031.invoke(reconcile_breakout_and_order_by_bucketing.clj:100) metabase_1 | at metabase.query_processor.middleware.auto_bucket_datetimes$auto_bucket_datetimes$fn__48380.invoke(auto_bucket_datetimes.clj:147) metabase_1 | at metabase.query_processor.middleware.resolve_source_table$resolve_source_tables$fn__45223.invoke(resolve_source_table.clj:45) metabase_1 | at metabase.query_processor.middleware.parameters$substitute_parameters$fn__48034.invoke(parameters.clj:111) metabase_1 | at metabase.query_processor.middleware.resolve_referenced$resolve_referenced_card_resources$fn__45296.invoke(resolve_referenced.clj:79) metabase_1 | at metabase.query_processor.middleware.expand_macros$expand_macros$fn__51353.invoke(expand_macros.clj:184) metabase_1 | at metabase.query_processor.middleware.add_timezone_info$add_timezone_info$fn__47813.invoke(add_timezone_info.clj:15) metabase_1 | at metabase.query_processor.middleware.splice_params_in_response$splice_params_in_response$fn__50298.invoke(splice_params_in_response.clj:32) metabase_1 | at metabase.query_processor.middleware.resolve_database_and_driver$resolve_database_and_driver$fn__49792$fn__49796.invoke(resolve_database_and_driver.clj:31) metabase_1 | at metabase.driver$do_with_driver.invokeStatic(driver.clj:60) metabase_1 | at metabase.driver$do_with_driver.invoke(driver.clj:56) metabase_1 | at metabase.query_processor.middleware.resolve_database_and_driver$resolve_database_and_driver$fn__49792.invoke(resolve_database_and_driver.clj:25) metabase_1 | at metabase.query_processor.middleware.fetch_source_query$resolve_card_id_source_tables$fn__45846.invoke(fetch_source_query.clj:274) metabase_1 | at metabase.query_processor.middleware.store$initialize_store$fn__46034$fn__46035.invoke(store.clj:11) metabase_1 | at metabase.query_processor.store$do_with_store.invokeStatic(store.clj:44) metabase_1 | at metabase.query_processor.store$do_with_store.invoke(store.clj:38) metabase_1 | at metabase.query_processor.middleware.store$initialize_store$fn__46034.invoke(store.clj:10) metabase_1 | at metabase.query_processor.middleware.validate$validate_query$fn__50081.invoke(validate.clj:10) metabase_1 | at metabase.query_processor.middleware.normalize_query$normalize$fn__50088.invoke(normalize_query.clj:22) metabase_1 | at metabase.query_processor.middleware.add_rows_truncated$add_rows_truncated$fn__47759.invoke(add_rows_truncated.clj:35) metabase_1 | at metabase.query_processor.middleware.results_metadata$record_and_return_metadata_BANG_$fn__49057.invoke(results_metadata.clj:147) metabase_1 | at metabase.query_processor.middleware.constraints$add_default_userland_constraints$fn__47777.invoke(constraints.clj:42) metabase_1 | at metabase.query_processor.middleware.process_userland_query$process_userland_query$fn__50067.invoke(process_userland_query.clj:146) metabase_1 | at metabase.query_processor.middleware.catch_exceptions$catch_exceptions$fn__50378.invoke(catch_exceptions.clj:169) metabase_1 | at metabase.query_processor.reducible$async_qp$qp_STAR___42481$thunk__42482.invoke(reducible.clj:103) metabase_1 | at metabase.query_processor.reducible$async_qp$qp_STAR___42481.invoke(reducible.clj:109) metabase_1 | at metabase.query_processor.reducible$sync_qp$qp_STAR___42490$fn__42493.invoke(reducible.clj:135) metabase_1 | at metabase.query_processor.reducible$sync_qp$qp_STAR___42490.invoke(reducible.clj:134) metabase_1 | at clojure.lang.AFn.applyToHelper(AFn.java:156) metabase_1 | at clojure.lang.AFn.applyTo(AFn.java:144) metabase_1 | at clojure.core$apply.invokeStatic(core.clj:669) metabase_1 | at clojure.core$apply.invoke(core.clj:662) metabase_1 | at metabase.query_processor$process_userland_query.invokeStatic(query_processor.clj:241) metabase_1 | at metabase.query_processor$process_userland_query.doInvoke(query_processor.clj:237) metabase_1 | at clojure.lang.RestFn.invoke(RestFn.java:423) metabase_1 | at metabase.query_processor$fn__51401$process_query_and_save_execution_BANG___51410$fn__51413.invoke(query_processor.clj:253) metabase_1 | at metabase.query_processor$fn__51401$process_query_and_save_execution_BANG___51410.invoke(query_processor.clj:245) metabase_1 | at metabase.query_processor$fn__51445$process_query_and_save_with_max_results_constraints_BANG___51454$fn__51457.invoke(query_processor.clj:265) metabase_1 | at metabase.query_processor$fn__51445$process_query_and_save_with_max_results_constraints_BANG___51454.invoke(query_processor.clj:258) metabase_1 | at metabase.api.dataset$run_query_async$fn__57767.invoke(dataset.clj:63) metabase_1 | at metabase.query_processor.streaming$streaming_response_STAR_$fn__38039$fn__38040.invoke(streaming.clj:160) metabase_1 | at metabase.query_processor.streaming$streaming_response_STAR_$fn__38039.invoke(streaming.clj:159) metabase_1 | at metabase.async.streaming_response$do_f_STAR_.invokeStatic(streaming_response.clj:65) metabase_1 | at metabase.async.streaming_response$do_f_STAR_.invoke(streaming_response.clj:63) metabase_1 | at metabase.async.streaming_response$do_f_async$fn__25228.invoke(streaming_response.clj:84) metabase_1 | at clojure.lang.AFn.applyToHelper(AFn.java:152) metabase_1 | at clojure.lang.AFn.applyTo(AFn.java:144) metabase_1 | at clojure.core$apply.invokeStatic(core.clj:667) metabase_1 | at clojure.core$with_bindings_STAR_.invokeStatic(core.clj:1977) metabase_1 | at clojure.core$with_bindings_STAR_.doInvoke(core.clj:1977) metabase_1 | at clojure.lang.RestFn.invoke(RestFn.java:425) metabase_1 | at clojure.lang.AFn.applyToHelper(AFn.java:156) metabase_1 | at clojure.lang.RestFn.applyTo(RestFn.java:132) metabase_1 | at clojure.core$apply.invokeStatic(core.clj:671) metabase_1 | at clojure.core$bound_fn_STAR_$fn__5767.doInvoke(core.clj:2007) metabase_1 | at clojure.lang.RestFn.invoke(RestFn.java:397) metabase_1 | at clojure.lang.AFn.run(AFn.java:22) metabase_1 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) metabase_1 | at java.base/java.util.concurrent.FutureTask.run(Unknown Source) metabase_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) metabase_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) metabase_1 | at java.base/java.lang.Thread.run(Unknown Source) metabase_1 | Caused by: cdjd.com.dremio.common.exceptions.UserRemoteException: PARSE ERROR: Failure parsing the query.

Baoqi commented 3 years ago

Hi, @urba1n 

I can reproduce this issue. The root cause of this is: dremio can not parse the following SQL
SET SESSION TIMEZONE TO 'Europe/Zurich';

Actually, dremio don't have anything similar with this statement. Currently, Dremio assume all datetime in UTC (GMT) timezone, and only support one function CONVERT_TIMEZONE to let user explicitly choose columns and explicitly select source/target timezone to convert. But this is far away from what "SET SESSIOn TIMEZONE" work.

So, currently, for Dremio in Metabase. the "Settings" -> "Localization" -> "Report Timezone" , we can only use "Database Default" now.

Hi @Baoqi,

I get some warnings and an error related to usage of timezone (see below). For the rest, everything seems to work correctly.

urba1n commented 3 years ago

Hi @Baoqi

Thank you for the information. "Database Default" solve the problem.

cbfranca commented 2 years ago

Hello,

Is it working with current versions of dremio and metabase?

Tks

slamer59 commented 2 years ago

@Baoqi it seems to work with the last dremio / metabase. However do you think we can implement an update to follow the drivers update ?

I can help on CI part but I am not really familiar with Clojure build.

Baoqi commented 2 years ago

@Baoqi it seems to work with the last dremio / metabase. However do you think we can implement an update to follow the drivers update ?

I can help on CI part but I am not really familiar with Clojure build.

Hi, @slamer59, I just got time to work on this recently. I just added the CI in main branch, using github's actions. (Note: I mainly copied the logic from https://github.com/firebolt-db/metabase-firebolt-driver project, which is Apache V2 licensed), seems works fine. So, I just close this issue currently. Thanks.