search

dacort / metabase-athena-driver

An Amazon Athena driver for Metabase 0.32 and later
Apache License 2.0
225 stars 32 forks source link

connection timeout #97

Closed sabban closed 2 years ago

sabban commented 2 years ago

hi,

I did install metabase 0.41.6 on ecs fargate with the metabase-athena-driver as described in the documentation. I added a user with the described set of permissions (but attaching permissions set to the container would be better though), and configured the stuff as required with Access key and Secret Key. I try to use eu-west-1 region.

I noticed that if the credz are not valid I get an authentication error, and that if the permissions set is not large enough I get some permissions error, so I guess connecting to the athena endpoint is not the culprit.

I've no idea on what happens here, and would be happy if someone could point me any direction I could dig ? Logs are pasted thereafter

metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at metabase.driver.util$can_connect_with_details_QMARK_.invokeStatic(util.clj:48)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at metabase.driver.util$can_connect_with_details_QMARK_.doInvoke(util.clj:31)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at clojure.lang.RestFn.invoke(RestFn.java:442)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at metabase.api.database$test_database_connection.invokeStatic(database.clj:421)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at metabase.api.database$test_database_connection.doInvoke(database.clj:411)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at clojure.lang.RestFn.invoke(RestFn.java:425)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at metabase.api.database$fn__77164$test_connection_details__77169$fn__77170.invoke(database.clj:468)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at metabase.api.database$fn__77164$test_connection_details__77169.invoke(database.clj:450)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at metabase.api.database$fn__77196.invokeStatic(database.clj:488)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at metabase.api.database$fn__77196.invoke(database.clj:474)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at compojure.core$wrap_response$fn__32066.invoke(core.clj:160)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at compojure.core$wrap_route_middleware$fn__32050.invoke(core.clj:132)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at compojure.core$wrap_route_info$fn__32055.invoke(core.clj:139)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at compojure.core$wrap_route_matches$fn__32059.invoke(core.clj:151)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at compojure.core$routes$fn__32078$f__32079.invoke(core.clj:198)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at compojure.core$routes$fn__32078.invoke(core.clj:200)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at metabase.server.middleware.auth$enforce_authentication$fn__70370.invoke(auth.clj:14)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at compojure.core$routes$fn__32078$f__32079.invoke(core.clj:198)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at compojure.core$routes$fn__32078.invoke(core.clj:200)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at compojure.core$make_context$handler__32106.invoke(core.clj:289)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at compojure.core$make_context$fn__32110.invoke(core.clj:299)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at compojure.core$routes$fn__32078$f__32079.invoke(core.clj:198)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at compojure.core$routes$fn__32078$f__32079$respond_SINGLEQUOTE___32080.invoke(core.clj:197)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at compojure.core$make_context$fn__32110.invoke(core.clj:300)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at compojure.core$routes$fn__32078$f__32079.invoke(core.clj:198)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at compojure.core$routes$fn__32078$f__32079$respond_SINGLEQUOTE___32080.invoke(core.clj:197)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at compojure.core$make_context$fn__32110.invoke(core.clj:300)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at compojure.core$routes$fn__32078$f__32079.invoke(core.clj:198)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at compojure.core$routes$fn__32078$f__32079$respond_SINGLEQUOTE___32080.invoke(core.clj:197)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at compojure.core$make_context$fn__32110.invoke(core.clj:300)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at compojure.core$routes$fn__32078$f__32079.invoke(core.clj:198)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at compojure.core$routes$fn__32078$f__32079$respond_SINGLEQUOTE___32080.invoke(core.clj:197)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at compojure.core$make_context$fn__32110.invoke(core.clj:300)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at compojure.core$routes$fn__32078$f__32079.invoke(core.clj:198)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at compojure.core$routes$fn__32078$f__32079$respond_SINGLEQUOTE___32080.invoke(core.clj:197)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at compojure.core$make_context$fn__32110.invoke(core.clj:300)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at compojure.core$routes$fn__32078$f__32079.invoke(core.clj:198)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at compojure.core$routes$fn__32078$f__32079$respond_SINGLEQUOTE___32080.invoke(core.clj:197)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at compojure.core$make_context$fn__32110.invoke(core.clj:300)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at compojure.core$routes$fn__32078$f__32079.invoke(core.clj:198)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at compojure.core$routes$fn__32078$f__32079$respond_SINGLEQUOTE___32080.invoke(core.clj:197)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at metabase.api.routes$fn__79847$fn__79850.invoke(routes.clj:56)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at compojure.core$routes$fn__32078$f__32079.invoke(core.clj:198)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at compojure.core$routes$fn__32078.invoke(core.clj:200)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at clojure.lang.AFn.applyToHelper(AFn.java:160)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at clojure.lang.AFn.applyTo(AFn.java:144)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at clojure.core$apply.invokeStatic(core.clj:667)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at clojure.core$apply.invoke(core.clj:662)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at metabase.server.routes$fn__79992$fn__79993.doInvoke(routes.clj:57)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at clojure.lang.RestFn.invoke(RestFn.java:436)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at compojure.core$routes$fn__32078$f__32079.invoke(core.clj:198)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at compojure.core$routes$fn__32078.invoke(core.clj:200)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at compojure.core$make_context$handler__32106.invoke(core.clj:289)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at compojure.core$make_context$fn__32110.invoke(core.clj:299)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at compojure.core$routes$fn__32078$f__32079.invoke(core.clj:198)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at compojure.core$routes$fn__32078$f__32079$respond_SINGLEQUOTE___32080.invoke(core.clj:197)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at compojure.core$wrap_route_matches$fn__32059.invoke(core.clj:153)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at compojure.core$routes$fn__32078$f__32079.invoke(core.clj:198)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at compojure.core$routes$fn__32078$f__32079$respond_SINGLEQUOTE___32080.invoke(core.clj:197)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at compojure.core$wrap_route_matches$fn__32059.invoke(core.clj:153)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at compojure.core$routes$fn__32078$f__32079.invoke(core.clj:198)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at compojure.core$routes$fn__32078$f__32079$respond_SINGLEQUOTE___32080.invoke(core.clj:197)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at compojure.core$wrap_route_matches$fn__32059.invoke(core.clj:153)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at compojure.core$routes$fn__32078$f__32079.invoke(core.clj:198)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at compojure.core$routes$fn__32078$f__32079$respond_SINGLEQUOTE___32080.invoke(core.clj:197)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at metabase.server.routes$fn__79980$fn__79982.invoke(routes.clj:41)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at compojure.core$routes$fn__32078$f__32079.invoke(core.clj:198)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at compojure.core$routes$fn__32078.invoke(core.clj:200)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at metabase.server.middleware.exceptions$catch_uncaught_exceptions$fn__77452.invoke(exceptions.clj:98)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at metabase.server.middleware.exceptions$catch_api_exceptions$fn__77449.invoke(exceptions.clj:86)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at metabase.server.middleware.log$log_api_call$fn__80447$fn__80448$fn__80449.invoke(log.clj:209)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at metabase.driver.sql_jdbc.execute.diagnostic$do_with_diagnostic_info.invokeStatic(diagnostic.clj:15)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at metabase.driver.sql_jdbc.execute.diagnostic$do_with_diagnostic_info.invoke(diagnostic.clj:9)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at metabase.server.middleware.log$log_api_call$fn__80447$fn__80448.invoke(log.clj:201)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at toucan.db$_do_with_call_counting.invokeStatic(db.clj:216)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at toucan.db$_do_with_call_counting.invoke(db.clj:209)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at metabase.server.middleware.log$log_api_call$fn__80447.invoke(log.clj:200)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at metabase.server.middleware.browser_cookie$ensure_browser_id_cookie$fn__83660.invoke(browser_cookie.clj:30)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at metabase.server.middleware.security$add_security_headers$fn__59285.invoke(security.clj:147)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at metabase.server.middleware.json$wrap_json_body$fn__82804.invoke(json.clj:62)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at metabase.server.middleware.json$wrap_streamed_json_response$fn__82822.invoke(json.clj:98)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at metabase.server.middleware.offset_paging$handle_paging$fn__59309.invoke(offset_paging.clj:42)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at ring.middleware.keyword_params$wrap_keyword_params$fn__83927.invoke(keyword_params.clj:55)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at ring.middleware.params$wrap_params$fn__83943.invoke(params.clj:69)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at metabase.server.middleware.misc$maybe_set_site_url$fn__34077.invoke(misc.clj:59)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at metabase.server.middleware.session$bind_current_user$fn__44982$fn__44983.invoke(session.clj:257)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at metabase.server.middleware.session$do_with_current_user.invokeStatic(session.clj:238)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at metabase.server.middleware.session$do_with_current_user.invoke(session.clj:230)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at metabase.server.middleware.session$bind_current_user$fn__44982.invoke(session.clj:256)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at metabase.server.middleware.session$wrap_current_user_info$fn__44969.invoke(session.clj:216)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at metabase.server.middleware.session$wrap_session_id$fn__44955.invoke(session.clj:162)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at metabase.server.middleware.auth$wrap_api_key$fn__70378.invoke(auth.clj:27)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at ring.middleware.cookies$wrap_cookies$fn__83847.invoke(cookies.clj:216)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at metabase.server.middleware.misc$add_content_type$fn__34060.invoke(misc.clj:27)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at metabase.server.middleware.misc$disable_streaming_buffering$fn__34085.invoke(misc.clj:76)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at ring.middleware.gzip$wrap_gzip$fn__83889.invoke(gzip.clj:86)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at metabase.server.middleware.misc$bind_request$fn__34088.invoke(misc.clj:93)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at metabase.server.middleware.ssl$redirect_to_https_middleware$fn__83676.invoke(ssl.clj:38)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at metabase.server$async_proxy_handler$fn__80220.invoke(server.clj:71)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at metabase.server.proxy$org.eclipse.jetty.server.handler.AbstractHandler$ff19274a.handle(Unknown Source)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at org.eclipse.jetty.server.Server.handle(Server.java:516)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:386)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at java.base/java.lang.Thread.run(Unknown Source)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78 Caused by: java.util.concurrent.TimeoutException: Timed out after 1.7 mins
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at metabase.util$deref_with_timeout.invokeStatic(util.clj:344)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at metabase.util$deref_with_timeout.invoke(util.clj:336)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at metabase.util$do_with_timeout.invokeStatic(util.clj:350)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at metabase.util$do_with_timeout.invoke(util.clj:347)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  at metabase.driver.util$can_connect_with_details_QMARK_.invokeStatic(util.clj:42)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78  ... 116 more
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78 2022-01-20 04:43:11,847 DEBUG middleware.log :: POST /api/database 400 1.7 mins (0 DB calls)
metabase ecs/metabase/1bfbad33e6de480592f4062c71723d78 {:valid false, :dbname "Timed out after 1.7 mins", :message "Timed out after 1.7 mins"}
dacort commented 2 years ago

Hi @sabban - please make sure your Fargate nodes have outbound access to Athena on both port 443 and 444. The latter is using for streaming result sets.

If you want, you can disable result set streaming by adding the following as an additional connection option when configuring your Athena database connection in Metabase. UseResultsetStreaming=0;

This looks like a network issue so not sure how much I'll be able to help. :) I haven't tested Metabase on ECS Fargate.

ricanol commented 2 years ago

Hello, today I had the same problem as @sabban and I released the output ports 443 and 444 and it worked!

Thanks @dacort

sabban commented 2 years ago

It seems to work, thanks for the tip !