search

apache / kyuubi

Apache Kyuubi is a distributed and multi-tenant gateway to provide serverless SQL on data warehouses and lakehouses.
https://kyuubi.apache.org/
Apache License 2.0
2.11k stars 915 forks source link

[Bug] HTTP_PROXY Server access error at url https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-aws/3.3.4/hadoop-aws-3.3.4.pom (java.net.ConnectException: Connection timed out (Connection timed out)) #6776

Closed christiangierschner closed 2 weeks ago

christiangierschner commented 4 weeks ago

Code of Conduct

Search before asking

Describe the bug

Hello Kyuubi Team,

we are trying to setup kyuubi in kubernetes. Every connection attempt to port 10009 results in timeout error. I assume, kyuubi needs our company proxy but it cannot be set. I tried this in my values.yaml:

command: 
  - /bin/bash
  - -c
  - >
    export HTTP_PROXY=http://$PROXY_HOST:$PROXY_PORT/ && export HTTPS_PROXY=http://$PROXY_HOST:$PROXY_PORT/ && export http_proxy=http://$PROXY_HOST:$PROXY_PORT/ && export https_proxy=http://$PROXY_HOST:$PROXY_PORT/ && ./bin/kyuubi run

# Environment variables (templated)
env:
  - name: JAVA_OPTS
    value: -Dhttp.nonProxyHosts="url1|url2" -Dhttp.proxyHost=$PROXY_HOST -Dhttp.proxyPort=$PROXY_PORT -Dhttps.proxyHost=$PROXY_HOST -Dhttps.proxyPort=$PROXY_PORT
  - name: JVM_ARGS
    value: -Dhttp.nonProxyHosts="url1|url2" -Dhttp.proxyHost=$PROXY_HOST -Dhttp.proxyPort=$PROXY_PORT -Dhttps.proxyHost=$PROXY_HOST -Dhttps.proxyPort=$PROXY_PORT
  - name: HTTP_PROXY
    value: http://$PROXY_HOST:$PROXY_PORT/
  - name: HTTPS_PROXY
    value: http://$PROXY_HOST:$PROXY_PORT/
  - name: http_proxy
    value: http://$PROXY_HOST:$PROXY_PORT/
  - name: https_proxy
    value: http://$PROXY_HOST:$PROXY_PORT/

It doesn't work...

How do i set the proxy in kyuubi? Thanks... Greetings Christian

Affects Version(s)

master/1.9.2

Kyuubi Server Log Output

2024-10-23 06:19:33.796 INFO KyuubiTHttpFrontendService: Thread-50 org.eclipse.jetty.server.Server: Started @3349ms
2024-10-23 06:21:02.137 INFO KyuubiTBinaryFrontendHandler-Pool: Thread-56 org.apache.kyuubi.server.KyuubiTBinaryFrontendService: Client protocol version: HIVE_CLI_SERVICE_PROTOCOL_V10
2024-10-23 06:21:02.145 INFO KyuubiTBinaryFrontendHandler-Pool: Thread-56 org.apache.kyuubi.session.KyuubiSessionManager: Opening session for anonymous@10.42.0.1
2024-10-23 06:21:02.166 INFO KyuubiTBinaryFrontendHandler-Pool: Thread-56 org.apache.kyuubi.operation.log.OperationLog: Creating operation log file /opt/kyuubi/work/server_operation_logs/5a93847e-5086-41aa-971b-73a3d9af3934/9f9d4505-5b84-440a-a501-36010fe9b821
2024-10-23 06:21:02.168 INFO KyuubiSessionManager-exec-pool: Thread-57 org.apache.kyuubi.operation.LaunchEngine: Processing anonymous's query[9f9d4505-5b84-440a-a501-36010fe9b821]: PENDING_STATE -> RUNNING_STATE, statement:
LaunchEngine
2024-10-23 06:21:02.168 INFO KyuubiTBinaryFrontendHandler-Pool: Thread-56 org.apache.kyuubi.session.KyuubiSessionManager: anonymous's KyuubiSessionImpl with SessionHandle [5a93847e-5086-41aa-971b-73a3d9af3934] is opened, current opening sessions 1
2024-10-23 06:21:02.170 INFO KyuubiSessionManager-exec-pool: Thread-57 org.apache.kyuubi.shaded.curator.framework.imps.CuratorFrameworkImpl: Starting
2024-10-23 06:21:02.170 INFO KyuubiSessionManager-exec-pool: Thread-57 org.apache.kyuubi.shaded.zookeeper.ZooKeeper: Initiating client connection, connectString=10.42.0.187:2181 sessionTimeout=60000 watcher=org.apache.kyuubi.shaded.curator.ConnectionState@5bf84f43
2024-10-23 06:21:02.171 INFO KyuubiSessionManager-exec-pool: Thread-57-SendThread(kyuubi-1.kyuubi-headless.kyuubi.svc.cluster.local:2181) org.apache.kyuubi.shaded.zookeeper.ClientCnxn: Opening socket connection to server kyuubi-1.kyuubi-headless.kyuubi.svc.cluster.local/10.42.0.187:2181. Will not attempt to authenticate using SASL (unknown error)
2024-10-23 06:21:02.175 INFO KyuubiSessionManager-exec-pool: Thread-57-SendThread(kyuubi-1.kyuubi-headless.kyuubi.svc.cluster.local:2181) org.apache.kyuubi.shaded.zookeeper.ClientCnxn: Socket connection established to kyuubi-1.kyuubi-headless.kyuubi.svc.cluster.local/10.42.0.187:2181, initiating session
2024-10-23 06:21:02.175 INFO NIOServerCxn.Factory:/10.42.0.187:2181 org.apache.kyuubi.shaded.zookeeper.server.NIOServerCnxnFactory: Accepted socket connection from /10.42.0.187:50066
2024-10-23 06:21:02.176 INFO NIOServerCxn.Factory:/10.42.0.187:2181 org.apache.kyuubi.shaded.zookeeper.server.ZooKeeperServer: Client attempting to establish new session at /10.42.0.187:50066
2024-10-23 06:21:02.183 INFO SyncThread:0 org.apache.kyuubi.shaded.zookeeper.server.ZooKeeperServer: Established session 0x10018a0fdaa0001 with negotiated timeout 60000 for client /10.42.0.187:50066
2024-10-23 06:21:02.183 INFO KyuubiSessionManager-exec-pool: Thread-57-SendThread(kyuubi-1.kyuubi-headless.kyuubi.svc.cluster.local:2181) org.apache.kyuubi.shaded.zookeeper.ClientCnxn: Session establishment complete on server kyuubi-1.kyuubi-headless.kyuubi.svc.cluster.local/10.42.0.187:2181, sessionid = 0x10018a0fdaa0001, negotiated timeout = 60000
2024-10-23 06:21:02.184 INFO KyuubiSessionManager-exec-pool: Thread-57-EventThread org.apache.kyuubi.shaded.curator.framework.state.ConnectionStateManager: State change: CONNECTED
2024-10-23 06:21:02.194 INFO ProcessThread(sid:0 cport:2181): org.apache.kyuubi.shaded.zookeeper.server.PrepRequestProcessor: Got user-level KeeperException when processing sessionid:0x10018a0fdaa0001 type:create cxid:0x2 zxid:0x5 txntype:-1 reqpath:n/a Error Path:/kyuubi_1.9.2_USER_SPARK_SQL_lock/anonymous/default/locks Error:KeeperErrorCode = NoNode for /kyuubi_1.9.2_USER_SPARK_SQL_lock/anonymous/default/locks
2024-10-23 06:21:02.197 WARN KyuubiSessionManager-exec-pool: Thread-57 org.apache.kyuubi.shaded.curator.utils.ZKPaths: The version of ZooKeeper being used doesn't support Container nodes. CreateMode.PERSISTENT will be used instead.
2024-10-23 06:21:02.224 INFO ProcessThread(sid:0 cport:2181): org.apache.kyuubi.shaded.zookeeper.server.PrepRequestProcessor: Got user-level KeeperException when processing sessionid:0x10018a0fdaa0001 type:create cxid:0xd zxid:0xb txntype:-1 reqpath:n/a Error Path:/kyuubi_1.9.2_USER_SPARK_SQL_lock/anonymous/default/leases Error:KeeperErrorCode = NoNode for /kyuubi_1.9.2_USER_SPARK_SQL_lock/anonymous/default/leases
2024-10-23 06:21:02.251 INFO KyuubiSessionManager-exec-pool: Thread-57 org.apache.kyuubi.engine.ProcBuilder: Creating anonymous's working directory at /opt/kyuubi/work/anonymous
2024-10-23 06:21:02.259 INFO KyuubiSessionManager-exec-pool: Thread-57 org.apache.kyuubi.Utils: Loading Kyuubi properties from /opt/spark/conf/spark-defaults.conf
2024-10-23 06:21:02.265 INFO KyuubiSessionManager-exec-pool: Thread-57 org.apache.kyuubi.engine.ProcBuilder: Logging to /opt/kyuubi/work/anonymous/kyuubi-spark-sql-engine.log.0
2024-10-23 06:21:02.273 INFO KyuubiSessionManager-exec-pool: Thread-57 org.apache.kyuubi.engine.EngineRef: Launching engine:
/opt/kyuubi/externals/spark-3.5.1-bin-hadoop3/bin/spark-submit \
        --class org.apache.kyuubi.engine.spark.SparkSQLEngine \
        --conf spark.hive.server2.thrift.resultset.default.fetch.size=1000 \
        --conf spark.kyuubi.client.ipAddress=10.0.98.36 \
        --conf spark.kyuubi.client.version=1.9.2 \
        --conf spark.kyuubi.engine.engineLog.path=/opt/kyuubi/work/anonymous/kyuubi-spark-sql-engine.log.0 \
        --conf spark.kyuubi.engine.share.level=USER \
        --conf spark.kyuubi.engine.submit.time=1729664462246 \
        --conf spark.kyuubi.engine.type=SPARK_SQL \
        --conf spark.kyuubi.frontend.connection.url.use.hostname=false \
        --conf spark.kyuubi.frontend.protocols=REST,THRIFT_BINARY,THRIFT_HTTP \
        --conf spark.kyuubi.ha.addresses=10.42.0.187:2181 \
        --conf spark.kyuubi.ha.engine.ref.id=5a93847e-5086-41aa-971b-73a3d9af3934 \
        --conf spark.kyuubi.ha.namespace=/kyuubi_1.9.2_USER_SPARK_SQL/anonymous/default \
        --conf spark.kyuubi.ha.zookeeper.auth.type=NONE \
        --conf spark.kyuubi.kubernetes.namespace=kyuubi \
        --conf spark.kyuubi.metrics.enabled=true \
        --conf spark.kyuubi.metrics.prometheus.port=10019 \
        --conf spark.kyuubi.metrics.reporters=PROMETHEUS \
        --conf spark.kyuubi.server.ipAddress=10.42.0.187 \
        --conf spark.kyuubi.session.connection.url=10.42.0.187:10009 \
        --conf spark.kyuubi.session.engine.initialize.timeout=PT10M \
        --conf spark.kyuubi.session.real.user=anonymous \
        --conf spark.app.name=kyuubi_USER_SPARK_SQL_anonymous_default_5a93847e-5086-41aa-971b-73a3d9af3934 \
        --conf spark.kubernetes.driver.label.kyuubi-unique-tag=5a93847e-5086-41aa-971b-73a3d9af3934 \
        --conf spark.kubernetes.driver.pod.name=kyuubi-user-spark-sql-anonymous-default-5a93847e-5086-41aa-971b-73a3d9af3934-driver \
        --conf spark.kubernetes.executor.podNamePrefix=kyuubi-user-spark-sql-anonymous-default-5a93847e-5086-41aa-971b-73a3d9af3934 \
        --conf spark.master=k8s://https://10.43.0.1:443 \
        --conf spark.kubernetes.driverEnv.SPARK_USER_NAME=anonymous \
        --conf spark.executorEnv.SPARK_USER_NAME=anonymous \
        --proxy-user anonymous /opt/kyuubi/externals/engines/spark/kyuubi-spark-sql-engine_2.12-1.9.2.jar
2024-10-23 06:24:33.028 INFO KyuubiSessionManager-timeout-checker: Thread-36 org.apache.kyuubi.session.KyuubiSessionManager: Checking sessions timeout, current count: 1
2024-10-23 06:29:33.030 INFO KyuubiSessionManager-timeout-checker: Thread-36 org.apache.kyuubi.session.KyuubiSessionManager: Checking sessions timeout, current count: 1
2024-10-23 06:31:02.833 INFO KyuubiSessionManager-exec-pool: Thread-57 org.apache.kyuubi.util.KubernetesUtils: Auto-configuring K8S client using current context from users K8S config file
2024-10-23 06:31:03.040 INFO KyuubiSessionManager-exec-pool: Thread-57 org.apache.kyuubi.engine.KubernetesApplicationOperation: [KubernetesInfo(None,Some(kyuubi))] Initialized Kubernetes Client connect to: https://10.43.0.1:443/
2024-10-23 06:33:12.286 ERROR OkHttp https://10.43.0.1/... io.fabric8.kubernetes.client.informers.impl.cache.Reflector: listSyncAndWatch failed for v1/namespaces/kyuubi/pods, will stop
java.util.concurrent.CompletionException: java.io.InterruptedIOException: timeout
        at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
        at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
        at java.util.concurrent.CompletableFuture.uniCompose(CompletableFuture.java:957)
        at java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:940)
        at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
        at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
        at io.fabric8.kubernetes.client.okhttp.OkHttpClientImpl$1.onFailure(OkHttpClientImpl.java:320)
        at okhttp3.RealCall$AsyncCall.execute(RealCall.java:211)
        at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)
Caused by: java.io.InterruptedIOException: timeout
        at okhttp3.RealCall.timeoutExit(RealCall.java:108)
        at okhttp3.RealCall$AsyncCall.execute(RealCall.java:205)
        ... 4 more
Caused by: java.net.SocketException: Socket closed
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
        at java.net.SocketInputStream.read(SocketInputStream.java:171)
        at java.net.SocketInputStream.read(SocketInputStream.java:141)
        at okio.Okio$2.read(Okio.java:140)
        at okio.AsyncTimeout$2.read(AsyncTimeout.java:237)
        at okio.RealBufferedSource.indexOf(RealBufferedSource.java:354)
        at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:226)
        at okhttp3.internal.http1.Http1Codec.readHeaderLine(Http1Codec.java:215)
        at okhttp3.internal.http1.Http1Codec.readResponseHeaders(Http1Codec.java:189)
        at okhttp3.internal.connection.RealConnection.createTunnel(RealConnection.java:384)
        at okhttp3.internal.connection.RealConnection.connectTunnel(RealConnection.java:220)
        at okhttp3.internal.connection.RealConnection.connect(RealConnection.java:161)
        at okhttp3.internal.connection.StreamAllocation.findConnection(StreamAllocation.java:258)
        at okhttp3.internal.connection.StreamAllocation.findHealthyConnection(StreamAllocation.java:135)
        at okhttp3.internal.connection.StreamAllocation.newStream(StreamAllocation.java:114)
        at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:42)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
        at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
        at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
        at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:127)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
        at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:257)
        at okhttp3.RealCall$AsyncCall.execute(RealCall.java:201)
        ... 4 more
2024-10-23 06:33:12.295 INFO Curator-Framework-0 org.apache.kyuubi.shaded.curator.framework.imps.CuratorFrameworkImpl: backgroundOperationsLoop exiting
2024-10-23 06:33:12.297 INFO ProcessThread(sid:0 cport:2181): org.apache.kyuubi.shaded.zookeeper.server.PrepRequestProcessor: Processed session termination for sessionid: 0x10018a0fdaa0001
2024-10-23 06:33:12.299 INFO KyuubiSessionManager-exec-pool: Thread-57 org.apache.kyuubi.shaded.zookeeper.ZooKeeper: Session: 0x10018a0fdaa0001 closed
2024-10-23 06:33:12.300 INFO KyuubiSessionManager-exec-pool: Thread-57-EventThread org.apache.kyuubi.shaded.zookeeper.ClientCnxn: EventThread shut down for session: 0x10018a0fdaa0001
2024-10-23 06:33:12.300 INFO NIOServerCxn.Factory:/10.42.0.187:2181 org.apache.kyuubi.shaded.zookeeper.server.NIOServerCnxn: Closed socket connection for client /10.42.0.187:50066 which had sessionid 0x10018a0fdaa0001
2024-10-23 06:33:12.306 INFO KyuubiSessionManager-exec-pool: Thread-57 org.apache.kyuubi.operation.LaunchEngine: Processing anonymous's query[9f9d4505-5b84-440a-a501-36010fe9b821]: RUNNING_STATE -> ERROR_STATE, time taken: 730.138 seconds
2024-10-23 06:33:12.337 INFO KyuubiTBinaryFrontendHandler-Pool: Thread-56 org.apache.kyuubi.server.KyuubiTBinaryFrontendService: Received request of closing SessionHandle [5a93847e-5086-41aa-971b-73a3d9af3934]
2024-10-23 06:33:12.339 INFO KyuubiTBinaryFrontendHandler-Pool: Thread-56 org.apache.kyuubi.session.KyuubiSessionManager: anonymous's KyuubiSessionImpl with SessionHandle [5a93847e-5086-41aa-971b-73a3d9af3934] is closed, current opening sessions 0
2024-10-23 06:33:12.340 INFO KyuubiTBinaryFrontendHandler-Pool: Thread-56 org.apache.kyuubi.operation.LaunchEngine: Processing anonymous's query[9f9d4505-5b84-440a-a501-36010fe9b821]: ERROR_STATE -> CLOSED_STATE, time taken: 730.172 seconds
2024-10-23 06:33:12.345 INFO KyuubiTBinaryFrontendHandler-Pool: Thread-56 org.apache.kyuubi.server.KyuubiTBinaryFrontendService: Finished closing SessionHandle [5a93847e-5086-41aa-971b-73a3d9af3934]

Kyuubi Engine Log Output

kyuubi@kyuubi-0:/opt/kyuubi$ cat work/anonymous/kyuubi-spark-sql-engine.log.0 
24/10/22 14:20:31 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
:: loading settings :: url = jar:file:/opt/kyuubi/externals/spark-3.5.1-bin-hadoop3/jars/ivy-2.5.1.jar!/org/apache/ivy/core/settings/ivysettings.xml
Ivy Default Cache set to: /home/kyuubi/.ivy2/cache
The jars for the packages stored in: /home/kyuubi/.ivy2/jars
org.apache.hadoop#hadoop-aws added as a dependency
com.amazonaws#aws-java-sdk-bundle added as a dependency
:: resolving dependencies :: org.apache.spark#spark-submit-parent-dbe367f8-5df9-4650-b217-1a3550f1fecf;1.0
        confs: [default]
:: resolution report :: resolve 1047769ms :: artifacts dl 0ms
        :: modules in use:
        ---------------------------------------------------------------------
        |                  |            modules            ||   artifacts   |
        |       conf       | number| search|dwnlded|evicted|| number|dwnlded|
        ---------------------------------------------------------------------
        |      default     |   2   |   0   |   0   |   0   ||   0   |   0   |
        ---------------------------------------------------------------------

:: problems summary ::
:::: WARNINGS
                module not found: org.apache.hadoop#hadoop-aws;3.3.4

        ==== local-m2-cache: tried

          file:/home/kyuubi/.m2/repository/org/apache/hadoop/hadoop-aws/3.3.4/hadoop-aws-3.3.4.pom

          -- artifact org.apache.hadoop#hadoop-aws;3.3.4!hadoop-aws.jar:

          file:/home/kyuubi/.m2/repository/org/apache/hadoop/hadoop-aws/3.3.4/hadoop-aws-3.3.4.jar

        ==== local-ivy-cache: tried

          /home/kyuubi/.ivy2/local/org.apache.hadoop/hadoop-aws/3.3.4/ivys/ivy.xml

          -- artifact org.apache.hadoop#hadoop-aws;3.3.4!hadoop-aws.jar:

          /home/kyuubi/.ivy2/local/org.apache.hadoop/hadoop-aws/3.3.4/jars/hadoop-aws.jar

        ==== central: tried

          https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-aws/3.3.4/hadoop-aws-3.3.4.pom

          -- artifact org.apache.hadoop#hadoop-aws;3.3.4!hadoop-aws.jar:

          https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-aws/3.3.4/hadoop-aws-3.3.4.jar

        ==== spark-packages: tried

          https://repos.spark-packages.org/org/apache/hadoop/hadoop-aws/3.3.4/hadoop-aws-3.3.4.pom

          -- artifact org.apache.hadoop#hadoop-aws;3.3.4!hadoop-aws.jar:

          https://repos.spark-packages.org/org/apache/hadoop/hadoop-aws/3.3.4/hadoop-aws-3.3.4.jar

                module not found: com.amazonaws#aws-java-sdk-bundle;1.12.262

        ==== local-m2-cache: tried

          file:/home/kyuubi/.m2/repository/com/amazonaws/aws-java-sdk-bundle/1.12.262/aws-java-sdk-bundle-1.12.262.pom

          -- artifact com.amazonaws#aws-java-sdk-bundle;1.12.262!aws-java-sdk-bundle.jar:

          file:/home/kyuubi/.m2/repository/com/amazonaws/aws-java-sdk-bundle/1.12.262/aws-java-sdk-bundle-1.12.262.jar

        ==== local-ivy-cache: tried

          /home/kyuubi/.ivy2/local/com.amazonaws/aws-java-sdk-bundle/1.12.262/ivys/ivy.xml

          -- artifact com.amazonaws#aws-java-sdk-bundle;1.12.262!aws-java-sdk-bundle.jar:

          /home/kyuubi/.ivy2/local/com.amazonaws/aws-java-sdk-bundle/1.12.262/jars/aws-java-sdk-bundle.jar

        ==== central: tried

          https://repo1.maven.org/maven2/com/amazonaws/aws-java-sdk-bundle/1.12.262/aws-java-sdk-bundle-1.12.262.pom

          -- artifact com.amazonaws#aws-java-sdk-bundle;1.12.262!aws-java-sdk-bundle.jar:

          https://repo1.maven.org/maven2/com/amazonaws/aws-java-sdk-bundle/1.12.262/aws-java-sdk-bundle-1.12.262.jar

        ==== spark-packages: tried

          https://repos.spark-packages.org/com/amazonaws/aws-java-sdk-bundle/1.12.262/aws-java-sdk-bundle-1.12.262.pom

          -- artifact com.amazonaws#aws-java-sdk-bundle;1.12.262!aws-java-sdk-bundle.jar:

          https://repos.spark-packages.org/com/amazonaws/aws-java-sdk-bundle/1.12.262/aws-java-sdk-bundle-1.12.262.jar

                ::::::::::::::::::::::::::::::::::::::::::::::

                ::          UNRESOLVED DEPENDENCIES         ::

                ::::::::::::::::::::::::::::::::::::::::::::::

                :: org.apache.hadoop#hadoop-aws;3.3.4: not found

                :: com.amazonaws#aws-java-sdk-bundle;1.12.262: not found

                ::::::::::::::::::::::::::::::::::::::::::::::

:::: ERRORS
        Server access error at url https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-aws/3.3.4/hadoop-aws-3.3.4.pom (java.net.ConnectException: Connection timed out (Connection timed out))

        Server access error at url https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-aws/3.3.4/hadoop-aws-3.3.4.jar (java.net.ConnectException: Connection timed out (Connection timed out))

        Server access error at url https://repos.spark-packages.org/org/apache/hadoop/hadoop-aws/3.3.4/hadoop-aws-3.3.4.pom (java.net.ConnectException: Connection timed out (Connection timed out))

        Server access error at url https://repos.spark-packages.org/org/apache/hadoop/hadoop-aws/3.3.4/hadoop-aws-3.3.4.jar (java.net.ConnectException: Connection timed out (Connection timed out))

        Server access error at url https://repo1.maven.org/maven2/com/amazonaws/aws-java-sdk-bundle/1.12.262/aws-java-sdk-bundle-1.12.262.pom (java.net.ConnectException: Connection timed out (Connection timed out))

        Server access error at url https://repo1.maven.org/maven2/com/amazonaws/aws-java-sdk-bundle/1.12.262/aws-java-sdk-bundle-1.12.262.jar (java.net.ConnectException: Connection timed out (Connection timed out))

        Server access error at url https://repos.spark-packages.org/com/amazonaws/aws-java-sdk-bundle/1.12.262/aws-java-sdk-bundle-1.12.262.pom (java.net.ConnectException: Connection timed out (Connection timed out))

        Server access error at url https://repos.spark-packages.org/com/amazonaws/aws-java-sdk-bundle/1.12.262/aws-java-sdk-bundle-1.12.262.jar (java.net.ConnectException: Connection timed out (Connection timed out))

:: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS
Exception in thread "main" java.lang.RuntimeException: [unresolved dependency: org.apache.hadoop#hadoop-aws;3.3.4: not found, unresolved dependency: com.amazonaws#aws-java-sdk-bundle;1.12.262: not found]
        at org.apache.spark.deploy.SparkSubmitUtils$.resolveMavenCoordinates(SparkSubmit.scala:1608)
        at org.apache.spark.util.DependencyUtils$.resolveMavenDependencies(DependencyUtils.scala:185)
        at org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:334)
        at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:964)
        at org.apache.spark.deploy.SparkSubmit$$anon$1.run(SparkSubmit.scala:176)
        at org.apache.spark.deploy.SparkSubmit$$anon$1.run(SparkSubmit.scala:174)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
        at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:174)
        at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:217)
        at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91)
        at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1120)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1129)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

Kyuubi Server Configurations

kyuubi@kyuubi-1:/opt/kyuubi$ cat conf/kyuubi-defaults.conf 
## Helm chart provided Kyuubi configurations
kyuubi.kubernetes.namespace=kyuubi
kyuubi.frontend.connection.url.use.hostname=false
kyuubi.frontend.thrift.binary.bind.port=10009
kyuubi.frontend.thrift.http.bind.port=10010
kyuubi.frontend.rest.bind.port=10099
kyuubi.frontend.mysql.bind.port=3309
kyuubi.frontend.protocols=REST,THRIFT_BINARY,THRIFT_HTTP

# Kyuubi Metrics
kyuubi.metrics.enabled=true
kyuubi.metrics.reporters=PROMETHEUS
kyuubi.metrics.prometheus.port=10019

## User provided Kyuubi configurations
kyuubi.authentication=NONE
kyuubi.engine.type=SPARK_SQL
kyuubi.engine.share.level=USER
kyuubi.session.engine.initialize.timeout=PT10M
kyuubi.ha.namespace=kyuubi
#kyuubi.frontend.bind.host=10.0.0.1
#kyuubi.ha.addresses=zk1:2181,zk2:2181,zk3:2181
#spark.submit.deployMode=cluster

Kyuubi Engine Configurations

kyuubi@kyuubi-1:/opt/kyuubi$ cat ../spark/conf/spark-defaults.conf 
spark.submit.deployMode=cluster
spark.kubernetes.container.image=apache/spark:3.5.3
spark.kubernetes.namespace=kyuubi
spark.kubernetes.authenticate.driver.serviceAccountName=spark
spark.kubernetes.authenticate.caCertFile=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
spark.kubernetes.authenticate.oauthTokenFile=/var/run/secrets/kubernetes.io/serviceaccount/token
spark.kubernetes.file.upload.path=s3a://logs/eventlogs

# S3 dependencies
spark.jars.packages=org.apache.hadoop:hadoop-aws:3.3.4,com.amazonaws:aws-java-sdk-bundle:1.12.262
spark.driver.extraJavaOptions=-Divy.cache.dir=/tmp -Divy.home=/tmp
# S3A configuration
spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem
spark.hadoop.fs.s3a.endpoint=https://min.io
spark.hadoop.fs.s3a.access.key=****
spark.hadoop.fs.s3a.secret.key=****
spark.hadoop.fs.s3a.path.style.access=true
spark.hadoop.fs.s3a.fast.upload=true

Additional context

No response

Are you willing to submit PR?

github-actions[bot] commented 4 weeks ago

Hello @christiangierschner, Thanks for finding the time to report the issue! We really appreciate the community's efforts to improve Apache Kyuubi.

pan3793 commented 3 weeks ago

@christiangierschner it indicates that spark-submit process does not respect HTTP(s)_PROXY env. AFAIK, most Java HTTP libraries do not support that. I would suggest 1) set env DEFAULT_ARTIFACT_REPOSITORY to your corp maven repo, instead of the https://repo1.maven.org/maven2/, or 2) rebuild your Spark and Kyuubi image to ship the additional jars

pan3793 commented 2 weeks ago

close as this is not a Kyuubi side issue