jstomphorst commented 3 years ago

Hi guys,

I am staring a thrift server, and that thrift server starten an exceuter.. But that exceuter wants to cimmunicate with the thrift server on weird ports. Below my errors and config.

Tnx!

CMD=(${JAVA_HOME}/bin/java "${SPARK_EXECUTOR_JAVA_OPTS[@]}" -Xms$SPARK_EXECUTOR_MEMORY -Xmx$SPARK_EXECUTOR_MEMORY -cp "$SPARK_CLASSPATH:$SPARK_DIST_CLASSPATH" org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url $SPARK_DRIVER_URL --executor-id $SPARK_EXECUTOR_ID --cores $SPARK_EXECUTOR_CORES --app-id $SPARK_APPLICATION_ID --hostname $SPARK_EXECUTOR_POD_IP)
exec /usr/bin/tini -s -- /usr/local/openjdk-8/bin/java -Dspark.driver.port=38267 -Xms512M -Xmx512M -cp ':/opt/spark/jars/*:' org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url spark://CoarseGrainedScheduler@spark-thrift-server:38267 --executor-id 232 --cores 1 --app-id spark-application-1624009104722 --hostname 10.233.67.42 Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties 21/06/18 09:57:46 INFO CoarseGrainedExecutorBackend: Started daemon with process name: 13@spark-sql-c64ed37a1e7c59dc-exec-232 21/06/18 09:57:46 INFO SignalUtils: Registered signal handler for TERM 21/06/18 09:57:46 INFO SignalUtils: Registered signal handler for HUP 21/06/18 09:57:46 INFO SignalUtils: Registered signal handler for INT 21/06/18 09:57:47 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 21/06/18 09:57:47 INFO SecurityManager: Changing view acls to: 185 21/06/18 09:57:47 INFO SecurityManager: Changing modify acls to: 185 21/06/18 09:57:47 INFO SecurityManager: Changing view acls groups to: 21/06/18 09:57:47 INFO SecurityManager: Changing modify acls groups to: 21/06/18 09:57:47 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(185); groups with view permissions: Set(); users with modify permissions: Set(185); groups with modify permissions: Set() Exception in thread "main" java.lang.reflect.UndeclaredThrowableException at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1761) at org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:61) at org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:283) at org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:272) at org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala) Caused by: org.apache.spark.SparkException: Exception thrown in awaitResult: at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:302) at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:75) at org.apache.spark.rpc.RpcEnv.setupEndpointRefByURI(RpcEnv.scala:101) at org.apache.spark.executor.CoarseGrainedExecutorBackend$.$anonfun$run$3(CoarseGrainedExecutorBackend.scala:303) at scala.runtime.java8.JFunction1$mcVI$sp.apply(JFunction1$mcVI$sp.java:23) at scala.collection.TraversableLike$WithFilter.$anonfun$foreach$1(TraversableLike.scala:877) at scala.collection.immutable.Range.foreach(Range.scala:158) at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:876) at org.apache.spark.executor.CoarseGrainedExecutorBackend$.$anonfun$run$1(CoarseGrainedExecutorBackend.scala:301) at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:62) at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:61) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1746) ... 4 more Caused by: java.io.IOException: Failed to connect to spark-thrift-server/10.233.35.136:38267 at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:253) at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:195) at org.apache.spark.rpc.netty.NettyRpcEnv.createClient(NettyRpcEnv.scala:204) at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:202) at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:198) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: spark-thrift-server/10.233.35.136:38267 Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:714) at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330) at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:702) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) root@itr-dsp-ot-k8sm001:/home/root@itr-dsp-ot-k8sm001:/home/intraffic# client_loop: sen

with kubernetes config.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: spark-thrift-server
  #  namespace: spark-operator
  labels:
    foo: bar
spec:
  replicas: 1
  selector:
    matchLabels:
      foo: bar
  template:
    metadata:
      labels:
        foo: bar
    spec:
      containers:
      - name: spark-thrift-server
        image: gcr.io/spark-operator/spark:v3.0.0
        args:
          -  /opt/spark/bin/spark-submit
          - --master
          - k8s://https://kubernetes.default.svc.itr-dsp-ot-k8s.privatehybridcloud.eu:443
          - --class
          - org.apache.spark.sql.hive.thriftserver.HiveThriftServer2
          - --deploy-mode
          - client
          - --name
          - spark-thrift
          - --hiveconf
          - hive.server2.thrift.port 10000
          - --conf
          - spark.executor.instances=1
          - --conf
          - spark.executor.memory=512M
          - --conf
          - spark.driver.memory=512M
          - --conf
          - spark.executor.cores=1
          - --conf
          - spark.kubernetes.namespace=default
          - --conf
          - spark.kubernetes.container.image=gcr.io/spark-operator/spark:v3.0.0
          - --conf
          - spark.kubernetes.authenticate.driver.serviceAccountName=spark
          - --conf
          - spark.kubernetes.driver.pod.name=$(THRIFT_POD_NAME)
          - --conf
          - spark.driver.bindAddress=$(THRIFT_POD_IP)
          - --conf
          - spark.driver.host=spark-thrift-server
        env:
        - name: THRIFT_POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: THRIFT_POD_IP
          valueFrom:
            fieldRef:
              fieldPath: status.podIP
        ports:
        - containerPort: 4040
          name: spark-ui
          protocol: TCP
        - containerPort: 10000
          name: spark-thrift
          protocol: TCP
      serviceAccount: spark
      serviceAccountName: spark
---
apiVersion: v1
kind: Service
metadata:
  name: spark-thrift-server
  #namespace: spark-operator
spec:
  ports:
  - name: spark-ui
    port: 4040
    protocol: TCP
    targetPort: 4040
  - name: spark-thrift
    port: 10000
    protocol: TCP
    targetPort: 10000
  - name: spark-thrift1
    port: 44477
    protocol: TCP
    targetPort: 10000
  selector:
    foo: bar
  sessionAffinity: None
  type: LoadBalancer

jdonnelly-apixio commented 3 years ago

@jstomphorst Not positive, but I think you might need spark.hadoop.hive.server2.thrift.port defined to 10k also. My working args section:

          - /opt/spark/bin/spark-submit
          - --master
          - k8s://https://$(KUBERNETES_SERVICE_HOST):443
          - --class
          - org.apache.spark.sql.hive.thriftserver.HiveThriftServer2
          - --deploy-mode
          - client
          - --name
          - spark-sql
          - --hiveconf
          - hive.server2.thrift.port 10000
          - --conf
          - spark.executor.instances=1
          - --conf
          - spark.executor.memory=2G
          - --conf
          - spark.driver.memory=2G
          - --conf
          - spark.executor.cores=2
          - --conf
          - spark.kubernetes.namespace=spark-operator
          - --conf
          - spark.kubernetes.container.image=xxx/spark-app:v1.0.3
          - --conf
          - spark.kubernetes.authenticate.driver.serviceAccountName=spark-operator
          - --conf
          - spark.kubernetes.driver.pod.name=$(THRIFT_POD_NAME)
          - --conf
          - spark.driver.bindAddress=$(THRIFT_POD_IP)
          - --conf
          - spark.hadoop.hive.metastore.client.connect.retry.delay=5
          - --conf
          - spark.hadoop.hive.metastore.client.socket.timeout=1800
          - --conf
          - spark.hadoop.hive.metastore.uris=thrift://my-metastore:9083
          - --conf
          - spark.hadoop.hive.server2.enable.doAs=false
          - --conf
          - spark.hadoop.hive.server2.thrift.port=10000
          - --conf
          - spark.hadoop.hive.server2.transport.mode=binary
          - --conf
          - spark.hadoop.metastore.catalog.default=spark
          - --conf
          - spark.hadoop.hive.execution.engine=spark
          - --conf
          - spark.hadoop.hive.input.format=io.delta.hive.HiveInputFormat
          - --conf
          - spark.hadoop.hive.tez.input.format=io.delta.hive.HiveInputFormat
          - --conf
          - spark.sql.warehouse.dir=s3a://xxx \
          - --conf
          - spark.hadoop.fs.defaultFS=s3a://xxx \
          - --conf
          - spark.hadoop.fs.s3a.connection.ssl.enabled=true \
          - --conf
          - spark.hadoop.fs.s3a.endpoint=https://s3.us-west-2.amazonaws.com \
          - --conf
          - spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem \
          - --conf
          - spark.hadoop.fs.s3a.fast.upload=true \
          - --conf
          - spark.hadoop.fs.s3a.path.style.access=true \

jstomphorst commented 3 years ago

I've found the problem

apiVersion: v1 kind: Service metadata: name: spark-thrift-server

namespace: spark-operator

spec: ports:

name: spark-ui port: 4040 protocol: TCP targetPort: 4040
name: spark-thrift port: 10000 protocol: TCP targetPort: 10000
name: spark-thrift1 port: 44477 protocol: TCP targetPort: 10000 selector: foo: bar sessionAffinity: None type: LoadBalancer

The Thriftserver needs a direct connection

apiVersion: v1 kind: Service metadata: name: spark-thrift-server spec: clusterIP: None ports:

name: spark-ui port: 4040 protocol: TCP targetPort: 4040
name: spark-thrift port: 10000 protocol: TCP targetPort: 10000 selector: foo: bar sessionAffinity: None type: ClusterIP

so my solution is, 2 services:

ind: Service metadata: name: spark-thrift-server spec: clusterIP: None ports:

name: spark-ui port: 4040 protocol: TCP targetPort: 4040
name: spark-thrift port: 10000 protocol: TCP targetPort: 10000 selector: foo: bar sessionAffinity: None type: ClusterIP
type: LoadBalancer

apiVersion: v1 kind: Service metadata: name: spark-thrift-server-connector spec: ports:
name: spark-ui port: 4040 protocol: TCP targetPort: 4040
name: spark-thrift port: 10000 protocol: TCP targetPort: 10000 selector: foo: bar sessionAffinity: None type: LoadBalancer

github-actions[bot] commented 6 days ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

kubeflow / spark-operator

executer cannot communicate with Thrift server #1283

I've found the problem

namespace: spark-operator

type: LoadBalancer