search

pingcap / tispark

TiSpark is built for running Apache Spark on top of TiDB/TiKV
Apache License 2.0
884 stars 244 forks source link

springboot + tispark spark driver 和executor时区不一致问题 #1513

Closed cln19921024 closed 4 years ago

cln19921024 commented 4 years ago

使用springboot + tispark编写demo tispark跑在centos系统上，测试demo跑在本机windows环境下。 查询数据库信息报如下错误。 com.pingcap.tikv.exception.TiInternalException: timezone are different! dirver: +08:00 executor:Asia/Shanghai please set user.timezone in spark.driver.extraJavaOptions and spark.executor.extraJavaOptions at org.apache.spark.sql.tispark.TiRDD$$anon$2.(TiRDD.scala:62) ~[tispark-core-2.1.9-spark_2.4.jar:2.1.9-spark_2.4] at org.apache.spark.sql.tispark.TiRDD.compute(TiRDD.scala:60) ~[tispark-core-2.1.9-spark_2.4.jar:2.1.9-spark_2.4] at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:346) ~[spark-core_2.11-2.4.6.jar:2.4.6] at org.apache.spark.rdd.RDD.iterator(RDD.scala:310) ~[spark-core_2.11-2.4.6.jar:2.4.6] at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52) ~[spark-core_2.11-2.4.6.jar:2.4.6] at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:346) ~[spark-core_2.11-2.4.6.jar:2.4.6] at org.apache.spark.rdd.RDD.iterator(RDD.scala:310) ~[spark-core_2.11-2.4.6.jar:2.4.6] at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52) ~[spark-core_2.11-2.4.6.jar:2.4.6] at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:346) ~[spark-core_2.11-2.4.6.jar:2.4.6] at org.apache.spark.rdd.RDD.iterator(RDD.scala:310) ~[spark-core_2.11-2.4.6.jar:2.4.6] at org.apache.spark.rdd.UnionRDD.compute(UnionRDD.scala:105) ~[spark-core_2.11-2.4.6.jar:2.4.6] at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:346) ~[spark-core_2.11-2.4.6.jar:2.4.6] at org.apache.spark.rdd.RDD.iterator(RDD.scala:310) ~[spark-core_2.11-2.4.6.jar:2.4.6] at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52) ~[spark-core_2.11-2.4.6.jar:2.4.6] at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:346) ~[spark-core_2.11-2.4.6.jar:2.4.6] at org.apache.spark.rdd.RDD.iterator(RDD.scala:310) ~[spark-core_2.11-2.4.6.jar:2.4.6] at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52) ~[spark-core_2.11-2.4.6.jar:2.4.6] at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:346) ~[spark-core_2.11-2.4.6.jar:2.4.6] at org.apache.spark.rdd.RDD.iterator(RDD.scala:310) ~[spark-core_2.11-2.4.6.jar:2.4.6] at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90) ~[spark-core_2.11-2.4.6.jar:2.4.6] at org.apache.spark.scheduler.Task.run(Task.scala:123) ~[spark-core_2.11-2.4.6.jar:2.4.6] at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408) ~[spark-core_2.11-2.4.6.jar:2.4.6] at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360) ~[spark-core_2.11-2.4.6.jar:2.4.6] at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414) ~[spark-core_2.11-2.4.6.jar:2.4.6] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_131] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_131] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_131] 按照提示要求配置user.timezone 无效，检查发现源码中默认获取系统时区。所以导致无法连通

cln19921024 commented 4 years ago

通过修改springboot默认时区 TimeZone.setDefault((TimeZone.getTimeZone("Asia/Shanghai"))); 问题已解决