search

pingcap / tispark

TiSpark is built for running Apache Spark on top of TiDB/TiKV
Apache License 2.0
880 stars 244 forks source link

[BUG] Fail to request TiFlash in TiSpark v3.2.1 #2707

Closed shiyuhang0 closed 1 year ago

shiyuhang0 commented 1 year ago

Version Spark: any TiSpark:v3.2.1

Describe the bug

Use TiSpark read from TiFlash with this configuration spark.tispark.isolation_read_engines.

We will find the spark fails with 

3/05/18 14:49:38 WARN RangeSplitter: Cannot find valid store on TiFlash(engine=tiflash)
23/05/18 14:49:38 WARN RangeSplitter: getRegionStorePairByKey error
java.lang.NullPointerException: fail to get store in regionStorePair by key 116,128,0,0,0,0,2,144,16,95,114,1,48,48,52,53,56,55,55,56,255,53,50,0,0,0,0,0,0,249
        at com.pingcap.tikv.util.RangeSplitter.splitRangeByRegion(RangeSplitter.java:199)
        at org.apache.spark.sql.tispark.TiRDD.getPartitions(TiRDD.scala:58)
        at org.apache.spark.rdd.RDD.$anonfun$partitions$2(RDD.scala:292)
        at scala.Option.getOrElse(Option.scala:189)
        at org.apache.spark.rdd.RDD.partitions(RDD.scala:288)
        at org.apache.spark.SparkContext.$anonfun$union$2(SparkContext.scala:1434)
        at org.apache.spark.SparkContext.$anonfun$union$2$adapted(SparkContext.scala:1434)
        at scala.collection.TraversableLike.noneIn$1(TraversableLike.scala:319)
        at scala.collection.TraversableLike.filterImpl(TraversableLike.scala:385)
        at scala.collection.TraversableLike.filterImpl$(TraversableLike.scala:297)
        at scala.collection.AbstractTraversable.filterImpl(Traversable.scala:108)
        at scala.collection.TraversableLike.filter(TraversableLike.scala:395)
        at scala.collection.TraversableLike.filter$(TraversableLike.scala:395)
        at scala.collection.AbstractTraversable.filter(Traversable.scala:108)

Root cause https://github.com/tikv/client-java/issues/749