Qihoo360 / XSQL

Unified SQL Analytics Engine Based on SparkSQL
https://qihoo360.github.io/XSQL/
Apache License 2.0
209 stars 62 forks source link

[Core] Fix deadLock between XSQLSessionCatalog and workingDataSource #68

Closed wenfang6 closed 4 years ago

wenfang6 commented 4 years ago

What changes were proposed in this pull request?

The driver hang when we use multiple processes trying to access the XSQLSessionCatalog in XSQL, We print jstack and find a DeadLock. as follows:

"Thread-491" #895 daemon prio=5 os_prio=0 tid=0x000000000219b000 nid=0x3ebf waiting for monitor entry [0x00007f17bb2ee000]
   java.lang.Thread.State: BLOCKED (on object monitor)
    at org.apache.spark.sql.xsql.XSQLExternalCatalog.setWorkingDataSource(XSQLExternalCatalog.scala:591)
    - waiting to lock <0x000000066bfb9cb8> (a java.util.concurrent.atomic.AtomicReference)
    at org.apache.spark.sql.xsql.XSQLSessionCatalog.setWorkingDataSource(XSQLSessionCatalog.scala:150)
    at org.apache.spark.sql.xsql.XSQLSessionCatalog.lookupRelation(XSQLSessionCatalog.scala:443)
    - locked <0x000000066be8f0f0> (a org.apache.spark.sql.xsql.XSQLSessionCatalog)
    at org.apache.spark.sql.xsql.XSQLSessionStateBuilder$XSQLResolveRelations$.org$apache$spark$sql$xsql$XSQLSessionStateBuilder$XSQLResolveRelations$$lookupTableFromCatalog(XSQLSessionStateBuilder.scala:283)
    at org.apache.spark.sql.xsql.XSQLSessionStateBuilder$XSQLResolveRelations$.resolveRelation(XSQLSessionStateBuilder.scala:236)
    at org.apache.spark.sql.xsql.XSQLSessionStateBuilder$XSQLResolveRelations$$anonfun$apply$1.applyOrElse(XSQLSessionStateBuilder.scala:266)
    at org.apache.spark.sql.xsql.XSQLSessionStateBuilder$XSQLResolveRelations$$anonfun$apply$1.applyOrElse(XSQLSessionStateBuilder.scala:259)
    at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$$anonfun$resolveOperatorsUp$1$$anonfun$apply$1.apply(AnalysisHelper.scala:90)
    at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$$anonfun$resolveOperatorsUp$1$$anonfun$apply$1.apply(AnalysisHelper.scala:90)
"Thread-490" #894 daemon prio=5 os_prio=0 tid=0x0000000002199000 nid=0x3ebe waiting for monitor entry [0x00007f17bb9f4000]
   java.lang.Thread.State: BLOCKED (on object monitor)
    at org.apache.spark.sql.catalyst.catalog.SessionCatalog.lookupFunction(SessionCatalog.scala:1276)
    - waiting to lock <0x000000066be8f0f0> (a org.apache.spark.sql.xsql.XSQLSessionCatalog)
    at org.apache.spark.sql.hive.HiveSessionCatalog.org$apache$spark$sql$hive$HiveSessionCatalog$$super$lookupFunction(HiveSessionCatalog.scala:131)
    at org.apache.spark.sql.hive.HiveSessionCatalog$$anonfun$3.apply(HiveSessionCatalog.scala:131)
    at org.apache.spark.sql.hive.HiveSessionCatalog$$anonfun$3.apply(HiveSessionCatalog.scala:131)
    at scala.util.Try$.apply(Try.scala:192)
    at org.apache.spark.sql.hive.HiveSessionCatalog.lookupFunction0(HiveSessionCatalog.scala:131)
    at org.apache.spark.sql.hive.HiveSessionCatalog.lookupFunction(HiveSessionCatalog.scala:117)
    at org.apache.spark.sql.xsql.XSQLSessionCatalog.org$apache$spark$sql$xsql$XSQLSessionCatalog$$super$lookupFunction(XSQLSessionCatalog.scala:801)
    at org.apache.spark.sql.xsql.XSQLSessionCatalog$$anonfun$lookupFunction$1.apply(XSQLSessionCatalog.scala:801)
    at org.apache.spark.sql.xsql.XSQLSessionCatalog$$anonfun$lookupFunction$1.apply(XSQLSessionCatalog.scala:801)
    at org.apache.spark.sql.xsql.XSQLExternalCatalog.setWorkingDataSource(XSQLExternalCatalog.scala:598)
    - locked <0x000000066bfb9cb8> (a java.util.concurrent.atomic.AtomicReference)
    at org.apache.spark.sql.xsql.XSQLSessionCatalog.setWorkingDataSource(XSQLSessionCatalog.scala:150)
    at org.apache.spark.sql.xsql.XSQLSessionCatalog.lookupFunction(XSQLSessionCatalog.scala:800)
Found 1 deadlock.

This PR solve this DeadLock.

How was this patch tested?

No UT.

beliefer commented 4 years ago

Good issue.

beliefer commented 4 years ago

Merged to master and branch-0.6. Thanks!