The driver hang when we use multiple processes trying to access the XSQLSessionCatalog in XSQL, We print jstack and find a DeadLock. as follows:
"Thread-491" #895 daemon prio=5 os_prio=0 tid=0x000000000219b000 nid=0x3ebf waiting for monitor entry [0x00007f17bb2ee000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.spark.sql.xsql.XSQLExternalCatalog.setWorkingDataSource(XSQLExternalCatalog.scala:591)
- waiting to lock <0x000000066bfb9cb8> (a java.util.concurrent.atomic.AtomicReference)
at org.apache.spark.sql.xsql.XSQLSessionCatalog.setWorkingDataSource(XSQLSessionCatalog.scala:150)
at org.apache.spark.sql.xsql.XSQLSessionCatalog.lookupRelation(XSQLSessionCatalog.scala:443)
- locked <0x000000066be8f0f0> (a org.apache.spark.sql.xsql.XSQLSessionCatalog)
at org.apache.spark.sql.xsql.XSQLSessionStateBuilder$XSQLResolveRelations$.org$apache$spark$sql$xsql$XSQLSessionStateBuilder$XSQLResolveRelations$$lookupTableFromCatalog(XSQLSessionStateBuilder.scala:283)
at org.apache.spark.sql.xsql.XSQLSessionStateBuilder$XSQLResolveRelations$.resolveRelation(XSQLSessionStateBuilder.scala:236)
at org.apache.spark.sql.xsql.XSQLSessionStateBuilder$XSQLResolveRelations$$anonfun$apply$1.applyOrElse(XSQLSessionStateBuilder.scala:266)
at org.apache.spark.sql.xsql.XSQLSessionStateBuilder$XSQLResolveRelations$$anonfun$apply$1.applyOrElse(XSQLSessionStateBuilder.scala:259)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$$anonfun$resolveOperatorsUp$1$$anonfun$apply$1.apply(AnalysisHelper.scala:90)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$$anonfun$resolveOperatorsUp$1$$anonfun$apply$1.apply(AnalysisHelper.scala:90)
"Thread-490" #894 daemon prio=5 os_prio=0 tid=0x0000000002199000 nid=0x3ebe waiting for monitor entry [0x00007f17bb9f4000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.spark.sql.catalyst.catalog.SessionCatalog.lookupFunction(SessionCatalog.scala:1276)
- waiting to lock <0x000000066be8f0f0> (a org.apache.spark.sql.xsql.XSQLSessionCatalog)
at org.apache.spark.sql.hive.HiveSessionCatalog.org$apache$spark$sql$hive$HiveSessionCatalog$$super$lookupFunction(HiveSessionCatalog.scala:131)
at org.apache.spark.sql.hive.HiveSessionCatalog$$anonfun$3.apply(HiveSessionCatalog.scala:131)
at org.apache.spark.sql.hive.HiveSessionCatalog$$anonfun$3.apply(HiveSessionCatalog.scala:131)
at scala.util.Try$.apply(Try.scala:192)
at org.apache.spark.sql.hive.HiveSessionCatalog.lookupFunction0(HiveSessionCatalog.scala:131)
at org.apache.spark.sql.hive.HiveSessionCatalog.lookupFunction(HiveSessionCatalog.scala:117)
at org.apache.spark.sql.xsql.XSQLSessionCatalog.org$apache$spark$sql$xsql$XSQLSessionCatalog$$super$lookupFunction(XSQLSessionCatalog.scala:801)
at org.apache.spark.sql.xsql.XSQLSessionCatalog$$anonfun$lookupFunction$1.apply(XSQLSessionCatalog.scala:801)
at org.apache.spark.sql.xsql.XSQLSessionCatalog$$anonfun$lookupFunction$1.apply(XSQLSessionCatalog.scala:801)
at org.apache.spark.sql.xsql.XSQLExternalCatalog.setWorkingDataSource(XSQLExternalCatalog.scala:598)
- locked <0x000000066bfb9cb8> (a java.util.concurrent.atomic.AtomicReference)
at org.apache.spark.sql.xsql.XSQLSessionCatalog.setWorkingDataSource(XSQLSessionCatalog.scala:150)
at org.apache.spark.sql.xsql.XSQLSessionCatalog.lookupFunction(XSQLSessionCatalog.scala:800)
Found 1 deadlock.
What changes were proposed in this pull request?
The driver hang when we use multiple processes trying to access the XSQLSessionCatalog in XSQL, We print jstack and find a DeadLock. as follows:
This PR solve this DeadLock.
How was this patch tested?
No UT.