Closed Shmuma closed 1 year ago
Root cause of this issue is concurrent creation of subconnections from the cached main connection (lines Cache hit, reuse existing
below):
23/08/28 08:15:36 INFO BlockManagerInfo: Removed broadcast_1_piece0 on 30c3b58ca6ef:34223 in memory (size: 7.5 KiB, free: 434.4 MiB)
23/08/28 08:15:36 INFO BlockManagerInfo: Removed broadcast_0_piece0 on 30c3b58ca6ef:34223 in memory (size: 7.5 KiB, free: 434.4 MiB)
23/08/28 08:15:36 INFO ExasolRelation: Creating Spark RDD from Exasol query 'SELECT "COL1", "COL2" FROM (select * from TEST.TEST)'.
23/08/28 08:15:36 INFO ExasolRelation: Creating Spark RDD from Exasol query 'SELECT "COL1", "COL2" FROM (select * from TEST.TEST)'.
23/08/28 08:15:36 INFO CodeGenerator: Code generated in 25.363054 ms
23/08/28 08:15:36 INFO ExasolConnectionManager: Making a connection using url = jdbc:exa:10.0.0.12:8563;validateservercertificate=0
23/08/28 08:15:36 INFO ExasolConnectionManager: Cache hit, reusing existing
23/08/28 08:15:36 INFO ExasolRDD: Initiated 2 parallel exasol (sub) connections
23/08/28 08:15:36 INFO ExasolRDD: Executing enriched query 'SELECT "COL1", "COL2" FROM (select * from TEST.TEST)'.
23/08/28 08:15:36 INFO ExasolRDD: The number of partitions is 2
23/08/28 08:15:36 INFO ExasolConnectionManager: Making a connection using url = jdbc:exa:10.0.0.12:8563;validateservercertificate=0
23/08/28 08:15:36 INFO ExasolConnectionManager: Cache hit, reusing existing
23/08/28 08:15:36 INFO ExasolRDD: Initiated 2 parallel exasol (sub) connections
23/08/28 08:15:36 INFO ExasolRDD: Executing enriched query 'SELECT "COL1", "COL2" FROM (select * from TEST.TEST)'.
23/08/28 08:15:36 INFO ExasolRDD: The number of partitions is 2
23/08/28 08:15:36 INFO SparkContext: Starting job: collect at <console>:1
In case of several datanodes in the DB installation, reading from two tables end up in "connection refused" error
To reproduce, database with more than one datanode is needed. Table
TEST.TEST
has some rows: