Fix for the thread contention issue

memsql / singlestore-spark-connector

A connector for SingleStore and Spark

Apache License 2.0

160 stars 54 forks source link

Fix for the thread contention issue #90

Closed AdalbertMemSQL closed 4 months ago

AdalbertMemSQL commented 4 months ago

* Changed retry during reading from result table to use exponential backoff
* Used ForkJoinPool instead of FixedThreadPool
* Added more logging

pmishchenko-ua commented 4 months ago

what's the motivation for using ForkJoinPool instead of FixedThreadPool?

AdalbertMemSQL commented 4 months ago

ForkJoinPool is designed to be used for CPU-intensive workloads. A customer suggested using it to improve situations when we run too many reading threads inside of one task.