http://www.russellspitzer.com/2017/02/27/Concurrency-In-Spark/ for concurrency in spark and it has a "
A singleton object that controls the parallelism on a Single Executor JVM", also some code for Cassandra which Im not suggesting we add at this point
looks like repartitioning may be best and quickest fix as done in IndexUtil.write
http://www.russellspitzer.com/2017/02/27/Concurrency-In-Spark/ for concurrency in spark and it has a " A singleton object that controls the parallelism on a Single Executor JVM", also some code for Cassandra which Im not suggesting we add at this point
looks like repartitioning may be best and quickest fix as done in IndexUtil.write
I am trying this on EMR with entryLevel