search

juanrh / sscheck

ScalaCheck for Spark
Apache License 2.0
63 stars 9 forks source link

Improve concurrency of DynSeqQueueInputDStream.compute() #7

Closed juanrh closed 8 years ago

juanrh commented 9 years ago

Consider using dstreams.par.foldLeft instead of dstreams.foldLeft in DynSeqQueueInputDStream.compute(). Using ParSeq is ok here because we don't grant relative order of different test cases. The only problem here is studying how many parallel tests cases we need for this to be an improvement, and how to configure the task support / execution context http://docs.scala-lang.org/overviews/parallel-collections/configuration.html to get the best performance

juanrh commented 8 years ago

Doesn't apply anymore, as now we are using testinputdstream from spark-testing-base