databricks / spark-perf

Performance tests for Apache Spark
Apache License 2.0
380 stars 203 forks source link

problem fine tuning stand-alone mode worker instances #105

Closed dotnwat closed 8 years ago

dotnwat commented 8 years ago

I am running spark-perf on a scale-up node with v1.6.0 in stand-alone mode, and I'm having trouble fine tuning the runs. The problem I am currently facing is that I'd like to run multiple workers (smaller JVM heaps) on a single node. But, no matter how I am configuring spark-perf I see:

So, I am able to get multiple worker instances fired up, but I only see one JVM running an actual test (e.g. glm-regression). Am I messing up or missing a configuration option? The other issue I'm curious about is the coarse-grained executors. I have not seen them running before any time that I've used stand-alone mode.

Thanks.

dotnwat commented 8 years ago

Argh... caught again by terminology. Closing this, as there is really no problem.