search

PredictionIO / template-scala-parallel-universal-recommendation

PredictiionIO Template for Universal Recommender
111 stars 48 forks source link

Spark Application Not Terminated #36

Closed ghost closed 8 years ago

ghost commented 8 years ago

The deployed engine doesn't use Spark, but a Spark application is still launched and remains running when the engine is deployed. The application can be killed without consequence (just error messages), but ideally it would do this automatically, or even better, not require Spark to deploy at all.

pferrel commented 8 years ago

Fixed in v0.3.0, you can pick it up from https://github.com/actionml/template-scala-parallel-universal-recommendation/tree/v0.3.0 https://github.com/actionml/template-scala-parallel-universal-recommendation/tree/v0.3.0

On Feb 20, 2016, at 9:39 AM, Justin Ramos notifications@github.com wrote:

The deployed engine doesn't use Spark, but a Spark application is still launched and remains running when the engine is deployed. The application can be killed without consequence (just error messages), but ideally it would do this automatically, or even better, not require Spark to deploy at all.

— Reply to this email directly or view it on GitHub https://github.com/PredictionIO/template-scala-parallel-universal-recommendation/issues/36.

pferrel commented 8 years ago

notice it's not in the PIO repos yet

ghost commented 8 years ago

Does that remove the need to have a Spark master running when deploying, or does it just prevent the Spark application from running?

ghost commented 8 years ago

Even with v0.3.0, the Spark application starts and continues running.

tobilg commented 8 years ago

I noticed the same behavior. The Spark app keeps running...

pferrel commented 8 years ago

The latest version creates an ephemeral context and will use master = local if you tell it to so no context is created on your cluster. You can pass in the master with pio train -- --master local I think, check the spark-submit cli and don't forget the blank -- as a separator.

BTW, sorry to miss these questions. If you post them to the root repo here you'll get noticed quicker

ghost commented 8 years ago

@pferrel -- that fixed it. Many thanks.