Add a Spark Jobserver Service and Connection to External Spark

benchflow / data-analyses-scheduler

A service scheduling Spark's data-transformers and analysers application according to messages published on Kafka's topics

Other

1 stars 0 forks source link

Add a Spark Jobserver Service and Connection to External Spark #20

Open VincenzoFerme opened 8 years ago

VincenzoFerme commented 8 years ago

TODO:

Implement as a scheduler, relying on already implemented solutions: https://github.com/ajvb/kala/blob/master/README.md, https://mesos.github.io/chronos/docs/
Add a spark-job-observer as a service: https://github.com/spark-jobserver/spark-jobserver
Have a test Dockerfile to do local runs with local Spark, just to test scripts

Cerfoglg commented 8 years ago

@VincenzoFerme Currently, because the Jobserver doesn't work with python on Spark (see https://github.com/spark-jobserver/spark-jobserver/issues/196), and even if it did it doesn't solve the issue of caching cross application for Spark, we instead opted to use Alluxio to have a memory speed caching alternative