Multiple spark drivers / one spree instance ?

hammerlab / spree

Live-updating Spark UI built with Meteor

Apache License 2.0

190 stars 29 forks source link

Multiple spark drivers / one spree instance ? #72

Open davvdg opened 7 years ago

davvdg commented 7 years ago

Hi,

Here we are trying to do multitenant spark, so we have several drivers running concurently on our cluster. How should I setup Spree for that ?

should I start one spree instance per driver or can one spree instance deals with all drivers ?
Aren't the incoming event messed up ? is there a way to determine from which driver each event comes ?
Can several spree instance share the same db ?

Regards

David

sadikovi commented 7 years ago

Hi David,

I have not used it in multitenant environment, but I believe that just running single instance of Spree should be enough. You just need to launch each driver programme with correct slim address and listener. Each event is shipped with app id, which is unique across Spark applications that belong to the same cluster.

Is there a way to test it on your installation? Would be useful to keep a note of installing in multitenant env with potential gotchas:)