Closed baddlan closed 6 years ago
Sparkler uses JVM system property pf4j.pluginsDir
to access plugins directory path.
As long as plugins directory is correctly set with that system property, it should work.
See for example, sparkler.sh for how it is set It simply adds-Dpf4j.pluginsDir=/Full/Path/to/dir
to command line
Note: since you are running it in a distributed fashion, pf4j.pluginsDir
should be set on all the workers. I know the documentation is lacking for this, and there are several ways to do it. Stackoverflow or Apache wiki should help
Thanks @thammegowda!
Explicitly setting the pluginsDir
property did the trick.
I was under the impression that Sparkler looks for plugins by default in $CWD\plugins
.
@buggtb Any ideas on how to deploy the Sparkler/pf4j plugins on a Spark cluster?
I managed to get the Sparkler app jar deployed and loaded by master and worker containers in the cluster but the executors fail to load the plugins (see error below). I have already tried copying the
plugins
jars and directory underSPARK_HOME
andSPARK_HOME/jars
with no luck.Environment and Version Information
-- Thanks