Open radhikabajaj123 opened 2 months ago
I see that you are submitting multiple jars. One is using an absolute path under /root
and others are using a relative path, which seems like it is maybe not intended?
Also, there is no need to submit the source jars or test source jars.
Could you try submitting just comet-spark-spark3.4_2.13-0.2.0-SNAPSHOT.jar
using an absolute path?
/root/datafusion-comet/spark/target/comet-spark-spark3.4_2.13-0.2.0-SNAPSHOT-sources.jar,
./datafusion-comet/spark/target/comet-spark-spark3.4_2.13-0.2.0-SNAPSHOT-test-sources.jar,
./datafusion-comet/spark/target/comet-spark-spark3.4_2.13-0.2.0-SNAPSHOT.jar,
./datafusion-comet/spark/target/original-comet-spark-spark3.4_2.13-0.2.0-SNAPSHOT-sources.jar,
./datafusion-comet/spark/target/original-comet-spark-spark3.4_2.13-0.2.0-SNAPSHOT.jar \
I had tried submitting just comet-spark-spark3.4_2.13-0.2.0-SNAPSHOT.jar
using an absolute path and that had also gave the same error.
spark.[driver|executor].extraClassPath
should be a semicolon-separated list of local jars with absolute local paths. spark-submit silently ignores errors in this config. That's why spark cannot find mentioned class in its classpath.
This example works for me
export JARS_LOCAL="/opt/spark-3.5.1/jars_ext/comet-spark-spark3.5_2.12-0.2.0-SNAPSHOT-210824.jar:/opt/spark-3.5.1/jars_ext/spark-metrics-3.5-1.0.0.jar";
spark-shell \
...
--conf spark.plugins=org.apache.spark.CometPlugin \
--conf spark.driver.extraClassPath=$JARS_LOCAL \
--conf spark.executor.extraClassPath=$JARS_LOCAL
...
Hi Nikita, thanks for the reply!
I am receiving the same error when I try submitting a single jar comet-spark-spark3.4_2.13-0.2.0-SNAPSHOT.jar
using an absolute local path.
It doesn't make sense. I also don't think this is related to Comet. Based on what you described, seems you cannot include any third-party classes through --jars
config.
Are you able to have any jar other than Comet in --jars
and import any class from it?
@radhikabajaj123 note that this local jar (with local path) should be present on all worker nodes of your cluster.
@radhikabajaj123
spark.[driver|executor].extraClassPath - is a part which will be added as classpath parameters so have restriction:
:
every time when you use spark-submit all libraries from --jars
will be loaded to local working dir, so you don't need to provide relative path, possible options:
--jars
parameterfor test purpose i recommend to use option 2 but for production better to use option 1, because you can rely on yarn-site.xml config and include this jars into classpath by default
Hello,
I am getting the following exception when running spark-submit:
These are the configurations I am using for spark-submit:
Does anyone have any insights as to what might be causing the error?