search

Qihoo360 / Quicksql

A Flexible, Fast, Federated(3F) SQL Analysis Middleware for Multiple Data Sources
https://quicksql.readthedocs.io
MIT License
2.06k stars 583 forks source link

do spark-submit job cause spark worker exited due to a missing qsql jar exception #19

Closed louis83 closed 5 years ago

louis83 commented 5 years ago

when i do this command "./spark-submit --master spark://master_url:7077 --deploy-mode cluster --class com.qihoo.qsql.CsvJoinWithEsExample qsql-example-0.5.jar" , then show the following exception:

Exception in thread "main" java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.worker.DriverWrapper$.main(DriverWrapper.scala:65) at org.apache.spark.deploy.worker.DriverWrapper.main(DriverWrapper.scala) Caused by: java.lang.NoClassDefFoundError: com/qihoo/qsql/utils/PropertiesReader at com.qihoo.qsql.env.RuntimeEnv.(RuntimeEnv.java:17) at com.qihoo.qsql.CsvJoinWithEsExample.main(CsvJoinWithEsExample.java:11) ... 6 more Caused by: java.lang.ClassNotFoundException: com.qihoo.qsql.utils.PropertiesReader at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 8 more

Functor10 commented 5 years ago

It is recommended that you execute example via the script run-example com.qihoo.qsql.CsvJoinWithEsExample. Executing example by spark-submit may fail to load related Quicksql jars. In addition, could you please join the QQ group, we will provide more timely technical support.

louis83 commented 5 years ago

It is recommended that you execute example via the script run-example com.qihoo.qsql.CsvJoinWithEsExample. Executing example by spark-submit may fail to load related Quicksql jars. In addition, could you please join the QQ group, we will provide more timely technical support.

Does it mean that it can only be run locally?

Functor10 commented 5 years ago

It is recommended that you execute example via the script run-example com.qihoo.qsql.CsvJoinWithEsExample. Executing example by spark-submit may fail to load related Quicksql jars. In addition, could you please join the QQ group, we will provide more timely technical support.

Does it mean that it can only be run locally?

For this example it is like your said. The reason is that Quicksql will start an embedded Elasticsearch server in the example, we are worried that the Spark cluster accessing the embedded server together will cause the application to CRASH. Therefore, it is recommended to test the example through the local mode. Of course, Quicksql itself can run on spark cluster, depending on your option --master in qsql command.