search

microsoft / Mobius

C# and F# language binding and extensions to Apache Spark
MIT License
941 stars 213 forks source link

Can't connect to DataStax Enterprise using Mobius 1.6.2 #598

Closed Aaronontheweb closed 7 years ago

Aaronontheweb commented 8 years ago

Working on running the built-in Cassandra sample to connect to a local DataStax Enterprise installation I've setup, and using the 1.6.2 release branch of the source code (the latest version of Spark supported by DataStax) I'm not able to successfully submit any of my jobs due to the following error:

JVM method execution failed: Constructor failed for class org.apache.spark.SparkContext when called with 1
[2016-11-26 17:32:04,071] [1] [ERROR] [Microsoft.Spark.CSharp.Interop.Ipc.JvmBridge] - JVM method execution failed: Constructor failed for class org.apache.spark.SparkContext when called with 1 parameters ([Index=1, Type=JvmObjectReference, Value=2], ) [2016-11-26 17:32:04,073] [1] [ERROR] [Microsoft.Spark.CSharp.Interop.Ipc.JvmBridge] - ******************************************************************************************************************************* at Microsoft.Spark.CSharp.Interop.Ipc.JvmBridge.CallJavaMethod(Boolean isStatic, Object classNameOrJvmObjectReference, String methodName, Object[] parameters) ******************************************************************************************************************************* Unhandled Exception: System.Exception: JVM method execution failed: Constructor failed for class org.apache.spark.SparkContext when called with 1 parameters ([Index=1, Type=JvmObjectReference, Value=2], ) at Microsoft.Spark.CSharp.Interop.Ipc.JvmBridge.CallJavaMethod(Boolean isStatic, Object classNameOrJvmObjectReference, String methodName, Object[] parameters) at Microsoft.Spark.CSharp.Interop.Ipc.JvmBridge.CallConstructor(String className, Object[] parameters) at Microsoft.Spark.CSharp.Proxy.Ipc.SparkCLRIpcProxy.CreateSparkContext(ISparkConfProxy conf) at Microsoft.Spark.CSharp.Core.SparkContext..ctor(String master, String appName, String sparkHome, SparkConf conf) at Microsoft.Spark.CSharp.Core.SparkContext..ctor(SparkConf conf) at Microsoft.Spark.CSharp.Examples.CassandraDataFrameExample.Main(String[] args) in C:\Repositories\Mobius\examples\Sql\CassandraDataFrame\Program.cs:line 55

I've followed the instructions related to submitting all of the necessary JARs in order to connect to Cassandra with no effect - here's what my commandline looks like:

C:\Repositories\Mobius> .\scripts\sparkclr-submit.cmd --master spark://10.0.69.2:7077 --jars "C:\Repositories\clients\qs\qs-etd\cluster-setup\spark-config\dse-config.jar;C:\Repositories\clients\qs\qs-etd\cluster-setup\spark-config\cassandra-driver-core-3.0.2.jar;C:\Repositories\clients\qs\qs-etd\cluster-setup\spark-config\guava-19.0.jar;C:\Repositories\clients\qs\qs-etd\cluster-setup\spark-config\spark-cassandra-connector-1.6.0-s_2.10.jar;C:\Repositories\clients\qs\qs-etd\cluster-setup\spark-config\jsr166e-1.1.0.jar" --exe csdriver.exe "C:\Repositories\Mobius\examples\Sql\CassandraDataFrame\bin\Debug\CassandraDataFrame.exe"

Running this on Windows 10 from PowerShell, with all of the proper environment variables resolving correctly. Any idea what could be going on here?

skaarthik commented 8 years ago

Can you share Spark log messages as well? It will help identify why creating SparkContext failed.

BTW, are you able to use Mobius with Spark in local mode or use spark-shell.cmd to connect to your Spark cluster? These steps will help validating code locally and connectivity with remote Spark cluster before submitting your code to the remote cluster.

Aaronontheweb commented 7 years ago

@skaarthik where can I find the Spark logs for this? I assumed it was all being written out ephemerally to standard out.

I can use spark-shell.cmd and the DataStax Spark / Cassandra connector to contact our remote cluster.

skaarthik commented 7 years ago

@Aaronontheweb check out https://www.mapr.com/blog/how-log-apache-spark for info on logging in Spark.

Are you using the correct version of Spark (1.6.*) for the version of Mobius (1.6.2) you are using?

Aaronontheweb commented 7 years ago

I'll give this another try - got the Spark Shell to connect to DSE on 1.6.2.

skaarthik commented 7 years ago

@Aaronontheweb - closing this issue due to inactivity. Feel free to reopen if needed.