search

memsql / singlestore-spark-connector

A connector for SingleStore and Spark
Apache License 2.0
160 stars 54 forks source link

New Spark version >= 2.0.0 doesn't have dataframes, these are replaced with dataset. So df.saveToMemSQL Not working. #36

Closed vinayairan closed 7 years ago

vinayairan commented 7 years ago

New Spark version >= 2.0.0 doesn't have dataframes, these are replaced with dataset. So df.saveToMemSQL Not working.

Exception in thread "main" java.lang.NoSuchMethodError: com.memsql.spark.connector.package$.DataFrameFunctions(Lorg/apache/spark/sql/Dataset;)Lcom/memsql/spark/connector/package$DataFrameFunctions; at com.olx.guru.KafkaWordCount$.main(KafkaWordCount.scala:130) at com.olx.guru.KafkaWordCount.main(KafkaWordCount.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:738) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

hhoughgg commented 7 years ago

Spark 2.0.0+ has DataFrames still.

val spark = SparkSession
  .builder()
  .getOrCreate()

  import spark.implicits._

  myRDD.toDF(dfSchema: _*).saveToMemSQL("database", "table")

This code works for me in Spark 2.1.0