parente
commented
7 years ago I have no insight into what's wrong after poking and doing a bit of research. There have been similar reports of issues like this in the past, with slight variance in the details:
- https://groups.google.com/forum/#!topic/spark-users/FwXualTA_sQ
- https://issues.cloudera.org/browse/LIVY-62
Interestingly enough, the case class works fine when using DataFrames and Datasets:
import org.apache.spark.sql.types.{StructType, StructField, StringType, IntegerType}
val first_name = StructField("first_name", StringType)
val last_name = StructField("last_name", StringType)
val age = StructField("age", IntegerType)
val schema = StructType(Array(first_name, last_name, age))
val df = spark.read.schema(schema).csv("person.txt")
val ds = df.as[Persons]
ds.collect()
leeivan
walmaaoui
zhangfiona
the version of docker: jupyter/all-spark-notebook:lastest
the way to start docker: docker run -it --rm -p 8888:8888 jupyter/all-spark-notebook:latest or docker ps -a docker start -i containerID
the steps:
Visit http://localhost:8888 Start an spylon-kernal notebook input code above
the output:res0: Array[Array[String]] = Array(Array(Barack, Obama, 53), Array(George, Bush, 68), Array(Bill, Clinton, 68))
the error message:
The above code is working with the spark-shell. From error message, I speculated that the driver program didn't correctly handle case class Persons to RDD partition.