I would now like to create a new DataFrame for further processing. When I do:
ranks.toDF("id", "pageRank")
I get the error:
Exception in thread "main" java.lang.IllegalArgumentException: kotlin.Any! is unsupported
at org.jetbrains.kotlinx.spark.api.EncodingKt.schema(Encoding.kt:359)
at org.jetbrains.kotlinx.spark.api.EncodingKt.schema(Encoding.kt:309)
at org.jetbrains.kotlinx.spark.api.EncodingKt.schema$default(Encoding.kt:186)
at org.jetbrains.kotlinx.spark.api.EncodingKt$memoizedSchema$1.invoke(Encoding.kt:368)
at org.jetbrains.kotlinx.spark.api.EncodingKt$memoizedSchema$1.invoke(Encoding.kt:367)
at org.jetbrains.kotlinx.spark.api.Memoize1.invoke(Encoding.kt:377)
at org.jetbrains.kotlinx.spark.api.EncodingKt.generateEncoder(Encoding.kt:146)
I'm not sure why this would not be allowed or what I should do to work around. Any hints are appreciated.
Apologies if this is a newbie question.
I have successfully created a Spark job that performs a page rank algorithm using Spark and GraphX. I'm able to print out the page ranks like:
I would now like to create a new DataFrame for further processing. When I do:
I get the error:
I'm not sure why this would not be allowed or what I should do to work around. Any hints are appreciated.