wrong number of arguments at pickler

[sporto]

Hi I copied the code verbatim from https://github.com/vincenzobaz/spark-scala3/blob/main/examples/src/main/scala/sql/StarWars.scala

But I'm getting

[error] 34 |    ).toDS.map((n,f) => Friends(n, f))
[error]    |          ^
[error]    |wrong number of arguments at pickler for (targetObject: org.apache.spark.sql.catalyst.expressions.Expression,
[error]    |  functionName
[error]    |: String, dataType: org.apache.spark.sql.types.DataType, arguments:
[error]    |  Seq[org.apache.spark.sql.catalyst.expressions.Expression]
[error]    |, methodInputTypes: Seq[org.apache.spark.sql.types.AbstractDataType],
[error]    |  propagateNull
[error]    |: Boolean, returnNullable: Boolean, isDeterministic: Boolean):
[error]    |  org.apache.spark.sql.catalyst.expressions.objects.Invoke: (org.apache.spark.sql.catalyst.expressions.objects.Invoke.apply :
[error]    |  (targetObject: org.apache.spark.sql.catalyst.expressions.Expression,
[error]    |    functionName
[error]    |  : String, dataType: org.apache.spark.sql.types.DataType, arguments:
[error]    |    Seq[org.apache.spark.sql.catalyst.expressions.Expression]
[error]    |  , methodInputTypes: Seq[org.apache.spark.sql.types.AbstractDataType],
[error]    |    propagateNull
[error]    |  : Boolean, returnNullable: Boolean, isDeterministic: Boolean):
[error]    |    org.apache.spark.sql.catalyst.expressions.objects.Invoke
[error]    |), expected: 8, found: 6
[error]    | This location contains code that was inlined from Serializer.scala:68

I'm using Scala 3.1.1 and Spark 3.2.1 Any help would be very appreciated

Thanks

[vincenzobaz]

Hello @sporto , thank you for reporting this and I am sorry for the late reply. I forgot to update the README, could you try with version 0.1.3 instead of 0.1.2?

[sporto]

Thanks, I won't be able to try it for the moment as I reverted to Scala 2 given the issues I was hitting with Scala 3. Might try again in the future.

[dylandoamaral]

I still have this issue using Spark 3.3.1 in 0.1.3 using a simple final case class Person(name: String, age: Long).

[vincenzobaz]

The library is using 3.2.0. Could you try with this version to check if this is the issue? I guess it's time to upgrade spark and scala in this project 😓

[dylandoamaral]

You are right, I guessed that a minor would not influence the result my bad ! We just need to push https://github.com/vincenzobaz/spark-scala3/pull/21 and make a release ?

[vincenzobaz]

yes

[carlos-verdes]

I'm using last version of Spark and having this issue @vincenzobaz I can try to fix it if you give me some guidance

My code uses a simple:

case class GenderCount(gender: String, genderCount: Long)

And having the same issue (expecting 8 arguments, passed 7)

[error] -- Error: /Users/cverdes/git/dataverse-dataproduct-engine-spark-runtime/spark/src/it/scala/me/xsight/productengine/spark/runtime/EtlRunnerIT.scala:38:83 
[error] 38 |          val count = spark.read.parquet("data/userEmails.parquet").as[GenderCount]
[error]    |                                                                                   ^
[error]    |wrong number of arguments at pickler for (targetObject: org.apache.spark.sql.catalyst.expressions.Expression, 
[error]    |  functionName
[error]    |: String, dataType: org.apache.spark.sql.types.DataType, arguments: 
[error]    |  Seq[org.apache.spark.sql.catalyst.expressions.Expression]
[error]    |, methodInputTypes: Seq[org.apache.spark.sql.types.AbstractDataType], 
[error]    |  propagateNull
[error]    |: Boolean, returnNullable: Boolean, isDeterministic: Boolean): 
[error]    |  org.apache.spark.sql.catalyst.expressions.objects.Invoke: (org.apache.spark.sql.catalyst.expressions.objects.Invoke.apply : 
[error]    |  (targetObject: org.apache.spark.sql.catalyst.expressions.Expression, 
[error]    |    functionName
[error]    |  : String, dataType: org.apache.spark.sql.types.DataType, arguments: 
[error]    |    Seq[org.apache.spark.sql.catalyst.expressions.Expression]
[error]    |  , methodInputTypes: Seq[org.apache.spark.sql.types.AbstractDataType], 
[error]    |    propagateNull
[error]    |  : Boolean, returnNullable: Boolean, isDeterministic: Boolean): 
[error]    |    org.apache.spark.sql.catalyst.expressions.objects.Invoke
[error]    |), expected: 8, found: 7
[error]    |----------------------------------------------------------------------------
[error]    |Inline stack trace
[error]    |- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
[error]    |This location contains code that was inlined from Serializer.scala:68
[error]     ----------------------------------------------------------------------------
[error] one error found
[error] (spark / IntegrationTest / compileIncremental) Compilation failed
[error] Total time: 2 s, completed 12 Dec 2022, 3:49:41 PM

[vincenzobaz]

Sorry for the late reply. I am running a publish for version 0.1.4 with spark 3.3.0