vincenzobaz / spark-scala3

Apache License 2.0
89 stars 15 forks source link

schema is empty when trying to create it in a method #58

Closed kostaskougios closed 10 months ago

kostaskougios commented 10 months ago

I have a weird issue where the schema is created OK when using it straight on with a case class but is empty when I try to create a more generic method:

import org.apache.spark.sql.{DataFrame, Dataset, Encoder, SparkSession}

import scala.deriving.Mirror.ProductOf
import scala.reflect.ClassTag

class SparkSessionExt(spark: SparkSession):
  import scala3encoders.given
  import spark.implicits.*

  def schemaOf[P: ClassTag: ProductOf] = summon[Encoder[P]].schema

Then I have a

case class Person(id: String, name: String)

and this test fails:

  test("schemaOf"):
    Using.resource(Sessions.newSparkSession()): spark =>
      val sp     = new SparkSessionExt(spark)
      val schema = sp.schemaOf[Person]
      println(schema)
      schema.toList.size should be(2)

the println prints StructType()

Is this a bug or am I doing something wrong?

kostaskougios commented 10 months ago

ok seems it is my mistake, this works:

  def schemaOf[P: Encoder] = summon[Encoder[P]].schema

ProductOf is something the compilation error pointed me at, but the correct one to use is Encoder

kostaskougios commented 10 months ago

closing