search

holdenk / spark-testing-base

Base classes to use when writing tests with Spark
Apache License 2.0
1.51k stars 359 forks source link

Replace deprecated SQLContext with SparkSession for DataFrame / DataSet / RDD generators #418

Open AndyLeGrand opened 3 months ago

AndyLeGrand commented 3 months ago

E.g. the DataframeGenerator.arbitraryDataFrame() method takes a SQLContext as parameter. According to the Spark docs [1], this has been deprecated since Spark 2.0 in favor of SparkSession. Would be nice to replace this.

[1] https://spark.apache.org/docs/latest/api/java/org/apache/spark/sql/SQLContext.html