search

delta-io / delta

An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs
https://delta.io
Apache License 2.0
7.62k stars 1.71k forks source link

[SPARK] Use better TahoeLogFileIndex constructor #3677

Closed scovich closed 2 months ago

scovich commented 2 months ago

Which Delta project/connector is this regarding?

Description

Tiny bit of code hygiene: Remove some redundant code from DeltaSharingFileIndex by invoking the existing alternate constructor for TahoeLogFileIndex that does the same thing.

How was this patch tested?

Existing tests cover this code.

Does this PR introduce any user-facing changes?

No.

scovich commented 2 months ago

It looks like some variant-related changes broke upstream DSM job?

 [error] /home/runner/work/delta/delta/spark/src/test/scala-spark-master/org/apache/spark/sql/delta/DeltaVariantSuite.scala:101:7: overloaded method checkError with alternatives:
[error]   (exception: org.apache.spark.SparkThrowable,condition: String,sqlState: Option[String],parameters: Map[String,String],context: DeltaVariantSuite.this.ExpectedContext)Unit <and>
[error]   (exception: org.apache.spark.SparkThrowable,condition: String,sqlState: String,context: DeltaVariantSuite.this.ExpectedContext)Unit <and>
[error]   (exception: org.apache.spark.SparkThrowable,condition: String,parameters: Map[String,String],context: DeltaVariantSuite.this.ExpectedContext)Unit <and>
[error]   (exception: org.apache.spark.SparkThrowable,condition: String,sqlState: String,parameters: Map[String,String],context: DeltaVariantSuite.this.ExpectedContext)Unit <and>
[error]   (exception: org.apache.spark.SparkThrowable,condition: String,sqlState: String,parameters: Map[String,String])Unit <and>
[error]   (exception: org.apache.spark.SparkThrowable,condition: String,sqlState: Option[String],parameters: Map[String,String],matchPVals: Boolean,queryContext: Array[DeltaVariantSuite.this.ExpectedContext])Unit
[error]  [which have no such parameter errorClass] cannot be applied to (org.apache.spark.SparkThrowable, errorClass: String, parameters: scala.collection.immutable.Map[String,String])
[error]       checkError(
[error]       ^