[Spark] Mark DeltaConfig.DATA_SKIPPING_NUM_INDEXED_COLS lazy

delta-io / delta

An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs

https://delta.io

Apache License 2.0

7.62k stars 1.71k forks source link

[Spark] Mark DeltaConfig.DATA_SKIPPING_NUM_INDEXED_COLS lazy #3707

Closed HyukjinKwon closed 1 month ago

HyukjinKwon commented 2 months ago

Which Delta project/connector is this regarding?

[x] Spark
[ ] Standalone
[ ] Flink
[ ] Kernel
[ ] Other (fill in here)

Description

This PR proposes to mark DeltaConfig.DATA_SKIPPING_NUM_INDEXED_COLS lazy. Otherwise, it might ending up with null because of the initialization order. See the stacktrace below:

How was this patch tested?

Existing Ci should test it out.

Does this PR introduce any user-facing changes?

No.

HyukjinKwon commented 1 month ago

Can we merge this please?

juliuszsompolski commented 1 month ago

I am looking at another resolution. Do not merge for now.

juliuszsompolski commented 1 month ago

Please do not merge this PR.