PARQUET-2461: Upgrade ZSTD-JNI to 1.5.6-2

[cxzl25]

Apr 4, 2024 https://github.com/luben/zstd-jni/releases/tag/v1.5.6-2

Mar 28, 2024 https://github.com/luben/zstd-jni/releases/tag/v1.5.6-1

Dec 1, 2023 https://github.com/luben/zstd-jni/releases/tag/v1.5.5-11

Jira

• [x] https://issues.apache.org/jira/browse/PARQUET-2461

Tests

• [ ] My PR adds the following unit tests OR does not need testing for this extremely good reason:

Commits

• [ ] My commits all reference Jira issues in their subject lines. In addition, my commits follow the guidelines from "How to write a good git commit message":
  1. Subject is separated from body by a blank line
  2. Subject is limited to 50 characters (not including Jira issue reference)
  3. Subject does not end with a period
  4. Subject uses the imperative mood ("add", not "adding")
  5. Body wraps at 72 characters
  6. Body explains "what" and "why", not "how"

Style

• [ ] My contribution adheres to the code style guidelines and Spotless passes.
  • To apply the necessary changes, run mvn spotless:apply -Pvector-plugins

Documentation

• [ ] In case of new functionality, my PR adds documentation that describes how to use it.
  • All the public functions and the classes in the PR contain Javadoc that explain what it does

[wgtmac]

@gszadovszky @Fokko

We see build errors for 1.14.0-SNAPSHOT against Apache Spark branch.3.5:

[warn] multiple main classes detected: run 'show discoveredMainClasses' to see the list
[error] java.lang.RuntimeException: found version conflict(s) in library dependencies; some are suspected to be binary incompatible:
[error] 
[error]     * com.github.luben:zstd-jni:1.5.6-2 (strict) is selected over {1.5.5-4}
[error]         +- org.apache.parquet:parquet-hadoop:1.14.0-SNAPSHOT  (depends on 1.5.6-2)
[error]         +- org.apache.spark:spark-core_2.12:3.5.2-SNAPSHOT    (depends on 1.5.5-4)
[error] 

I'm not sure if this is a blocking issue for the release.

[wgtmac]

FYI: zstd-jni versions on different branches of Apache Spark as of 2024-04-29: master: 1.5.6-3 branch-3.5: 1.5.5-4 branch-3.4: 1.5.2-5

[Fokko]

@wgtmac Thanks for raising this. Looking at the history, there seem te be some important patches in there: https://github.com/luben/zstd-jni/commits/master/?before=c77a7658aeba94ccd3da52e61ce87d06e7292826+35

We can't backport this to the 3.5 branch anyway. So, we're probably good with keeping this in line with the Spark main branch. WDYT?

[wgtmac]

So we just need to pair 1.14.0 with Apache Spark 4.0.0? I'm not sure if this is a good idea. It looks better if Apache Spark 3.5.2 can adopt Parquet 1.14.0.

[Fokko]

I think they don't allow backporting dependency upgrades in Spark unless there are CVE's :(

[wgtmac]

I'm not sure if that is the case. For Apache ORC, we use the following mapping to pair it with Apache Spark and always update the latest minor version to Apache Spark:

• Apache ORC 2.0.x with Apache Spark 4.0.y
• Apache ORC 1.9.x with Apache Spark 3.5.y
• Apache ORC 1.8.x with Apache Spark 3.4.y