[INFRA] Fix and create new workflow to compile delta-spark examples - Githubissues

delta-io / delta

An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs

https://delta.io

Apache License 2.0

6.96k stars 1.58k forks source link

[INFRA] Fix and create new workflow to compile delta-spark examples #3011

Closed scottsand-db closed 2 weeks ago

scottsand-db commented 2 weeks ago

Which Delta project/connector is this regarding?

[ ] Spark
[ ] Standalone
[ ] Flink
[ ] Kernel
[X] Other (INFRA)

Description

Create new workflow spark_examples_test.yaml that compiles the examples/scala/build.sbt project.

Also fixes such compilation so that it uses the local jars (previously it was incorrectly hardcoded to Delta 3.0).

This requires running publishM2 beforehand.

How was this patch tested?

CI tests and specifically tested on a commit that uses a new clusterBy API in 3.2: https://github.com/delta-io/delta/pull/3012

Does this PR introduce any user-facing changes?

No.