Update to Spark 3.4/3.5

projectglow / glow

An open-source toolkit for large-scale genomic analysis

https://projectglow.io

Apache License 2.0

266 stars 111 forks source link

Update to Spark 3.4/3.5 #546

Closed henrydavidge closed 8 months ago

henrydavidge commented 8 months ago

What changes are proposed in this pull request?

Update supported Spark versions to 3.4 and 3.5
Remove Hail integration
Remove features that frequently cause incompatibilities between versions (aggregate_by_index, CSV pipe transformer). Workarounds will be documented in a follow-up PR.
Update Scala and Python dependencies to match recent DBR releases
Fix unit tests that are failing in master
Clean up test logs

How is this patch tested?

[x] Unit tests
[ ] Integration tests
[x] Manual tests

(Details)

henrydavidge commented 8 months ago

Merging since all tests pass (except for the known-broken notebook tests)

a-li commented 8 months ago

Thanks for completing the Spark upgrade! @henrydavidge @williambrandler Are we planning a release to get this into maven repo? If so, what is the approximate timeline? We are planning to integrate with the upgraded Spark build ASAP once it's available on Maven.

henrydavidge commented 8 months ago

@a-li There will definitely be a maven release, but I'm planning on adding a few features so it might be a couple weeks. If you want to try it out before then, you follow these instructions: https://github.com/projectglow/glow?tab=readme-ov-file#testing-code-on-a-databricks-cluster. The artifacts built this way can be used anywhere, not just in Databricks.