Feature spark331 update

[zavoraad]

What changes are proposed in this pull request?

This pull request contains feature upgrades to Spark 3.3.1 support and passes all Scala based testing. Minimal to no code changes made unless needed for either upgrading Spark or passing Scala integration testing

How is this patch tested?

• [X] Unit tests (Scala)
• [X] Integration tests (Scala)
• [ ] Manual tests

(Details)

[a0x8o]

@zavoraad do any of the tests pass locally?

[zavoraad]

@zavoraad do any of the tests pass locally?

[a0x8o]

BGEN tests are failing in CircleCI

[error] Failed tests:
[error]     io.projectglow.bgen.BgenWriterSuite
[error]     io.projectglow.vcf.SingleFileVCFWriterSuite
[error] (core / Test / test) sbt.TestsFailedException: Tests unsuccessful

[zavoraad]

Not super familiar with Circe, but at the top of the logs indicates running Spark 3.2.1 instead of Spark 3.3.1.

#!/bin/bash -eo pipefail
export PATH=$HOME/conda/envs/glow/bin:$PATH
export SPARK_VERSION="3.2.1"
export SCALA_VERSION="2.12.8"
sbt core/test exit

[a0x8o]

Good exercise then would be to get up to speed with CircleCI and make the updates.

[zavoraad]

Sorry @a0x8o I won't have time to go further with this project. Figured I would share some basic cleanup and Spark 3.3.1 upgrades that some folks around Life Sciences were looking for with regards to extending Databricks Runtime Support.

Thinking someone could use this work as it needs done anyway to increment the DBR. If this isn't something needed or of interest please feel free to close this request.

[henrydavidge]

@zavoraad Apologies for not addressing this PR earlier. I recently merged a PR to support to Spark 3.4 and 3.5: #546. Please let us know if it suits your needs. And thank you for the contribution :). I'm going to close this PR now.