Maintenance Improvements

[bsharifi]

Adds the following improvements as described in the CHANGELOG:

• Support Spark 3.4.1 and 3.5.0
• Upgrade sbt to version 1.9
• Upgrade Redshift JDBC driver to version 2.1.0.18
• Resolve CVE-2023-2976 and CVE-2020-8908
• Cancel any running queries upon shutdown.
• Improve traceability between Spark and Redshift with introduction of 'trace_id' Spark configuration.
• Improve performance of integration tests by parallelizing their execution.
• Resolve 'expression_tree_walker' query exceptions for sort-limit-project queries.
• Fix issues with cross-cluster queries not working correctly with pushdown.