Performance, Security, and Usability Improvements

[bsharifi]

Adds the following performance, security, and usability improvements as described in the CHANGELOG:

• Automatic translation and pushdown of Spark expressions down to Redshift for local execution (via new 'autopushdown' parameter)
• Upgrade JDBC driver from 1.x to 2.x for improved security and compatibility with Redshift
• IAM-based authentication
• AWS Secret Manager based authentication
• Parquet support for reading and writing
• S3 result caching (when autopushdown is enabled)
• Spark 3.4.0 support
• Improved cross-region support between Spark cluster and S3 bucket (via new 'tempdir_region' parameter)
• Automatic retry during Redshift COPY operations (via new 'copydelay' and 'copyretrycount' parameters)
• Ability to pass additional options during Redshift UNLOAD operations (via new 'extraunloadoptions' parameter)
• Microsecond precision when reading or writing time-based columns to/from Redshift.
• Complex type support for using Spark ArrayType, MapType, and StructType with Redshift SUPER data type.
• Support for parallel Redshift SQL execution during autopushdown.
• Support for passing JDBC parameters through the connector to the underlying JDBC driver (via new 'jdbc.*' parameter)
• Query group support for annotating Redshift queries from connector (via new 'label' parameter)

[smoy]

@bsharifi It's fantastic AWS contributing these set of new features and improvements. Is there published docs on how to setup some of those features?

IAM-based authentication AWS Secret Manager based authentication

Even if its from other AWS service, if its similar enough, I can take a stab on writing up the community usage documentation.