Open zaza opened 10 months ago
@zaza Thank you for the suggestion on ways to improve the connector. While I cannot promise any specific timeframes, we are considering this suggestion for future improvements to the connector. As always, please feel free to contribute your own code improvements as a PR and we will be happy to review and merge it.
Thanks @bsharifi ! I will definitely consider contributing a fix. I already created https://github.com/spark-redshift-community/spark-redshift/pull/148 which is related.
In order to squeeze as much out of the connector as possible, we would like to have a way to disable certain superfluous (in our opinion) S3 checks like
Utils.checkThatBucketHasObjectLifecycleConfiguration
.There's definitely value in having them on by default, but once addressed, it would be great to have an option to turn them off, so no extra calls are made except for the essentials.