search

spark-redshift-community / spark-redshift

Performant Redshift data source for Apache Spark
Apache License 2.0
135 stars 62 forks source link

Add a flag to disable superfluous S3 checks #144

Open zaza opened 10 months ago

zaza commented 10 months ago

In order to squeeze as much out of the connector as possible, we would like to have a way to disable certain superfluous (in our opinion) S3 checks like Utils.checkThatBucketHasObjectLifecycleConfiguration.

There's definitely value in having them on by default, but once addressed, it would be great to have an option to turn them off, so no extra calls are made except for the essentials.

bsharifi commented 9 months ago

@zaza Thank you for the suggestion on ways to improve the connector. While I cannot promise any specific timeframes, we are considering this suggestion for future improvements to the connector. As always, please feel free to contribute your own code improvements as a PR and we will be happy to review and merge it.

zaza commented 9 months ago

Thanks @bsharifi ! I will definitely consider contributing a fix. I already created https://github.com/spark-redshift-community/spark-redshift/pull/148 which is related.