search

delta-io / delta-rs

A native Rust library for Delta Lake, with bindings into Python
https://delta-io.github.io/delta-rs/
Apache License 2.0
1.98k stars 365 forks source link

fix(rust, python): fixed differences in storage options between log and object stores #2500

Closed mightyshazam closed 1 month ago

mightyshazam commented 1 month ago

The object store changes the keys used for creating the s3 options, while the log store does not. The difference between the two, and their usage together was causing some weird fallback behavior in the AWS SDK that invoked the IMDS service despite explicitly disabling it in multiple places. This fix handles the potentially different casing along with passing the options unmolested. It should stop any future, unwanted, IMDS behavior

Description

The description of the main changes of your pull request

Related Issue(s)