Closed emanueledomingo closed 3 months ago
I haven't deep-dived too much into rust code as I'm still a beginner rustacean, but it seems to me that the issue arises from the following lines: https://github.com/delta-io/delta-rs/blob/4da7d66d06985d386e61d3fb124cad6680594dcc/rust/src/storage/config.rs#L163-L176
The comparison between the env key and the storage_options key is done on the lowered version of the env key, resulting in bugs like the one @emanueledomingo reported.
Given the above comment, a temporary workaround is to pass already-lowered keys in the storage_options
Looks like it was introduced here: https://github.com/delta-io/delta-rs/commit/02b3cea58505212454509828660dd534eb2ba6e6.
A possible fix would be using a case-insensitive hash map unless there is a scenario when key case is important (I don't see one). If case insensitive hash map is not a good idea then changing the logic in with...
methods to loop through keys
performing case-insensitive comparison instead of using contains_key
I haven't checked the codebase fully, but did some work on that recently. Specifically the S3 code is still quite messy. IIRC the priority issue should actually be that if there are multiple credentials available the priority is based on the order in which object store checks credentials.
Unfortunately we cannot just try to get a credential, since there are configuration-less credentials (i.e. metadata endpoint) which means the logic to check if there is a valid credential w/o the environment is a bit more involved. I tried once, but ended up "guessing" too much what the user intend may be, while also not being too familiar with how AWS does things.
That said, we definitely need to do some work on the s3 config...
removed, different issue, unrelated
I stopped not getting this error in deltalake 0.16.1.
Yes i confrim that also on 0.16.0 the error does not occur. This issue can be closed.
Environment
Delta-rs version: 0.11.0
Binding: Python
Environment:
Bug
What happened:
I'm not able to write on AWS S3 using AssumeRole. Same code is working with deltalake 0.10.1.
What you expected to happen:
Write to AWS S3 using credentials in storage_options instead of environment ones
How to reproduce it:
More details:
Delta tries to connect using the environment variables instead of the storage options.