Open rwhaling opened 2 hours ago
@rwhaling and this worked in 0.19.x?
@ion-elgreco No idea, doing this for the first time. I can try with 0.19.
Thank you for the reproduction case! With a fresh environment I am consistently getting Unable to locate credentials
The problem is coming from boto3
My guess is that you may have environment variables set that boto3 is picking up, which are different from what are being passed as storage options into deltalake
with
Thank you! I seem to get the same thing on 0.19.2 as well. Let me check out those environment vars. (Yes, I did have the AWS env vars set as well, apologies)
And so I understand - is write_deltalake using boto3 internally? Is there a way for me to turn up the logging?
s3 = boto3.client('s3', aws_access_key_id=access_key, aws_secret_access_key=secret_key, endpoint_url=endpoint_url)
Gets the repro case to the error message you describe
@rwhaling don't worry about trying to reproduce this on older versions, I found the error :smile: It exists going back many versions!
This was a good Sunday morning brain exercise!
The problem here is that the stack is expecting TLS communication. Add AWS_ALLOW_HTTP
as "true"
to the storage_options
and you'll be sorted!
If you're feeling extra thankful, I would love a pull request to update any relevant documentation in the docs/
directory which would have helped you here :pray:
Bingo, it works! I love writing doc PR's, would be happy to - and thank y'all for this great project!
Environment
Delta-rs version: 0.20.0
Binding: Python
Environment:
Bug
What happened: Running Minio locally via docker-compose (.yml spec below), attempted to write a 20-row Pyarrow table via the write_deltalake function, and got the opaque error message:
Generic S3 error: Error after 0 retries in 71.583µs, max_retries:10, retry_timeout:180s, source:builder error for url (http://localhost:9000/test-bucket/test_delta_table/_delta_log/_last_checkpoint)
Attempted to write a 20-row pandas dataframe via the polars write_delta function as well, and got the exact same error:
Generic S3 error: Error after 0 retries in 71.583µs, max_retries:10, retry_timeout:180s, source:builder error for url (http://localhost:9000/test-bucket/test_delta_table/_delta_log/_last_checkpoint)
What you expected to happen: I expected to be able to write tables out to Minio via S3. I have tested that I can write to Minio just fine with boto3. I'm happy to do more footwork chasing this down, turning up logging, or reproducing it deeper in the stack if someone can point me in the right direction!
How to reproduce it:
More details: docker-compose.yml: