search

duckdb / duckdb_aws

MIT License
40 stars 14 forks source link

`duckdb.connect()` `config` argument doesn't accept any S3 configuration options in Python API #50

Open nicklausroach opened 1 month ago

nicklausroach commented 1 month ago

When trying to add s3-specific configurations to the connection config, I'm met with any of:

duckdb.duckdb.InvalidInputException: Invalid Input Error: Unrecognized configuration property "s3_access_key_id" duckdb.duckdb.InvalidInputException: Invalid Input Error: Unrecognized configuration property "s3_secret_access_key" duckdb.duckdb.InvalidInputException: Invalid Input Error: Unrecognized configuration property "s3_region"

I could be doing something wrong here, but all other config options seem to work other than the s3-related options. Example snippet:

>>> duckdb.connect(database=':memory:', read_only=False, config={'s3_secret_access_key': ''})
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
duckdb.duckdb.InvalidInputException: Invalid Input Error: Unrecognized configuration property "s3_secret_access_key"
samansmink commented 1 month ago

This indeed does not work because at startup the httpfs extension is not yet loaded! please use secrets instead!

nicklausroach commented 1 month ago

Ah I knew it was something stupid I was doing! That may be worth calling out in the docs, I assumed I could add those s3 configurations at connection time and then load the httpfs extension. Regardless, thank you for your help @samansmink!