Cursor does not inherit S3 configuration when using SET/PRAGMA statements.

[mustafahasankhan]

What happens?

When using CALL load_aws_credentials(); cursor inherits S3 configuration but when we are using SET/PRAGMA statements to set S3 configuration it does not. Ideally the behaviour should be same.

For other configuration properties like for example temp_directory, cursor inherits correctly.

To Reproduce

import duckdb
import os

con = duckdb.connect('file.db')

con.execute("CALL load_aws_credentials()")

conn_result = con.execute("SELECT current_setting('s3_secret_access_key');").fetchone()
print(f"Connection result: {conn_result[0]}")

cursor = con.cursor()

cursor.execute("SELECT current_setting('s3_secret_access_key');")
cursor_result = cursor.fetchone()
print(f"Cursor result: {cursor_result[0]}")

cursor.close()
con.close()

con = duckdb.connect('file.db')

hmac_access_key = os.environ['HMAC_ACCESS_KEY']
hmac_secret = os.environ['HMAC_SECRET']
con.execute("PRAGMA s3_endpoint='storage.googleapis.com';")
con.execute(f"PRAGMA s3_access_key_id='{hmac_access_key}';")
con.execute(f"PRAGMA s3_secret_access_key='{hmac_secret}';")

conn_result = con.execute("SELECT current_setting('s3_secret_access_key');").fetchone()
print(f"Connection result: {conn_result[0]}")

cursor = con.cursor()

cursor.execute("SELECT current_setting('s3_secret_access_key');")
cursor_result = cursor.fetchone()
print(f"Cursor result: {cursor_result[0]}")

cursor.close()
con.close()

The output looks like: Connection result: actualaccessKey Cursor result: actualaccessKey Connection result: actualaccessKey Cursor result: None

OS:

macOS Sonoma

DuckDB Version:

0.9.1

DuckDB Client:

Python

Full Name:

Mustafa Hasan Khan

Affiliation:

Atlan

Have you tried this on the latest main branch?

I have tested with a main build

Have you tried the steps to reproduce? Do they include all relevant data and configuration? Does the issue you report still appear there?

• [X] Yes, I have

[mustafahasankhan]

@szarnyasg Do we have any updates on it? Is there any discussion that I may follow, thanks.

[samansmink]

Hey @mustafahasankhan! I'm going to pick up this issue, but we're in the middle of a bit of a rework to the mechanism of these types of configuration in DuckDB. I will take a look into this after that takes shape. Any updates will be in this issue!

[samansmink]

@mustafahasankhan I stumbled upon the solution here.

The AWS extensions sets the parameters globally (in the DBConfig) while the SET/PRAGMA statement sets them locally by default. However, you can set extension parameters globally using SET GLOBAL s3_<setting> this should achieve the desired behaviour. This isn't really documented at the moment so thats something I will look into.

The load_aws_credentials should probably have an option to set globally or locally. However, the load_aws_credentials will be replaced soon, so i will close this issue for now.