How to get verbose logs?

mmaitre314 commented 4 months ago

Is it possible to get verbose logs from the DuckDB Azure extension? In particular logs which would show what happened during authentication?

I am hitting an auth error trying to query a blob in an Azure Storage Account. I am guessing my config is wrong, or maybe the Azure CLI auth failed for some reason, but the error message is not giving enough to root-cause the issue.

Notebook repro:

%sql duckdb:///:memory:

%sql CREATE SECRET azduckdb (TYPE AZURE, PROVIDER CREDENTIAL_CHAIN, SCOPE 'az://azduckdb.blob.core.windows.net/')

%sql SELECT count(*) FROM 'az://azduckdb.blob.core.windows.net/data/HuggingFaceFW/fineweb/CC-MAIN-2013-20/000_00000.parquet'

RuntimeError: (duckdb.duckdb.IOException) IO Error: AzureBlobStorageFileSystem open file 'az://azduckdb.blob.core.windows.net/data/HuggingFaceFW/fineweb/CC-MAIN-2013-20/000_00000.parquet' failed with code'',
Reason Phrase: 'Server failed to authenticate the request. Please refer to the information in the www-authenticate header.',
Message: ''

(in GitHub: https://github.com/mmaitre314/az-duckdb/blob/main/main.ipynb)

quentingodeau commented 4 months ago

Greetings,

I think you can try with AZURE_LOG_LEVEL

Regards, Quentin

mmaitre314 commented 4 months ago

Thanks! That did the trick. Somehow the logs were not getting displayed in VSCode Notebook but it worked running a plain py file. I'll see if I can expand the docs to mention logging.

For future reference:

import os
import duckdb

os.environ['AZURE_LOG_LEVEL'] = 'verbose'

duckdb.sql("CREATE SECRET myaccount (TYPE AZURE, PROVIDER CREDENTIAL_CHAIN, SCOPE 'az://myaccount.blob.core.windows.net/')")
duckdb.sql("SELECT count(*) FROM 'az://myaccount.blob.core.windows.net/path/to/blob.parquet'")

duckdb / duckdb_azure

How to get verbose logs? #63