finos / legend-engine

Legend Engine module
https://legend.finos.org
Apache License 2.0
80 stars 233 forks source link

DuckDB iceberg_scan post processor #3066

Open rafaelbey opened 2 weeks ago

rafaelbey commented 2 weeks ago

This propose how we can support iceberg_scan thru duckdb.

The proposal has two parts: S3 auth, and iceberg_scan.

For S3 authentication, at the moment there is a new DuckDBS3AuthenticationStrategy, and when executing, is responsible of:

The second part is the connection post processor. This post processor translate table names following some rules:

Few challenges/questions/todos:

github-actions[bot] commented 2 weeks ago

Test Results

     837 files       837 suites   56m 11s :stopwatch:   8 726 tests   8 659 :heavy_check_mark: 66 :zzz: 1 :x: 13 220 runs  13 153 :heavy_check_mark: 66 :zzz: 1 :x:

For more details on these failures, see this check.

Results for commit 3e496025.