search

MotherDuck-Open-Source / duckdb-power-query-connector

DuckDB Power Query Custom Connector by MotherDuck
MIT License
38 stars 3 forks source link

Load Iceberg Tables #35

Open jkleinkauff opened 4 weeks ago

jkleinkauff commented 4 weeks ago

Hey friends,

Would be possible to load iceberg tables within this connector? (actually, this is a view on top of a iceberg_scan function)

image
elefeint commented 3 weeks ago

Are you connecting to plain DuckDB or to MotherDuck using this connector? Most likely, if connecting to "md:", the operation will be done on our backend where the iceberg extension is installed.

guenp commented 3 weeks ago

@jkleinkauff could you also share a repro of what you are trying to run (ideally the DuckDB SQL or Power Query statement you're running)?

jkleinkauff commented 3 weeks ago

In motherduck I have a view similar to

create view house_watcher_listings_view_iceberg_full as 
with cte as (
  select _airbyte_data::json as data_json
  from iceberg_scan('s3://bucket/bronze/iceberg/airbyte_raw_listings', skip_schema_inference=True)
)
select
    data_json->'data'->>'id' as id
    , data_json->'data'->>'sourceId' as sourceId
    , etc
from cte

Then I configure the connector with my motherduck db and token:

image

The connector will correctly list my tables and views:

image

I select the view I mentioned above and click "Load". I'm not waiting for the preview to complete:

image

Sync will start:

image

After a few seconds, the error pops out:

image

mez file:

image

and duckdb_odbc version 0.1.3

jkleinkauff commented 3 weeks ago

@guenp @elefeint I don't want to bother, just check if I may be doing something wrong. Thank you.

guenp commented 2 weeks ago

@jkleinkauff can you try upgrading your duckdb-odbc version to the latest one? see https://duckdb.org/docs/installation/?version=stable&environment=odbc&platform=win&download_method=direct&architecture=x86_64

jkleinkauff commented 2 weeks ago

The config was the same as above, but when trying to list the tables:

image