duckdb / duckdb_delta

DuckDB extension for Delta Lake
MIT License
139 stars 15 forks source link

can not load delta table from minio #45

Closed huaxk closed 4 months ago

huaxk commented 4 months ago

I set up a minio server and placed some delta data in url: s3://deltalake/nyc_taxi, execute the following query:

CREATE SECRET (
  TYPE S3,
  KEY_ID 'xxx',
  SECRET 'xxx',
  ENDPOINT 'minio.xxx.com',
  REGION '',
  USE_SSL true,
  URL_STYLE 'path'
);

select * from delta_scan('s3://deltalake/nyc_taxi');

the following error occurs:

IO Error: Hit DeltaKernel FFI error (from: While trying to read from delta table: 's3://deltalake/nyc_taxi/'): Hit error: 8 (ObjectStoreError) with message (Error interacting with object store: Generic S3 error: Error after 10 retries in 1.842569293s, max_retries:10, retry_timeout:180s, source:error sending request for url (https://s3..amazonaws.com/deltalake/nyc_taxi/_delta_log/_last_checkpoint): error trying to connect: Invalid Server Name)

ENDPOINT parameters do not work, it still connected to amazonaws.com url.

set REGION to 'us-east-1', then execute the quey, the following error occurs:

IO Error: Hit DeltaKernel FFI error (from: While trying to read from delta table: 's3://deltalake/nyc_taxi/'): Hit error: 8 (ObjectStoreError) with message (Error interacting with object store: Generic S3 error: Received redirect without LOCATION, this normally indicates an incorrectly configured region)

It seems that the ENDPOINT parameters still don't work, and minio has no region. What went wrong? But when I query the parquet file with delta lake, there is no problem:

select * from **read_parquet('s3://deltalake/nyc_taxi/0-fe638ed0-e514-4c6b-887d-e3b2637df81c-0.parquet');
huaxk commented 4 months ago

Maybe the same problem #14