apache / iceberg-rust

Apache Iceberg
https://rust.iceberg.apache.org/
Apache License 2.0
672 stars 159 forks source link

Error running data fusion queries - Physical input schema should be the same as the one converted from logical input schema #655

Open a-agmon opened 1 month ago

a-agmon commented 1 month ago

I have a simple iceberg table that I can query using DataFusion

+----+----------+-----+------------+
| id | name     | age | dt         |
+----+----------+-----+------------+
| 2  | Jane     | 20  | 2024-01-02 |
| 1  | John     | 10  | 2021-01-02 |
...

However, whenever I run a query such as select count(*) from tbl or select name from tbl group by 1 I get the following exception:

Error: Internal error: Physical input schema should be the same as the one converted from logical input schema..

If this is reproducible with others, then I think we have some issue with the provider.

The code I use to run queries against the table:

let warehouse_location = String::from("/tmp/testwarehouse");
let file_io = FileIO::from_path(warehouse_location)?.build()?;
let metatdata_location = "/tmp/testwarehouse/test/test_table/metadata/v1.metadata.json";
let table_indent = TableIdent::from_strs(["test", "test_table"])?;
let static_table =
    StaticTable::from_metadata_file(metatdata_location, table_indent, file_io).await?;
let table = static_table.into_table();
let ctx = SessionContext::new();
let table_provider = IcebergTableProvider::try_new_from_table(table).await?;
ctx.register_table("tbl", Arc::new(table_provider))?;
let df = ctx.sql("select count(*) from tbl").await?;
df.show().await?;
FANNG1 commented 1 month ago

seems caused by https://github.com/apache/datafusion/issues/12687, I will propose a PR to upgrade datafusion version.

a-agmon commented 1 month ago

Thanks @FANNG1 FYI @sdd