search

apache / iceberg-python

Apache PyIceberg
https://py.iceberg.apache.org/
Apache License 2.0
309 stars 114 forks source link

fix: schema check of iceberg logical types #856

Open raphaelauv opened 1 week ago

raphaelauv commented 1 week ago

close: https://github.com/apache/iceberg-python/issues/855

raphaelauv commented 1 week ago

this is a trivial implementation , probably not enough. thanks all

raphaelauv commented 6 days ago

@kevinjqliu wdyt ?

kevinjqliu commented 6 days ago

I'm +1 on this change in theory. I feel like _check_schema_compatible should be as non-blocking as possible, i.e. if pyarrow can write the dataset, _check_schema_compatible should allow it.

I wonder if there's a more generalized solution for this instead of hardcoding UUID to FixedType(16) conversion. @syun64 @HonahX @Fokko wdyt

raphaelauv commented 6 days ago

thanks for the review @Fokko, to make it part of pyarrow_to_schema we must change a lot of things to propagate the table_schema ( that is an iceberg schema ) , that's what I tried first and then I reverted and made a separate function _apply_logical_conversion