search

apache / iceberg-python

Apache PyIceberg
https://py.iceberg.apache.org/
Apache License 2.0
342 stars 122 forks source link

Getting Original Schema of a DataFile in a FileScanTask? #401

Open srilman opened 5 months ago

srilman commented 5 months ago

Question

Is there a recommended way to getting the base / original schema or schema-id of a data file in a FileScanTask returned during FileTableScan.plan_files? This is useful to determine what kind of schema evolution occurred with the subset of files we are reading, and group files together with the same schemas for reads.

I had a hard time accomplishing this in the Java library, but found it much easier to do in Python. In plan_files, we can get the snapshot id a data file was created by looking at the snapshot_id of the associated manifest entry (or added_snapshot_id of the manifest list if the previous is null). From there, we can get the associated schema per snapshot.

Is this a logical approach, or is there a better way to get the original schema? Happy to open a PR to integrate this into FileTableScan if it would be useful!

srilman commented 5 months ago

@Fokko any thoughts?

Fokko commented 5 months ago

Hey @srilman sorry for not replying earlier. It slipped somewhere through the cracks, thanks for pinging me!

Is there a recommended way to getting the base / original schema or schema-id of a data file in a FileScanTask returned during FileTableScan.plan_files? This is useful to determine what kind of schema evolution occurred with the subset of files we are reading, and group files together with the same schemas for reads.

Unfortunately, no :(

From there, we can get the associated schema per snapshot.

A snapshot can have files with multiple schemas. See the example in https://github.com/duckdb/duckdb_iceberg/issues/40

Is this a logical approach, or is there a better way to get the original schema? Happy to open a PR to integrate this into FileTableScan if it would be useful!

The only source of truth is in the Parquet file. The crux is that you don't want to do lookups based on names, but only on field field-IDs. Names can change over time, but field-id won't. So if you filter on a specific field-id, that will always work correctly. The same goes when reading parquet files, it can be that the names change over time, and you don't want to rewrite petabytes of data, so you can leave them as is. When reading, you want to make the field-ids to the current schema.