PyIceberg Production Use case survey

[kevinjqliu]

Feature Request / Improvement

As part of the journey toward version 1.0, we want to capture how this library is used in "production" environments.

Would love to hear from current users (and potential users) on different use cases. This will better inform the future roadmap.

Please include use cases in this issue, or if necessary I can start a Google Survey.

[mariotaddeucci]

Hey, actually I'm using in production for small datasets in combination with duckdb specially to avoid small files with webscrapping.

For ingestion, reading many raw files (json, csv, and parquet), all off then with a key using ulid (sortable id is necessary) in combination with overwrite specifying this key as overwrite filter. Duckdb generates a record_batach_reader, which allows to generate the table and schema without load all in memory, after creating the table is necessary to converte into a arrow table to write the final iceberg table.

Because of the sortable id, it's possible to use the the filter predicate overwriting the data between upper and lower bound the data set to be ingested.

The table maintenance still using spark for expiring snapshot.

To avoid small files, after certain period using the duckdb native iceberg read, I reload the entire dataset and overwrite it fully (a workaround for rewrite files procedure)

I would love to expand it for more scenarios but some features are necessary like

• allow to write using record_batch_reader, so no need to load a full arrow table in memory.
• clear snapshots from pyiceberg, that's turns the maintenance easier, no external engine or tool
• maybe a simple optimization like binpack, is not the best but it's better than read all and overwrite it.
• Maybe an integration with duckdb, just taking the last metada location and creating a view on it using their native iceberg reader
• a truly merge operation, so avoiding errors when doing upserts, making not necessary to use the upper and lower bound of DF key as overwrite filter.

These pipelines are leaving from spark server and running on isolated containers.

[andreapiso]

Using pyiceberg alongside Trino. Our ETL is in Trino, pyiceberg Is great for assets where we are doing things like grabbing data from APIs. Instead of storing files and crawling them with something like glue into iceberg tables, we can directly write that data into iceberg so that our Trino pipelines can process it directly, super convenient!