datapythonista
commented
2 years ago Personally, I think it's limiting and not very scalable if pandas has to support every format. I think pandas should provide a standard way to load I/O plugins, and this (and many other readers/writers) should be implemented as third-party projects. Not only maintaining them will be easier and more efficient, but having competing options for certain formats would be beneficial. Like a csv format that makes loads of assumptions and making things easier for users, and another that requires more programming but is faster and safer.
mroeschke
ehsantn
Apache Iceberg is an open source table format designed for efficiently reading from large datasets. By leveraging detailed statistics information and hidden partitioning, reading becomes efficient for extremely large datasets, even up to pedabytes according to the author.
As a result, there is a lot of interest in Iceberg inside of the Data Engineering community and many people seem willing/eager to try running their production workloads using Iceberg. To help meet this growing demand, we believe that Pandas should be able to read DataFrames from datasets using Iceberg format.
Bodo can be used underneath Pandas to read Iceberg, probably through
pd.read_sql/pd.read_sql_table.