search

JanKaul / iceberg-rust

Rust implementation of Apache Iceberg with integration for Datafusion
Apache License 2.0
74 stars 11 forks source link

Project status #11

Closed sanderpick closed 5 months ago

sanderpick commented 5 months ago

Hey there! I've been playing around with your lib, nice work :muscle:. I am curious though, have you put this down in favor of https://github.com/apache/iceberg-rust ? This project seems a bit ahead of that one, but has less activity.

JanKaul commented 5 months ago

Hi there, thanks for your interest. Let me try to provide some perspective. I started this library to use it for some personal projects. And for that the library has reached a state where it has enough features to be useful. Then the larger iceberg community showed interest in a rust client and the www.github.com/apache/iceberg-rust repo was created. I then contributed some code to the official repo. But since the official repo doesn't yet have all the features I require, I lately started focusing on this repo again. Since I require some exotic features like materialized views, I'm probably going to stick with this repo for a while and actively work in it. If time allows I might try to contribute some of the code from this repo to the official repo. I hope this explains the current state.

Feel free to create some issues if you require certain features.

sanderpick commented 5 months ago

Thanks for the context. I believe all features we need are already built, with the possible exception of an object store based catalog as mentioned here: https://github.com/apache/arrow-rs/blob/master/object_store/src/lib.rs#L395. If that turns out to be the case, I'll open an issue! Thanks for the help.

JanKaul commented 5 months ago

At some point this library used to support file system tables but I dropped the support because of the additional complexity and the whole iceberg community is moving away from it. The easiest alternative is to use a Postgres or MySql database as the catalog.