Relation to official iceberg-rust repo

[matthewmturner]

Hi there - I think what you are working on here is great. Im just curious if theres a reason for not contributing to what i believe is the official rust implementation. no judgement im just trying to get a lay of the land as im evaluating the options if i wanted to start contributing to rust / datafusion implementations for iceberg.

[JanKaul]

I haven't been able to contribute to the official repo as much as I wanted to lately. I needed an iceberg implementation that works with datafusion rather quickly. Since this repo already had basic support for datafusion when the official repo was started, it was faster to implement the missing functionality here than to wait for the official repo. Additionally this repo is closer to the arrow ecosystem by using the object_store crate.

I hope I will have more time in the future to contribute to the official repo.

It would be great to have you on board for either repo.

[matthewmturner]

Noted, thanks! I'm going to study up some more on Iceberg and then figure out what to work on next.

[matthewmturner]

Actually, can you expand on what you mean by closer to the arrow ecosytem by using object_store crate?

[JanKaul]

Well, the official apache iceberg-rust crate uses Opendal to handle the interaction with cloud object stores while I preferred to use the object_store crate which is part of the arrow-rs ecosystem.

But I have to admit that this is a bit of a constructed argument because I think there is a crate that implements the ObjectStore trait for opendal.

Since I mostly focused on Datafusion, which uses object_store internally, it was easier for me to use it directly.

[matthewmturner]

Great, thank you for the information @JanKaul