search

JanKaul / iceberg-rust

Rust implementation of Apache Iceberg with integration for Datafusion
Apache License 2.0
74 stars 11 forks source link

Status of Writing to Iceberg? #2

Closed jacksonrnewhouse closed 5 months ago

jacksonrnewhouse commented 1 year ago

Hi there, I'm building something that uses datafusion for parts of the planning and would like to be able to write results directly to Iceberg. Is the intent of this project to support that eventually? How much effort would it take and could I potentially help with that?

JanKaul commented 1 year ago

Hi, it's definitely planned to support writing to iceberg. I just didn't get around to implementing it. And I haven't figured out a perfect way to do so. I think Datafusion currently doesn't have the functionality for INSERT INTO operations. So the current solution probably requires some external writer. We can have a look at how delta_rs is doing it.

jacksonrnewhouse commented 1 year ago

I'm currently only using the parsing and planning components of DataFusion and performing the execution separately. Since 17.0 the DataFusion grammar now supports WriteOps, via this PR: https://github.com/apache/arrow-datafusion/pull/4902

JanKaul commented 1 year ago

Okay great, I will have a look. If you have an idea how this could be implemented it would be great if you could share it here. I will also report my findings.

doki23 commented 1 year ago

Do we have rust api for writing?

JanKaul commented 1 year ago

No, not yet. But I will look into it in the coming days.

JanKaul commented 5 months ago

It's been a while but the crate now supports writing to iceberg tables with datafusion.