search

apache / iceberg-rust

Apache Iceberg
https://rust.iceberg.apache.org/
Apache License 2.0
673 stars 159 forks source link

Tracking issue: Writing iceberg tables #346

Open ZENOTME opened 7 months ago

ZENOTME commented 7 months ago

Inspired by https://github.com/apache/iceberg-rust/pull/275#issuecomment-2041304545, I created this issue to track all our write task. It's based on doc.

c-thiel commented 5 days ago

Dear all, we have been talking to many people recently that are working in the Rust data ecosystem for which writes or compaction would be very important. Many are also willing to take up some of the issues here - me included.

My goal is to align with the community on good next steps and then distribute issues among us to get more focus, attention and developer time for the issues that are needed to implement writes and compaction.

In addition to the issues listed above, there are currently open PRs that are also important for writes:

Related PRs in iceberg-rust

Related other Issues / Discussions

I am sure I missed a few - please feel free to extend.

My proposal for a way forward would be as follows:

  1. It would be great to get some eyes on on the already open PRs - I think especially https://github.com/apache/iceberg-rust/pull/349 deserves some attention.
  2. If there are any issues that are ready to be implemented from the list above - that nobody else is working on already, it would be great to highlight a few as an answer to this post that people can take and start working on. Maybe @Xuanwo, @liurenjie1024, @Fokko you have an overview?
  3. We will have the first Iceberg Rust Community sync shortly on 28th of November! (Public Google Calendar, Mailing List). In my opinion this would be a great opportunity to align on next steps and maybe hand out some tasks.

Let me know what you think!

@adisheshkishore, @amitgilad3, @JanKaul, @jaychia, @kevinzwang, @mkarbo, @mehmetozsoy-synnada, @rampage644, @twuebi

sdd commented 5 days ago

I've been intending on switching focus from reads to writes once my delete file read support PR is merged. I'll do my best to attend the meet on the 28th (day before my birthday! 😁) and look forward to collaborating with whoever else gets involved to land writes in iceberg-rust!

ZENOTME commented 5 days ago

Thank @c-thiel for raising! I am also working on #340 and can send it as a PR soon. Recently, I will also dedicate more effort to writing support. I think #349 is the first step to writing support in iceberg-rust so welcome review.