search

superstreamlabs / memphis

Memphis.dev is a highly scalable and effortless data streaming platform
https://docs.memphis.dev
Other
3.26k stars 218 forks source link

Feature: Apache Iceberg remote message storage #1174

Open rupurt opened 1 year ago

rupurt commented 1 year ago

Current behavior

Currently messages are stored in a row based format which is not optimal for vectorized query engines such as DuckDB.

Suggested solution

Add option to store messages in Apache Iceberg tables. Redpanda is currently working on adding this https://twitter.com/emaxerrno/status/1681727055451308064

Additional context

No response

Code of Conduct

Contributing Docs

yanivbh1 commented 1 year ago

Hey @rupurt , Thank you for bringing this one up.

yanivbh1 commented 1 year ago

Also, adding @bjwschaap :)