parquet: introduce file writer - Githubissues

redpanda-data / redpanda

Redpanda is a streaming data platform for developers. Kafka API compatible. 10x faster. No ZooKeeper. No JVM!

https://redpanda.com

9.65k stars 589 forks source link

parquet: introduce file writer #24039

Closed rockwotj closed 1 week ago

rockwotj commented 1 week ago

container/contiguous_range_map: add at accessor
parquet/column_writer: limit exception message length
parquet/metadata: adjust unused metadata
parquet/schema: also index the path of the schema for fast lookup
parquet/metadata: use chunked_vector everywhere
parquet/column_writer: move constructor to impl
parquet: introduce file writer
bazel/format_cc: use root dir

Backports Required

[x] none - not a bug fix
[ ] none - this is a backport
[ ] none - issue does not exist in previous branches
[ ] none - papercut/not impactful enough to backport
[ ] v24.2.x
[ ] v24.1.x
[ ] v23.3.x

Release Notes

none

rockwotj commented 1 week ago

It's ready! (I temporarily jacked up the number of randomized tests and we get the same results between parquet-go and our writer!)

Executed 1024 out of 1032 tests: 1032 tests pass.

vbotbuildovich commented 1 week ago

the below tests from https://buildkite.com/redpanda/redpanda/builds/57788#01930781-605d-4871-be88-61be5461d92d have failed and will be retried

translator_test_rpfixture