feat: support ingesting Timestamp(nanoseconds) from Iceberg

risingwavelabs / risingwave

Best-in-class stream processing, analytics, and management. Perform continuous analytics, or build event-driven applications, real-time ETL pipelines, and feature stores in minutes. Unified streaming and batch. PostgreSQL compatible.

https://go.risingwave.com/slack

Apache License 2.0

7.07k stars 581 forks source link

feat: support ingesting Timestamp(nanoseconds) from Iceberg #19163

Open lmatz opened 3 weeks ago

lmatz commented 3 weeks ago

Right now

ERROR:  Failed to run the query

Caused by:
  DataInvalid => Unsupported Arrow data type: Timestamp(Nanosecond, None)

https://github.com/apache/iceberg/milestone/42?closed=1 https://github.com/apache/iceberg/issues/8657 https://github.com/apache/iceberg-rust/pull/542

chenzl25 commented 3 weeks ago

If risingwave doesn't have a nanoseconds timestamp type, we can't support it. The issue as decimal with 38 digits.

lmatz commented 3 weeks ago

how about a struct of timestamp + integer (for the nanosecond part) as a workaround

chenzl25 commented 3 weeks ago

how about a struct of timestamp + integer (for the nanosecond part) as a workaround

Do you mean use a struct of timestamp + integer to implement a nanoseconds timestamp in risingwave?

chenzl25 commented 3 weeks ago

@lmatz The Iceberg table currently doesn't contain a nano timestamp column type (because no engine can create such a table at least for now). The original time stamp issued came from the parquet file source.