ArroyoSystems / arroyo

Distributed stream processing engine in Rust
https://arroyo.dev
Apache License 2.0
3.8k stars 220 forks source link

Add timestamp metadata field to kafka connector #776

Closed mwylde closed 3 weeks ago

mwylde commented 3 weeks ago

This PR adds a metadata field for the timestamp to the Kafka source, such that this now works:

create table workflow (
    id TEXT,
    topic TEXT GENERATED ALWAYS AS (metadata('topic')) STORED,
    timestamp BIGINT GENERATED ALWAYS AS (metadata('timestamp')) STORED
) with (
    connector = 'kafka',
    bootstrap_servers = 'localhost:9092',
    topic = 'workflow-events',
    type = 'source',
    format = 'json'
);

This closes #631.

Also addresses an issue where metadata fields could not be added to sources with raw_string/raw_bytes/unstructured json formats.