ArroyoSystems / arroyo

Distributed stream processing engine in Rust
https://arroyo.dev
Apache License 2.0
3.8k stars 220 forks source link

Non-data fields - Metadata fields to Kafka Connector #765

Closed vaibhawvipul closed 3 weeks ago

vaibhawvipul commented 4 weeks ago

enabling users to query using the following syntax -

create table users (
    id TEXT,
    name TEXT,
    offset BIGINT GENERATED ALWAYS AS (metadata('offset_id')) STORED
) with (
    connector = 'kafka',
    ...
);

Demo -

Screenshot 2024-10-24 at 11 21 10 AM
vaibhawvipul commented 3 weeks ago

@mwylde can you please trigger the CI again? I am asking because my last commit ran successfully, the failed CI is on master merge.