feldera / feldera

The Feldera Incremental Computation Engine
https://feldera.com
Other
743 stars 47 forks source link

LATENESS doesn't propagate through asof-join #2530

Closed ryzhyk closed 1 month ago

ryzhyk commented 1 month ago

I think this is a known limitation, but I didn't find an issue for it. I believe that in this program view v should have a waterline.

create table r(
    id BIGINT NOT NULL,
    ts timestamp NOT NULL LATENESS INTERVAL 0 days
);

create table l (
    id BIGINT NOT NULL,
    ts timestamp NOT NULL LATENESS INTERVAL 0 days
);

create view v as
select
    l.id as id,
    l.ts as lts,
    r.ts as rts
from
    l left asof join r
    MATCH_CONDITION (r.ts <= l.ts)
ON
    l.id = r.id;

CREATE VIEW agg1 as 
SELECT
    MAX(id)
FROM
    v
GROUP BY lts;
blp commented 1 month ago

I came across this yesterday too. I was able to use the LATENESS statement extension for views to fix the problem that I had

ryzhyk commented 1 month ago

Unfortunately I think it's unsound in this case, i.e., you cannot manually express a safe lateness bound for this view as it should be a function of the waterlines of the two inputs to the as of join