For loading data from an external DB (like MySQL) we can use insert ... select technique to pull data from the remote server. The source table should have an updated_at DatimeTime column and index on it.
with (select max(updated_at) from dest) as max
select * from mysql(db1,table=XXX) where created >= max and created < now() - delay
"last" position will be updated not by the processor (it doesn't know it), but by MV connected to the dest table
step here should be the number of seconds, not rows.
source table could be any string that will be substituted by the template processor:
select * from {src}
select * from mysql(named_collection, table={src})
select * from mysql({tag}, table={src})
The topic is always ready to run and only repeat timeout stops it from executing continuously.
For loading data from an external DB (like MySQL) we can use insert ... select technique to pull data from the remote server. The source table should have an updated_at DatimeTime column and index on it.
"last" position will be updated not by the processor (it doesn't know it), but by MV connected to the dest table step here should be the number of seconds, not rows.
source table could be any string that will be substituted by the template processor:
The topic is always ready to run and only repeat timeout stops it from executing continuously.