search

GreptimeTeam / greptimedb

An open-source, cloud-native, unified time series database for metrics, logs and events with SQL/PromQL supported. Available on GreptimeCloud.
https://greptime.com/
Apache License 2.0
4.35k stars 315 forks source link

TTL=0 data aka flow-only data ingestion #5033

Open sunng87 opened 2 days ago

sunng87 commented 2 days ago

What type of enhancement is this?

API improvement

What does the enhancement do?

From one of our key user who only wants to generate derived data without persist original data. We will need to provide a special case for this type of TTL=0 data: feed it to flow and drop it before entering memtable.

Implementation challenges

No response

killme2008 commented 2 days ago

I think we can design a virtual table for such cases just like

create table monitor(ts timestamp time index, cpu double) with(engine='virtual');

It serves solely as an ingestion endpoint and a source for the flow engine, but cannot be queried.

sunng87 commented 1 day ago

I think ttl=0 is a better abstraction. We'd better not to introduce new concept unless it's necessary.

killme2008 commented 1 day ago

I agree, but if we can implement the ttl=0 without consuming any memory, that would be better. Just drop the write requests before they reach the memtable, maybe we don't create any memtables for such tables.