tricorder-observability / Starship

Starship: next-generation Observability platform built with eBPF+WASM
GNU Affero General Public License v3.0
164 stars 24 forks source link

Support bcc/wasm code data storing TTL settings #31

Open JaredTan95 opened 1 year ago

JaredTan95 commented 1 year ago

for now, we use timescaldb stored metrics and trace data by promescale.

In addition, bcc will send data to timescaldb, so , we need to consider adding TTL settings for startship apiserver.

nascentcore-eng commented 1 year ago

High-level 2 aspects:

Right now we have TTL retention in promscale-connector. Next setup timescaledb's own data retention policy. And make sure promscale-connector retention and timescaledb's retention does not conflict.

Also users needs to be able to set TTL, and storage limits for the data collected by each eBPF+WASM module. TTL & storage limits are the properties of eBPF+WASM modules.

nascentcore-eng commented 1 year ago

This TTL needs to be propagated to promescale-connector and/or timescale db layer.

On the other hand, we could allow a global TTL configuration on all data exported to a generic "database/storage" component. This is different from the above setup, in that, this generic "database/storage" component could be promscale, clickhouse, or some other system. And this TTL applies to all of the data.

The key is to let the configurations be specified upfront by the users, instead of exposing the underlying storage system. So that Starship can translate storage limits and TTL to specific dependent storage sub-system.