SUSE / telemetry-server

Proof of Concept Telemetry Server scaffolding
Apache License 2.0
2 stars 0 forks source link

Annotations: Implement simplified tags storage #12

Closed rtamalin closed 1 day ago

rtamalin commented 1 month ago

The proof of concept approach to storing annotation tags in the Telemetry DB will not scale well over time with large numbers of clients. A multi-table hierarchy will be required to store the tags associated with a telemetry data item in an efficient fashion.

A simpler approach would be to simply have a tags table that uniquely stores the set of tags associated with telemetry data items, and have the telemetry data table store a reference to that tags table entry.

The tag set type provided by the Telemetry Client Library SUSE/telemetry#5 should help with implementing this.

How the tags table actually stores the tag set will need to be finalised, though the likely options are:

  1. as a sorted list of tags with a delimiter character, e.g. |, wrapping each tag entry in the list, allowing easy text matching for specific entries even in SQL queries.
  2. store the list of tags as a JSON array, programmatically searchable, though SQL support for doing so will be Telemetry DB backend specific.
rtamalin commented 4 weeks ago

Note that for now I have implemented a limited for of option (1) as part of prototyping the telemetry storage transformation workflow.

rtamalin commented 1 day ago

Closing this issue as the implemented solution seems sufficient for now.