Open danielmitterdorfer opened 5 months ago
Another potential step to take is to go to Time Series Data Streams, for reduction in stored size. This limits support for pre 8.7 (was in technical preview from 8.5) however.
In some ways, the index template already being used is friendly to this - keyword for all strings as would be for a dimension. Though there are limits of this (max 21 dimensions). TSDS also wants to know if your metric is a gauge or counter, which could be generalized with value.gauge and value.counter mappings. But would require some routing logic when emitting the events.
So far we have used a single index to store k6 metrics. However, datastreams are preferable for a couple of reasons, one of them being able define a retention period via an ILM policy. We should therefore move away from the single index and instead create a datastream. Storing data in an index will not be supported anymore.
Datastream Details
metrics-k6-default
. The namespace - here:default
- is overridable via a new settingK6_ELASTICSEARCH_DATASTREAM_NAMESPACE
. The existing settingK6_ELASTICSEARCH_INDEX_NAME
will be removed. It will not be possible to override the entire datastream name. This would increase complexity significantly as we would need to make sure that the index template matches the chosen datastream name. Note: We might allow overriding the entire datastream name in the future if and only if the datastream is managed by the user.These calls will be issued internally:
Note: previously the timestamp field was called
Time
. We can rename this in a reindex script.Behavior for existing installations
When the
k6-metrics
index exists, we can issue a warning that the index pattern has changed. This is only best effort and won't catch cases where users have overridden the index name though.Migration
We won't automatically migrate data but can provide a reindex and cleanup script that users can execute if required.
Permissions
We might need to adapt the initial permission check as the output extension needs to create a datastream and associated ILM policy. Finally, we should allow to make this process optional as advanced users might want to create the datastream themselves and tighten the cluster permissions of the k6 user to allow only write access. This behavior will be controlled by the flag
K6_ELASTICSEARCH_AUTOCREATE_DATASTREAM
which istrue
by default. If it set tofalse
, the output extension assumes that the datastream is already setup properly (without any further checks).