Danouchka
commented
1 week ago Hi @syepes
The reasons why a transform cannot write into a datastream are explained here https://github.com/elastic/elasticsearch/issues/62712 Because transforms in general update data in the destination whereas data streams can be only appended This also explained why destination have data a retention settings so as not to be ruled by an ILM Just thinking at loud , maybe à workaround would be alias + rollover like one indice per month … You are right nevertheless, there should be room for supporting data streams
elasticsearchmachine
Description
It would be a significant enhancement if we could establish transform jobs (pivots) where the destination index is a Time Series Data Stream (TSDS). This capability would open up a wide range of use cases that could leverage TSDS features, such as optimized storage, speed and particularly the efficient downsampling of data points.
Additionally, an important aspect of this feature is the ability to directly generate metrics from our logs, which would greatly simplify the creation of dashboards and fulfill various reporting and alerting requirements.
Furthermore, it’s worth noting that similar functionality for deriving metrics from logs has been available in many of the competitor products for years now, making this addition essential for maintaining the competitive edge of keeping the Elastic Stack a viable solution.
@Danouchka