Support dates in elasticsearch index name

[joernott]

Is your feature request related to a problem? Please describe.

We are using retention policies to delete old data based on index patterns. Logstash supports a "indexname-%{+YYYY.MM.dd}" pattern (and for high volume indexes we also use hourly rotation). I would like to use a similar feature here.

Describe the solution you'd like

A possibility to support Joda type date declarations in the "index" field for the Elasticsearch output

[joernott]

Also, the data sent to elasticsearch is missing the "@timestamp" field. If that would exist, we could use a datastream instead of a static index name. Maybe, using system.systemTimestamp to fill "@timestamp" would be a good idea, too

[joernott]

At the moment, I am working around like this:

1. I have a pipeline for this defined:

   PUT _ingest/pipeline/ltm_statistics_add_timestamp
   {
   "description": "Add timestamp as f5 metrics does not populate that field",
   "processors": [
   {
     "set": {
       "field": "@timestamp",
       "copy_from": "system.systemTimestamp"
     }
   }
   ]
   }

2. I have a retention policy defined, which rolls over daily (or at 1 petabyte of statistics) and deletes after 30 days.

   PUT _ilm/policy/retention30rollover
   {
   "policy": {
   "phases": {
     "hot": {
       "min_age": "0ms",
       "actions": {
         "rollover": {
           "max_primary_shard_size": "1pb",
           "max_age": "1d"
         }
       }
     },
     "delete": {
       "min_age": "30d",
       "actions": {
         "delete": {
           "delete_searchable_snapshot": true
         }
       }
     }
   }
   }
   }

3. I have 2 component templates and a composite template defined to create a datastream for statistics. F5 telemetry is not aware that it is not writing to an index, but a datastream instead.

   
   PUT _component_template/ltm_statistics_timestamp
   {
   "template": {
   "mappings": {
     "properties": {
       "@timestamp": {
         "type": "date"
       },
       "system.systemTimestamp": {
         "type": "date"
       }
     }
   }
   }
   }

PUT _component_template/ltm_statistics_settings { "template": { "settings": { "index": { "lifecycle": { "name": "retention30rollover" }, "codec": "best_compression", "routing": { "allocation": { "include": { "datacenter": "DC1,DC2" } } }, "number_of_shards": "1", "number_of_replicas": "1", "default_pipeline": "ltm_statistics_add_timestamp" } } } }

PUT _index_template/ltm_statistics { "index_patterns": [ "ltm_statistics*" ], "data_stream": { }, "composed_of": ["ltm_statistics_settings","ltm_statistics_timestamp"], "priority": 1000 }

[G-gonzalezjimenez]

Hi, thank you for your comments and suggestions. Telemetry Streaming was moved to maintenance mode last year.

F5 BIG-IP Telemetry Streaming is entering a phase of ongoing maintenance and support. A product in maintenance mode continues to receive support and ensures its stability with regular critical fixes and security updates. This maintenance approach helps maintain the longevity and reliability of the product for the long term. Enhancement requests for this product will be evaluated on an individual basis, taking into consideration their overall impact and alignment with our business objectives. Only those with a strong case for improvement will be considered for implementation. There is no plan to deprecate this product.

https://clouddocs.f5.com/products/extensions/f5-telemetry-streaming/latest/

We are not taking any requests for enhancements or new features.

Thank you