opensearch-project / data-prepper

OpenSearch Data Prepper is a component of the OpenSearch project that accepts, filters, transforms, enriches, and routes data at scale.
https://opensearch.org/docs/latest/clients/data-prepper/index/
Apache License 2.0
266 stars 204 forks source link

[BUG] Index template_file and template_content truncates index pattern incorrectly #3432

Open graytaylor0 opened 1 year ago

graytaylor0 commented 1 year ago

Describe the bug

Index patterns are being created with a truncated index pattern from the index template index patterns passed in the config

This bug occurs when there is no ism_policy_file passed to the source. This is here in the code (https://github.com/opensearch-project/data-prepper/blob/463fa4590d5f9d7f0a397430de0dbd634384dd03/data-prepper-plugins/opensearch/src/main/java/org/opensearch/dataprepper/plugins/sink/opensearch/index/IndexManagerFactory.java#L103) where the decision to use NoIsmPolicyManagement class is made. This class has the following method which does not add a pattern back (https://github.com/opensearch-project/data-prepper/blob/463fa4590d5f9d7f0a397430de0dbd634384dd03/data-prepper-plugins/opensearch/src/main/java/org/opensearch/dataprepper/plugins/sink/opensearch/index/NoIsmPolicyManagement.java#L43), which is called here (after removing the pattern suffix) (https://github.com/opensearch-project/data-prepper/blob/463fa4590d5f9d7f0a397430de0dbd634384dd03/data-prepper-plugins/opensearch/src/main/java/org/opensearch/dataprepper/plugins/sink/opensearch/index/AbstractIndexManager.java#L230). With an ism policy file, this method will be called instead, which adds back the pattern (https://github.com/opensearch-project/data-prepper/blob/463fa4590d5f9d7f0a397430de0dbd634384dd03/data-prepper-plugins/opensearch/src/main/java/org/opensearch/dataprepper/plugins/sink/opensearch/index/IsmPolicyManagement.java#L145)

To Reproduce Steps to reproduce the behavior:

  1. Create a pipeline with the following opensearch sink configuration
sink:
  - opensearch:
       index: "test-index-1"
       template_type: "index-template"
       template_content: |
          {
            "index_patterns": [
              "test-*"
            ],
            "template": {
              "aliases": {
                  "my_test_logs": {}
              },
              "settings": {
                  "number_of_shards": 5,
                  "number_of_replicas": 2,
                  "refresh_interval": -1
              },
              "mappings": {
                "properties": {
                  "timestamp": {
                    "type": "date",
                    "format": "yyyy-MM-dd HH:mm:ss||yyyy-MM-dd||epoch_millis"
                  },
                  "value": {
                    "type": "double"
                  }
                }
              }
            }
          }
  1. Run data prepper and then go to OpenSearch Dashboards to look at the created index template
GET _index_template/test-index-template

You will see that the index_pattern in the template is only test and not test-*

Expected behavior To not truncate the index pattern

Additional context

LHozzan commented 7 months ago

Maybe same here.