OpenSearch Data Prepper is a component of the OpenSearch project that accepts, filters, transforms, enriches, and routes data at scale.
266
stars
204
forks
source link
[BUG] Index template_file and template_content truncates index pattern incorrectly #3432
Open
graytaylor0 opened 1 year ago
Describe the bug
Index patterns are being created with a truncated index pattern from the index template index patterns passed in the config
This bug occurs when there is no
ism_policy_file
passed to the source. This is here in the code (https://github.com/opensearch-project/data-prepper/blob/463fa4590d5f9d7f0a397430de0dbd634384dd03/data-prepper-plugins/opensearch/src/main/java/org/opensearch/dataprepper/plugins/sink/opensearch/index/IndexManagerFactory.java#L103) where the decision to useNoIsmPolicyManagement
class is made. This class has the following method which does not add a pattern back (https://github.com/opensearch-project/data-prepper/blob/463fa4590d5f9d7f0a397430de0dbd634384dd03/data-prepper-plugins/opensearch/src/main/java/org/opensearch/dataprepper/plugins/sink/opensearch/index/NoIsmPolicyManagement.java#L43), which is called here (after removing the pattern suffix) (https://github.com/opensearch-project/data-prepper/blob/463fa4590d5f9d7f0a397430de0dbd634384dd03/data-prepper-plugins/opensearch/src/main/java/org/opensearch/dataprepper/plugins/sink/opensearch/index/AbstractIndexManager.java#L230). With an ism policy file, this method will be called instead, which adds back the pattern (https://github.com/opensearch-project/data-prepper/blob/463fa4590d5f9d7f0a397430de0dbd634384dd03/data-prepper-plugins/opensearch/src/main/java/org/opensearch/dataprepper/plugins/sink/opensearch/index/IsmPolicyManagement.java#L145)To Reproduce Steps to reproduce the behavior:
You will see that the index_pattern in the template is only
test
and nottest-*
Expected behavior To not truncate the index pattern
Additional context