index templates other than logs and metrics when using data streams

[maxgruber19]

index templates other than logs and metrics when using data streams #751

Problem

In most real world scenarios a user is not in control over the default index templates "logs" and "metrics". There are many usecases where the user wants to define a custom template. In my case I want to use the data retention feature to delete messages stored in elasticsearch after a certain amount of time.

Currently this connector is able to use the predefined index templates "logs" and "metrics" when data streams are enabled. User created templates are not supported yet.

Solution

Defining the type of the data stream as String to make it flexible for custom inputs. The enum will not be removed to still be able to use the recommender when deploying the connector via control-center.

Does this solution apply anywhere else?

• [ ] yes
• [ ] no

Test Strategy

Started a local cluster via docker-compose (cp-quickstart and elastic+kibana) and installed the connector.zip in the cluster. Both connectors with "logs" and connectors with "custom" as data stream types were running perfectly fine after creating the "custom" index template in Kibana.

Testing done:

• [x] Unit tests
• [x] Integration tests
• [x] System tests
• [x] Manual tests

[cla-assistant[bot]]

All committers have signed the CLA.

[cla-assistant[bot]]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[ge1sterfahrer]

Having this feature would massively help us in day to day business. What's your opinion on this? @sp-gupta