Automatically create data stream

[moharana-subhashree]

I am trying to send sample logs from logstash to opensearch data stream using below configuration-

    configFile: |
      input {
        heartbeat {
          interval => 5
          message  => 'Hello from Logproc'
        }
      }
      filter {
        grok{
        }
      }
      output {
        opensearch {
            hosts => [ '{{ .Values.indexsearch.serviceName }}.{{.Release.Namespace}}:{{.Values.indexsearch.port }}' ]
            ssl => false
            index => "log"
            action => "create"
                    }
              }

when i check in opensearch i see only one index has been created with this configuration, but that is a regular index, not part of the data stream backing index. is there any extra configuration which I might be missing because the data is getting ingested to data stream from logstash.

[dblock]

@moharana-subhashree I could be wrong, but I don't think the plugin is actually capable of knowing that you want a data stream here, and it implies that you should create that data stream index yourself. Does that work?

In that case we should 1) turn this into a feature request, 2) document this better in https://opensearch.org/docs/latest/tools/logstash/ship-to-opensearch/#data-streams

[moharana-subhashree]

hi @dblock, as in the above post, it is concluded that the creation of data stream on ourselves is mandatory and doesn't support creating it dynamically by the plugin for now. Recently i was trying below opensearch configuration as below to create a legacy index template as part of the opensearch configurations as below

output {
    opensearch {
      hosts => "http://localhost:9200"
      index => "logs"
      action => "create"
      document_id => "%{id0}"
      template_name => "cas-template"
      template => "/usr/share/logstash/pipeline/template/cas-template.json"
      manage_template => "true"
      template_overwrite => "true"
    }
}

I have a legacy index template format as below provided in the configurations:

    {
      "index_patterns": ["logs-*"],
      "settings": {
        "number_of_shards": 1,
        "number_of_replicas": 1
      },
      "mappings": {
        "properties": {
          "timestamp": {
            "type": "date"
          },
          "message": {
            "type": "text"
          }
        }
      },
      "aliases": {
        "logstash": {}
      }
    }

but with this configuration it ideally shall create a index template and that index template could be used for data stream creation, is my understanding correct? will this be a right way to proceed?

[dblock]

but with this configuration it ideally shall create a index template and that index template could be used for data stream creation, is my understanding correct? will this be a right way to proceed?

Yes, I can't think of a reason why one would not want this, so this is a feature request. I'll rename the issue.