opensearch: data streams

[ecerulm]

Is your feature request related to a problem? Please describe.

fluentd-plugin-opensearch supports opensearch data stream (at least in beta) . I want the same support in fluent-bit out_opensearch, or at least some mention of data stream in the fluent-bit opensearch documentation page stating that is not supported.

Describe alternatives you've considered

I can use a regular index instead, but data stream are more suitable for logs / time.

Quoting from data streams documentation:

If you’re ingesting continuously generated time-series data such as logs, events, and metrics into OpenSearch, [...] Data streams simplify this process and enforce a setup that best suits time-series data, such as being designed primarily for append-only data and ensuring that each document has a timestamp field.

I guess it's also possible to create the data stream manually in opensearch prior to start any fluent-bit. Then point fluent-bit to that data stream (as index) and I believe that works. But if this is the "wordaround" then it should be mentioned in fluent-bit opensearch documentation page

Additional context

I'm looking into migrating from a fluentd configuration that usees data_stream_name into fluent-bit, and I feel that either it should be supported or maybe some note regarding data streams should be added to the fluent-bit opensearch documentation page, either stating that they are not supported, or workaround, but some mention of the word "data stream" should appear in the docs.

My current /etc/fluent/fluentd.conf

    <source>
      @type forward
      @id input_forward
    </source>

    <match microservices.**>
      @type opensearch_data_stream
      data_stream_name logs-microservices
      data_stream_template_name logs-template

      ssl_version TLSv1_3

      with_transporter_log true
      @log_level info

      http_backend typhoeus

      <buffer>
        flush_interval 60s
        retry_type periodic
        retry_wait 10s
        retry_max_times 15
      </buffer>

      <endpoint>
        url https://opensearch.xxxxx.com
        region us-east-1
        assume_role_arn arn:aws:iam::xxxxxxxxx:role/MasterUserRole
        assume_role_session_name microservices-ec2
        refresh_credentials_interval 59m # default is 5h (five hours).
      </endpoint>
    </match>

[ecerulm]

I realized that there was another issue https://github.com/fluent/fluent-bit/issues/7079 that was closed 2023-07-30.

Also there is PR https://github.com/fluent/fluent-bit/pull/7371

[github-actions[bot]]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[github-actions[bot]]

This issue was closed because it has been stalled for 5 days with no activity.