Data stream ingestion throws `400 - Rejected by OpenSearch`

casabre commented 2 years ago

(check apply)

[x] read the contribution guideline
[x] (optional) already reported 3rd party upstream repository or mailing list if you use k8s addon or helm charts.

Steps to replicate

<source>
  @type forward
</source>

<match **>
  @type opensearch
  host os-http
  port 9200
  scheme http
  user "#{ENV['OS_USERNAME']}"
  password "#{ENV['OS_PASSWORD']}"
  index_name ${tag}
  include_timestamp true
  <buffer tag, time>
    @type file
    path /tmp/log/fluent/buffer_${tag}
    timekey 3600
    flush_mode interval
    flush_interval 5
  </buffer>
</match>

<match my_datastream**>
  @type opensearch_data_stream
  data_stream_name my_datastream
  host os-http
  port 9200
  scheme http
  user "#{ENV['OS_USERNAME']}"
  password "#{ENV['OS_PASSWORD']}"
  include_timestamp true
  <buffer tag, time>
    @type file
    path /tmp/log/fluent/buffer
    timekey 3600
    flush_mode interval
    flush_interval 5
  </buffer>
</match>

Data stream shows following error when ingesting at Opensearch.

2022-07-01 09:14:21 +0000 [warn]: #0 dump an error event: error_class=Fluent::Plugin::OpenSearchErrorHandler::OpenSearchError error="400 - Rejected by OpenSearch" location=nil tag="my_datastream" time=2022-07-01 09:04:38.007341687 +0000 record={"cpu_p"=>43.5, "user_p"=>40.0, "system_p"=>3.5000000000000004, "cpu0.p_cpu"=>53.0, "cpu0.p_user"=>50.0, "cpu0.p_system"=>3.0, "cpu1.p_cpu"=>35.0, "cpu1.p_user"=>30.0, "cpu1.p_system"=>5.0}

Expected Behavior or What you need to ask

How should the data stream setup look like in order to ingest data successfully? Is this more a Opensearch configuration problem or related to the plugin?

The first non-datastream part creates and ingests data the right way.

Using Fluentd and OpenSearch plugin versions

OS version Docker image: docker.io/bitnami/fluentd:1.15.0-debian-11-r1
Helm chart https://artifacthub.io/packages/helm/bitnami/fluentd
Bare Metal or within Docker or Kubernetes or others? Openshift 4.x
Fluentd v1.0 or later fluentd 1.15.0
OpenSearch plugin version LOCAL GEMS

abbrev (default: 0.1.0) activesupport (7.0.3) addressable (2.8.0) aws-eventstream (1.2.0) aws-partitions (1.601.0) aws-sdk-core (3.131.2) aws-sdk-kms (1.57.0) aws-sdk-s3 (1.114.0) aws-sdk-sqs (1.51.1) aws-sigv4 (1.5.0) base64 (default: 0.1.1) benchmark (default: 0.2.0) bigdecimal (default: 3.1.1) bundler (2.3.16, 2.3.13) cgi (default: 0.3.1) concurrent-ruby (1.1.10) cool.io (1.7.1) csv (default: 3.2.2) date (default: 3.2.2) debug (1.4.0) delegate (default: 0.2.0) did_you_mean (default: 1.6.1) digest (default: 3.1.0) digest-crc (0.6.4) domain_name (0.5.20190701) drb (default: 2.1.0) elastic-transport (8.0.1) elasticsearch (8.3.0) elasticsearch-api (8.3.0) elasticsearch-xpack (7.17.1) english (default: 0.7.1) erb (default: 2.2.3) error_highlight (default: 0.3.0) etc (default: 1.3.0) excon (0.92.3) faraday (1.10.0) faraday-em_http (1.0.0) faraday-em_synchrony (1.0.0) faraday-excon (1.1.0) faraday-httpclient (1.0.1) faraday-multipart (1.0.4) faraday-net_http (1.0.1) faraday-net_http_persistent (1.2.0) faraday-patron (1.0.0) faraday-rack (1.0.0) faraday-retry (1.0.3) faraday_middleware-aws-sigv4 (0.6.1) fcntl (default: 1.0.1) ffi (1.15.5) ffi-compiler (1.0.1) fiddle (default: 1.1.0) fileutils (default: 1.6.0) find (default: 0.1.1) fluent-config-regexp-type (1.0.0) fluent-plugin-concat (2.5.0) fluent-plugin-detect-exceptions (0.0.14) fluent-plugin-elasticsearch (5.2.3) fluent-plugin-grafana-loki (1.2.18) fluent-plugin-kafka (0.17.5) fluent-plugin-kubernetes_metadata_filter (2.11.1) fluent-plugin-multi-format-parser (1.0.0) fluent-plugin-opensearch (1.0.7) fluent-plugin-prometheus (2.0.3) fluent-plugin-record-modifier (2.1.0) fluent-plugin-rewrite-tag-filter (2.4.0) fluent-plugin-s3 (1.7.0) fluent-plugin-systemd (1.0.5) fluentd (1.15.0, 1.14.6) forwardable (default: 1.3.2) getoptlong (default: 0.1.1) http (4.4.1) http-accept (1.7.0) http-cookie (1.0.5) http-form_data (2.3.0) http-parser (1.2.3) http_parser.rb (0.8.0) i18n (1.10.0) io-console (default: 0.5.11) io-nonblock (default: 0.1.0) io-wait (default: 0.2.1) ipaddr (default: 1.2.4) irb (default: 1.4.1) jmespath (1.6.1) json (default: 2.6.1, 2.1.0) jsonpath (1.1.2) kubeclient (4.9.3) logger (default: 1.5.0) lru_redux (1.1.0) ltsv (0.1.2) matrix (0.4.2) mime-types (3.4.1) mime-types-data (3.2022.0105) minitest (5.16.1, 5.15.0) msgpack (1.5.2) multi_json (1.15.0) multipart-post (2.2.3) mutex_m (default: 0.1.1) net-ftp (0.1.3) net-http (default: 0.2.0) net-imap (0.2.3) net-pop (0.1.1) net-protocol (default: 0.1.2) net-smtp (0.3.1) netrc (0.11.0) nkf (default: 0.1.1) observer (default: 0.1.1) oj (3.3.10) open-uri (default: 0.2.0) open3 (default: 0.1.1) opensearch-api (2.0.2) opensearch-ruby (2.0.2) opensearch-transport (2.0.0) openssl (default: 3.0.0) optparse (default: 0.2.0) ostruct (default: 0.5.2) pathname (default: 0.2.0) power_assert (2.0.1) pp (default: 0.3.0) prettyprint (default: 0.1.1) prime (0.1.2) prometheus-client (4.0.0) pstore (default: 0.1.1) psych (default: 4.0.3) public_suffix (4.0.7) racc (default: 1.6.0) rake (13.0.6) rbs (2.1.0) rdoc (default: 6.4.0) readline (default: 0.0.3) readline-ext (default: 0.1.4) recursive-open-struct (1.1.3) reline (default: 0.3.0) resolv (default: 0.2.1) resolv-replace (default: 0.1.0) rest-client (2.1.0) rexml (3.2.5) rinda (default: 0.1.1) rss (0.2.9) ruby-kafka (1.5.0) ruby2_keywords (0.0.5) rubygems-update (3.3.13) securerandom (default: 0.1.1) serverengine (2.3.0) set (default: 1.0.2) shellwords (default: 0.1.0) sigdump (0.2.4) singleton (default: 0.1.1) stringio (default: 3.0.1) strptime (0.2.5) strscan (default: 3.0.1) syslog (default: 0.1.0) systemd-journal (1.4.2) tempfile (default: 0.1.2) test-unit (3.5.3) time (default: 0.2.0) timeout (default: 0.2.0) tmpdir (default: 0.1.2) tsort (default: 0.1.0) typeprof (0.21.2) tzinfo (2.0.4) tzinfo-data (1.2022.1) un (default: 0.2.0) unf (0.1.4) unf_ext (0.0.8.2) uri (default: 0.11.0) weakref (default: 0.1.1) webrick (1.7.0) yajl-ruby (1.4.3) yaml (default: 0.2.0) zlib (default: 2.1.1)

OpenSearch version 1.3.0
OpenSearch template(s) (optional)

toby181 commented 2 years ago

Did you already try with enabling this parameter: https://github.com/fluent/fluent-plugin-opensearch#log_os_400_reason? os_400 is in my cases a mapping conflict between the data type that is send and was OS expects.

casabre commented 2 years ago

@toby181 thanks for the hint 😄. I will check that log_os_400_reason flag. It could be actually the the mapping because I didn't set it for the trial run.

leowinterde commented 1 year ago

Relates #82

fluent / fluent-plugin-opensearch