mulog-elasticsearch does not work with data-streams

[handerpeder]

Running the elasticsearch publisher against a data-stream fails with the following exception:

{:mulog/event-name :mulog/publisher-error,
 :mulog/timestamp 1659460044208,
 :mulog/trace-id #mulog/flake "4kTL6YN4f77IE0tEy63XNCSRzwFuG_77",
 :mulog/action :publish,
 :mulog/namespace "clojure.core",
 :mulog/origin :mulog/core,
 :app-name "no-name",
 :env :dev,
 :exception #error {
 :cause "Elasticsearch Bulk API reported errors"
 :data {:errors ({:index {:_index "logs-planner", :_id "4kTL6Qr4GMeaQG0adaPZE0kNLR_7wyeM", :status 400, :error {:type "illegal_argument_exception", :reason "only write ops with an op_type of create are allowed in data streams"}}})}
 :via
 [{:type clojure.lang.ExceptionInfo
   :message "Elasticsearch Bulk API reported errors"
   :data {:errors ({:index {:_index "logs-planner", :_id "4kTL6Qr4GMeaQG0adaPZE0kNLR_7wyeM", :status 400, :error {:type "illegal_argument_exception", :reason "only write ops with an op_type of create are allowed in data streams"}}})}
   :at [com.brunobonacci.mulog.publishers.elasticsearch$post_records invokeStatic "elasticsearch.clj" 138]}]
 :trace
 [[com.brunobonacci.mulog.publishers.elasticsearch$post_records invokeStatic "elasticsearch.clj" 138]
  [com.brunobonacci.mulog.publishers.elasticsearch$post_records invoke "elasticsearch.clj" 119]
  [com.brunobonacci.mulog.publishers.elasticsearch.ElasticsearchPublisher publish "elasticsearch.clj" 230]
  [com.brunobonacci.mulog.core$start_publisher_BANG_$publish_attempt__2755 invoke "core.clj" 194]
  [clojure.core$binding_conveyor_fn$fn__5823 invoke "core.clj" 2050]
  [clojure.lang.AFn applyToHelper "AFn.java" 154]
  [clojure.lang.RestFn applyTo "RestFn.java" 132]
  [clojure.lang.Agent$Action doRun "Agent.java" 114]
  [clojure.lang.Agent$Action run "Agent.java" 163]
  [java.util.concurrent.ThreadPoolExecutor runWorker "ThreadPoolExecutor.java" 1136]
  [java.util.concurrent.ThreadPoolExecutor$Worker run "ThreadPoolExecutor.java" 635]
  [java.lang.Thread run "Thread.java" 833]]},
 :git-tag "e4bb9119-dirty",
 :publisher-id "4kTL6Ex9fhwc5MOpwJdP8J6ufZOAWJ9X",
 :publisher-type :elasticsearch}

This surprised me, seeing as this fix (switching from create to index) was tested as part of #92. Seems the version I tested 0.9.0-SNAPSHOT does not actually contain the change introduced in a0bde3d7.

[BrunoBonacci]

That's strange.... the test we performed was to make the publisher believe it was publishing to a normal index (see: https://github.com/BrunoBonacci/mulog/issues/92#issuecomment-1178946239)

[handerpeder]

Same error when using :index-pattern "'data-stream-name'". Not sure how, but I could have messed up the test somehow. Beginner user of elasticsearch and I'm finding the terminology exceptionally confusing.

[BrunoBonacci]

I recall testing it on Elasticsearch 7.x. I must have missed something too. Sorry about that, I will look into it again.