CloudWatch Publisher: error when transform function returns no events

[MastroCiccio]

Hi, I noticed the CloudWatch publisher to generate an error event whenever the transform function returns the empty sequence since the AWS API PutLogEvents does not accept the empty array as the input.

Here the details of the error from the mulog event:

{:publisher-type   :cloudwatch,
 :publisher-id     "4xdL2TzAC-JxUQldM0ZhQtavsek67jsa",
 :mulog/namespace  "clojure.core",
 :mulog/action     :publish,
 :mulog/timestamp  1718780832832,
 :mulog/origin     :mulog/core,
 :mulog/trace-id   #mulog/flake "4xdL2tisnXGbnpUNPIerAvVG4Yt_DBh-",
 :mulog/event-name :mulog/publisher-error,
 :exception        #error {:cause "μ/log cloudwatch publisher publish failure, group 'dev-log' stream '4xdL2SoPQ1pBbbsx2Gx7TZCZnlg53jc0' reason: 1 validation error detected: Value '[]' at 'logEvents' failed to satisfy constraint: Member must have length greater than or equal to 1 "
                           :data  {:rs          {:__type                       "InvalidParameterException",
                                                 :message                      "1 validation error detected: Value '[]' at 'logEvents' failed to satisfy constraint: Member must have length greater than or equal to 1",
                                                 :cognitect.aws.http/status    400,
                                                 :cognitect.anomalies/category :cognitect.anomalies/incorrect,
                                                 :cognitect.aws.error/code     "InvalidParameterException"},
                                   :group-name  "dev-log",
                                   :stream-name "4xdL2SoPQ1pBbbsx2Gx7TZCZnlg53jc0"}
                           :via   [{:type clojure.lang.ExceptionInfo
                                    :message "μ/log cloudwatch publisher publish failure, group 'dev-log' stream '4xdL2SoPQ1pBbbsx2Gx7TZCZnlg53jc0' reason: 1 validation error detected: Value '[]' at 'logEvents' failed to satisfy constraint: Member must have length greater than or equal to 1 "
                                    :data {:rs          {:__type                       "InvalidParameterException",
                                                         :message                      "1 validation error detected: Value '[]' at 'logEvents' failed to satisfy constraint: Member must have length greater than or equal to 1",
                                                         :cognitect.aws.http/status    400,
                                                         :cognitect.anomalies/category :cognitect.anomalies/incorrect,
                                                         :cognitect.aws.error/code     "InvalidParameterException"},
                                           :group-name  "dev-log",
                                           :stream-name "4xdL2SoPQ1pBbbsx2Gx7TZCZnlg53jc0"}
                                    :at [com.brunobonacci.mulog.publishers.cloudwatch$publish_BANG_ invokeStatic "cloudwatch.clj" 58]}]
                           :trace [[com.brunobonacci.mulog.publishers.cloudwatch$publish_BANG_ invokeStatic "cloudwatch.clj" 58]
                                   [com.brunobonacci.mulog.publishers.cloudwatch$publish_BANG_ invoke "cloudwatch.clj" 43]
                                   [com.brunobonacci.mulog.publishers.cloudwatch$put_log_events invokeStatic "cloudwatch.clj" 76]
                                   [com.brunobonacci.mulog.publishers.cloudwatch$put_log_events invoke "cloudwatch.clj" 70]
                                   [com.brunobonacci.mulog.publishers.cloudwatch.CloudwatchPublisher publish "cloudwatch.clj" 103]
                                   [com.brunobonacci.mulog.core$start_publisher_BANG_$publish_attempt__8976 invoke "core.clj" 194]
                                   [clojure.core$binding_conveyor_fn$fn__5823 invoke "core.clj" 2050]
                                   [clojure.lang.AFn applyToHelper "AFn.java" 154]
                                   [clojure.lang.RestFn applyTo "RestFn.java" 132]
                                   [clojure.lang.Agent$Action doRun "Agent.java" 114]
                                   [clojure.lang.Agent$Action run "Agent.java" 163]
                                   [java.util.concurrent.ThreadPoolExecutor runWorker "ThreadPoolExecutor.java" 1136]
                                   [java.util.concurrent.ThreadPoolExecutor$Worker run "ThreadPoolExecutor.java" 635]
                                   [java.lang.Thread run "Thread.java" 840]]}}

Here a possible implementation to replicate the scenario:

(require '[[com.brunobonacci.mulog :as ml])
(ml/start-publisher! {:type :console :pretty? true})
(ml/start-publisher! {:type :cloudwatch :group-name "dev-log" :transform (partial filter (comp :test-event :mulog/event-name))})
(ml/log :test-event)

Is this a desired behavior? I'm implementing a solution with different publishers: since the CloudWatch publisher matches only a specific set of events, it wouldn't be unusual for the transform function to return the empty seq, thus generating the error event catched by the other publishers. For now I'm filtering out this specific error from the other publishers, but it's kind of annoying...

Thanks for the help

[BrunoBonacci]

It looks like a bug to me. I'll fix it in the next release.

[MastroCiccio]

Thank you!

[MastroCiccio]

Any news about this issue?

I've looked at the code of the CloudwatchPublisher type in the namespace com.brunobonacci.mulog.publishers.cloudwatch I think the bug could be solved by updating the method publish as follows:

(publish [_ buffer]
         ;; items are pairs [offset <item>]
         (let [items (take (:max-items config) (rb/items buffer))
               last-offset (-> items last first)]
           (if-not (seq items)
             buffer
             ;; else send to cloudwatch
             (let [transformed-items (transform (map second items))]
               (when (seq transformed-items)
                 (put-log-events cw-client stream-name config transformed-items next-token))
               (rb/dequeue buffer last-offset)))))

I can submit a PR if you prefer.

[BrunoBonacci]

Sorry, I've been swamped with work.

I hope to release it in the next few weeks. I'd like to check this issue across all publishers and not jut the CloudWatch one.