[Feature request] - ability to suppress certain bulk insertion errors

[ryn9]

Is your feature request related to a problem? Please describe. There are certain errors that get logged (sometimes many logs) that I would like to see suppressed, via configuration.

For example the following log is produced during a create action due to version conflict. [2023-03-21T15:50:24,743][WARN ][logstash.outputs.opensearch][output_opensearch_ds][a89a25b76c99767d555e0dd399e03e4ec77529effac1a077b7f2f7acffa4d665] Failed action {:status=>409, :action=>["create", {:_id=>"53e1de39-20f3-4ccd-ad28-806132c01eee", :_index=>"testtenant-testtype-1.0.0-2023.11", :routing=>nil}, {"message"=>"REDACTED"], :response=>{"create"=>{"_index"=>"testtenant-testtype-1.0.0-2023.11", "_id"=>"53e1de39-20f3-4ccd-ad28-806132c01eee", "status"=>409, "error"=>{"type"=>"version_conflict_engine_exception", "reason"=>"[53e1de39-20f3-4ccd-ad28-806132c01eee]: version conflict, document already exists (current version [1])", "index"=>"testtenant-testtype-1.0.0-2023.11", "shard"=>"0", "index_uuid"=>"VJ6E1jgvT5CdZqcd7SBbmQ"}}}} In this case - this rejection is by design to avoid duplicates being inserting into a datastream.

Describe the solution you'd like Would like configs options to suppress errors with some granularity. Some idea:

• 'status' (which would be '409' in the example above)
• 'error.type' (which would be 'version_conflict_engine_exception' in the example above)
• 'error.reason' (maybe could do a string-contains style function to match 'version conflict, document already exists' in the example above)

Describe alternatives you've considered I have looked at customizing the plugins logging using log4j as described here (https://www.elastic.co/guide/en/logstash/current/logging.html) There does not seem to be sufficient granularity there. It appears the only way to suppress logs are by log level.

[ryn9]

@dlvenable any questions about the request? Did I provide enough information?

[caviliar]

I came across also wanting to silence certain error types.

I found this in the code

# Set the OpenSearch errors in the whitelist that you don't want to log.
# A useful example is when you want to skip all 409 errors
# which are `document_already_exists_exception`.
:failure_type_logging_whitelist => { :validate => :array, :default => [] }

the wording is unfortunate as the flag is called a whitelist but you put logs you don't want to see in it - i.e a blacklist...

my config is:

opensearch {
        hosts => [%ES_HOSTS%]
        index => "index"
        ...
        failure_type_logging_whitelist => ["document_already_exists_exception","version_conflict_engine_exception"]
    }

note: i know the comment says document_already_exists_exception, but i think it's meant to be version_conflict_engine_exception.

looks like elastic also deprecated this flag in lieu of silence_errors_in_log. but who knows if opensearch will follow.

[ryn9]

@caviliar - thank you!

@dlvenable - Do you think that can get added to the documentation? Will there be any plans to implement the alternate name 'silence_errors_in_log' ?