[BUG] Encountered a retryable error. Will Retry with exponential backoff {:code=>400, :url=>"https://localhost:9200/_bulk"}

[sultanovich]

Describe the bug When I try to send logstash OSS events to elastic I get the following error:

[2023-02-03T02:22:23,881][ERROR][logstash.outputs.elasticsearch][main][0b55d7846aa836aa6ab591125c3a16163b6a5fb910bc6b218f089797b2c2d4f1] Encountered a retryable error. Will Retry with exponential backoff  {:code=>400, :url=>"https://wazuh-indexer:9200/_bulk"}

This error occurs after upgrading from opensearch 1.3.0 to 2.4.1

To Reproduce Steps to reproduce the behavior:

1. Start a logstash pod with the following data
   • Dockerfile

     FROM docker.elastic.co/logstash/logstash-oss:7.16.2

   • Configmap (pipeline)

     
     input {
     beats {
         port => 5000
     }
     }
     filter {
     json {
       source => "message"
     }
     }
     output {
     elasticsearch {
         hosts => ["https://localhost:9200"]
         ilm_enabled => false
         index => "%{@index_name}"
         user => "${LOGSTASH_KS_USER}"
         password => "${LOGSTASH_KS_PASS}"
         ssl => true
         ssl_certificate_verification => true
         cacert => "/usr/share/logstash/config/ca-chain.cert.pem"
     }
     stdout { codec => rubydebug }
     }

2. Start a pod with elasticsearch opensearch 1.3.0. 
3. Send an event to logstash
4. View error in logstash logs.

[2023-02-07T15:13:19,164][ERROR][logstash.outputs.elasticsearch][main][4fa2049b909d01b7125bffcde8b40504c18ebda16a870126c55093583efa9846] Encountered a retryable error. Will Retry with exponential backoff {:code=>400, :url=>"https://localhost:9200/_bulk"}

**Expected behavior**
The expected behavior is that the event is correctly sent to elastic as it happened with opensearch 1.3.

**Plugins Logstash**
The logstash image includes a lot of plugins, but only `logstash-output-elasticsearch` is used.

[root@logstash-oss-86c7bf9485-4b6pv logstash]# grep logstash-output-elasticsearch Gemfile gem "logstash-output-elasticsearch", ">= 10.4.2" [root@logstash-oss-86c7bf9485-4b6pv logstash]# [root@logstash-oss-86c7bf9485-4b6pv logstash]# logstash-plugin list --installed --verbose --group output | grep logstash-output-elasticsearch OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release. WARNING: An illegal reflective access operation has occurred WARNING: Illegal reflective access by org.jruby.ext.openssl.SecurityHelper (file:/usr/share/logstash/vendor/jruby/lib/ruby/stdlib/jopenssl.jar) to field java.security.MessageDigest.provider WARNING: Please consider reporting this to the maintainers of org.jruby.ext.openssl.SecurityHelper WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations WARNING: All illegal access operations will be denied in a future release logstash-output-elasticsearch (10.7.3) [root@logstash-oss-86c7bf9485-4b6pv logstash]#

<details>

[root@logstash-oss-86c7bf9485-4b6pv logstash]# logstash-plugin list Using bundled JDK: /usr/share/logstash/jdk OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release. WARNING: An illegal reflective access operation has occurred WARNING: Illegal reflective access by org.jruby.ext.openssl.SecurityHelper (file:/usr/share/logstash/vendor/jruby/lib/ruby/stdlib/jopenssl.jar) to field java.security.MessageDigest.provider WARNING: Please consider reporting this to the maintainers of org.jruby.ext.openssl.SecurityHelper WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations WARNING: All illegal access operations will be denied in a future release logstash-codec-avro logstash-codec-cef logstash-codec-collectd logstash-codec-dots logstash-codec-edn logstash-codec-edn_lines logstash-codec-es_bulk logstash-codec-fluent logstash-codec-graphite logstash-codec-json logstash-codec-json_lines logstash-codec-line logstash-codec-msgpack logstash-codec-multiline logstash-codec-netflow logstash-codec-plain logstash-codec-rubydebug logstash-filter-aggregate logstash-filter-anonymize logstash-filter-cidr logstash-filter-clone logstash-filter-csv logstash-filter-date logstash-filter-de_dot logstash-filter-dissect logstash-filter-dns logstash-filter-drop logstash-filter-elasticsearch logstash-filter-fingerprint logstash-filter-geoip logstash-filter-grok logstash-filter-http logstash-filter-json logstash-filter-kv logstash-filter-memcached logstash-filter-metrics logstash-filter-mutate logstash-filter-prune logstash-filter-ruby logstash-filter-sleep logstash-filter-split logstash-filter-syslog_pri logstash-filter-throttle logstash-filter-translate logstash-filter-truncate logstash-filter-urldecode logstash-filter-useragent logstash-filter-uuid logstash-filter-xml logstash-input-azure_event_hubs logstash-input-beats logstash-input-couchdb_changes logstash-input-dead_letter_queue logstash-input-elasticsearch logstash-input-exec logstash-input-file logstash-input-ganglia logstash-input-gelf logstash-input-generator logstash-input-graphite logstash-input-heartbeat logstash-input-http logstash-input-http_poller logstash-input-imap logstash-input-jms logstash-input-pipe logstash-input-redis logstash-input-s3 logstash-input-snmp logstash-input-snmptrap logstash-input-sqs logstash-input-stdin logstash-input-syslog logstash-input-tcp logstash-input-twitter logstash-input-udp logstash-input-unix logstash-integration-jdbc ├── logstash-input-jdbc ├── logstash-filter-jdbc_streaming └── logstash-filter-jdbc_static logstash-integration-kafka ├── logstash-input-kafka └── logstash-output-kafka logstash-integration-rabbitmq ├── logstash-input-rabbitmq └── logstash-output-rabbitmq logstash-output-cloudwatch logstash-output-csv logstash-output-elastic_app_search logstash-output-elasticsearch logstash-output-email logstash-output-file logstash-output-graphite logstash-output-http logstash-output-lumberjack logstash-output-nagios logstash-output-null logstash-output-pipe logstash-output-redis logstash-output-s3 logstash-output-sns logstash-output-sqs logstash-output-stdout logstash-output-tcp logstash-output-udp logstash-output-webhdfs logstash-patterns-core [root@logstash-oss-86c7bf9485-4b6pv logstash]#

</details>

**Plugins Opensearch**

opensearch-alerting opensearch-anomaly-detection opensearch-asynchronous-search opensearch-cross-cluster-replication opensearch-geospatial opensearch-index-management opensearch-job-scheduler opensearch-knn opensearch-ml opensearch-neural-search opensearch-notifications opensearch-notifications-core opensearch-observability opensearch-performance-analyzer opensearch-reports-scheduler opensearch-security opensearch-security-analytics opensearch-sql repository-s3

**Host/Environment (please complete the following information):**
 - OS: CentOS
 - Version 7.9.2009

**Additional context**
The error occurs when deploying in EKS the new version of Wazuh v4.4. You can replicate the environment using the [Wazuh documentation](https://documentation.wazuh.com/current/deployment-options/deploying-with-kubernetes/kubernetes-deployment.html) and adding the logstash component to send events to Wazuh-indexer.

git clone https://github.com/wazuh/wazuh-kubernetes.git -b 4.4 --depth=1 cd wazuh-kubernetes

[dblock]

Does the 2.4.1 come up? Is there another error in the logs?

[sultanovich]

Hi @dblock. No, but upon investigation, we have come to the conclusion that it is because of this breaking change on Opensearch 2.x.

The question here then would be, is there any version of logstash OSS with the logstash-output-elasticsearch plugin that works on Opensearch 2.4.1?

[dblock]

Yes, it's https://rubygems.org/gems/logstash-output-opensearch. Does this work?

[sultanovich]

hi @dblock , just what we see is that with logstash-output-elasticsearch plugin in version 10.7.3 it does not work (I have updated the main comment with this information).

So the question is, is there any version of the logstash-output-elasticsearch plugin that works with Opensearch 2.4.1?

[dblock]

AFAIK no, but you can always ask folks that maintain logstash-output-elasticsearch.

We are actively maintaining logstash-output-opensearch. The errors above in the issue are all -elasticsearch, I don't clearly see an error with logstash-output-opensearch when trying to ingest data. Help narrow down the problem with logstash-output-elasticsearch and let's fix it? I'll move this issue into that project.

[sultanovich]

I understand what you are saying @dblock, the problem here is that we are still with one foot in each product. Surely when I ask the maintainer of logstash-output-elasticsearch which version works correctly on Opensearch 2.x he will tell me that since it is a different product he does not have that information. That is why I was asking here if you have information to confirm if any version higher than 10.7.3 of logstash-output-elasticsearch supports Opensearch 2.x.

For us, it is clear that the definitive solution is to replace the plugin and start using logstash-output-opensearch.

[dblock]

I understand what you are saying @dblock, the problem here is that we are still with one foot in each product. Surely when I ask the maintainer of logstash-output-elasticsearch which version works correctly on Opensearch 2.x he will tell me that since it is a different product he does not have that information. That is why I was asking here if you have information to confirm if any version higher than 10.7.3 of logstash-output-elasticsearch supports Opensearch 2.x.

I am 99% sure that there's no version of logstash-output-elasticsearch that supports OpenSearch 2.x.

[sultanovich]

I understand, thank you anyway for your help @dblock .

[dlvenable]

@sultanovich ,

In order to send data to an OpenSearch cluster from Logstash, you will need to use the logstash-output-opensearch plugin. The latest version of this plugin supports OpenSearch 1.x and 2.x.

You should be able modify your Dockerfile to include a command to run the following command.

bin/logstash-plugin install logstash-output-opensearch

And you can then change your output configuration to use the opensearch plugin instead of elasticsearch.

output {
        opensearch {
            hosts => ["https://localhost:9200"]
            index => "%{@index_name}"
            user => "${LOGSTASH_KS_USER}"
            password => "${LOGSTASH_KS_PASS}"
            ssl => true
            ssl_certificate_verification => true
            cacert => "/usr/share/logstash/config/ca-chain.cert.pem"
        }
        stdout { codec => rubydebug }
    }

[sultanovich]

Gracias @dlvenable ,

That is exactly what we ended up doing. What we wanted to confirm as I mentioned before, before proceeding to replace the plugin used, is if it was possible to get the same result temporarily with some higher version of logstash-output-elasticsearch to avoid a change in production that we would like to test further.