Impact of LetsEncrypt certificate expiry

[enys]

Logstash information:

Please include the following information:

1. Logstash version (e.g. bin/logstash --version)
2. Logstash installation source (e.g. built from source, with a package manager: DEB/RPM, expanded from tar or zip archive, docker)
3. How is Logstash being run (e.g. as a service/service manager: systemd, upstart, etc. Via command line, docker/kubernetes)
4. How was the Logstash Plugin installed

5. This was tested will all the official logstash docker images since 7.9.1 (Such as docker.elastic.co/logstash/logstash:7.10.2)

OS version (uname -a if on a Unix-like system): Linux 6925850df20f 5.4.0-1036-kvm #37-Ubuntu SMP Fri Mar 19 20:15:18 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Description of the problem including expected versus actual behavior: Connection to object storage behind https with LetsEncryt certificate fails, with either expired certificate, or missing certificate in the case of an updated certificate trust store

My current understanding is that it could be linked to: https://github.com/jruby/jruby-openssl/issues/236 Steps to reproduce:

With the following pipeline output

output {
  s3{
    endpoint => "https://s3.fr-par.scw.cloud"
    region => "fr-par"           #         (optional, default = "us-east-1")
    bucket => "xxx"     #             (required)
    prefix => "%{+YYYY}/%{+MM}/%{+dd}"
    size_file => 10485760           #         (optional) - Bytes 10MB
    time_file => 30                  #         (optional) - Minutes
    temporary_directory => "/usr/share/logstash/data/s3tmp/"
    codec => "json_lines"                #         (optional)
    encoding => "gzip"             #
    #server_side_encryption => true
    canned_acl => "private"        #          (optional. Options are "private", "public-read", "public-read-write", "authenticated-read", "aws-exec-read", "bucket-owner-read", "bucket-owner-full-control", "log-delivery-write". Defaults to "private" )
    validate_credentials_on_root_bucket => true
    additional_settings => {
      force_path_style => true
      follow_redirects => false
      ssl_verify_peer => false
    }
  }
}

Provide logs (if relevant):

logstash_1  | [2021-10-01T15:11:39,916][ERROR][logstash.outputs.s3      ][main] Error validating bucket write permission
s! {:message=>"certificate verify failed", :class=>"Seahorse::Client::NetworkingError", :backtrace=>["uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http.rb:1002:in `connect'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http.rb:924:in `do_start'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http.rb:919:in `start'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/delegate.rb:83:in `method_missing'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/seahorse/client/net_http/connection_pool.rb:285:in `start_session'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/seahorse/client/net_http/connection_pool.rb:92:in `session_for'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/seahorse/client/net_http/handler.rb:119:in `session'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/seahorse/client/net_http/handler.rb:71:in `transmit'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/seahorse/client/net_http/handler.rb:45:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/seahorse/client/plugins/content_length.rb:12:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/aws-sdk-core/plugins/s3_request_signer.rb:88:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/aws-sdk-core/plugins/s3_request_signer.rb:23:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/aws-sdk-core/plugins/s3_host_id.rb:14:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/aws-sdk-core/xml/error_handler.rb:8:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/aws-sdk-core/plugins/helpful_socket_errors.rb:10:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/aws-sdk-core/plugins/s3_request_signer.rb:65:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/aws-sdk-core/plugins/s3_redirects.rb:15:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/aws-sdk-core/plugins/retry_errors.rb:108:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/aws-sdk-core/plugins/retry_errors.rb:139:in `retry_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/aws-sdk-core/plugins/retry_errors.rb:122:in `retry_if_possible'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/aws-sdk-core/plugins/retry_errors.rb:110:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/aws-sdk-core/plugins/retry_errors.rb:139:in `retry_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/aws-sdk-core/plugins/retry_errors.rb:122:in `retry_if_possible'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/aws-sdk-core/plugins/retry_errors.rb:110:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/aws-sdk-core/plugins/retry_errors.rb:139:in `retry_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/aws-sdk-core/plugins/retry_errors.rb:122:in `retry_if_possible'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/aws-sdk-core/plugins/retry_errors.rb:110:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/aws-sdk-core/plugins/s3_dualstack.rb:32:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/aws-sdk-core/plugins/s3_accelerate.rb:49:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/aws-sdk-core/plugins/s3_md5s.rb:31:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/aws-sdk-core/plugins/s3_expect_100_continue.rb:21:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/aws-sdk-core/plugins/s3_bucket_name_restrictions.rb:12:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/aws-sdk-core/rest/handler.rb:7:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/aws-sdk-core/plugins/user_agent.rb:12:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/seahorse/client/plugins/endpoint.rb:41:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/aws-sdk-core/plugins/param_validator.rb:21:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/seahorse/client/plugins/raise_response_errors.rb:14:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/aws-sdk-core/plugins/s3_sse_cpk.rb:19:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/aws-sdk-core/plugins/s3_dualstack.rb:24:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/aws-sdk-core/plugins/s3_accelerate.rb:34:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:20:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/aws-sdk-core/plugins/idempotency_token.rb:18:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/aws-sdk-core/plugins/param_converter.rb:20:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/seahorse/client/plugins/response_target.rb:21:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/seahorse/client/request.rb:70:in `send_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/seahorse/client/base.rb:207:in `block in define_operation_methods'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.632/lib/aws-sdk-resources/services/s3/file_uploader.rb:42:in `block in put_object'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.632/lib/aws-sdk-resources/services/s3/file_uploader.rb:52:in `open_file'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.632/lib/aws-sdk-resources/services/s3/file_uploader.rb:41:in `put_object'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.632/lib/aws-sdk-resources/services/s3/file_uploader.rb:34:in `upload'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.632/lib/aws-sdk-resources/services/s3/object.rb:252:in `upload_file'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-s3-4.3.2/lib/logstash/outputs/s3/write_bucket_permission_validator.rb:43:in `upload_test_file'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-s3-4.3.2/lib/logstash/outputs/s3/write_bucket_permission_validator.rb:18:in `valid?'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-s3-4.3.2/lib/logstash/outputs/s3.rb:210:in `register'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:131:in `register'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:68:in `register'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:228:in `block in register_plugins'", "org/jruby/RubyArray.java:1809:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:227:in `register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:585:in `maybe_setup_out_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:240:in `start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:185:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:137:in `block in start'"]}

[rhanka]

Had exactly the same problem since yesterday, with logstash 7.14.1 and v3.7.0 of s3 input pluging

[enys]

looks related to https://github.com/elastic/logstash/issues/13261

[jsvd]

We're exploring ways to both fix this and workaround it until a fix is out. In the case of this plugin we don't have much access to how the AWS SDK performs requests. The only option I see at the moment is adding a line in the plugins register method to disable verification. The SDK seems to provide Aws.config[:ssl_verify_peer] = false, can anyone test it?

[jsvd]

if that works, a way to inject it without editing code would be leverage the ruby filter:

filter {
  # ...
  ruby { init => 'Aws.config[:ssl_verify_peer] = false' code => "" } 
  # ...
}

This is not ideal at all, but may get data flowing until there's a fix. beware that this lowers security so only do it if you're comfortable with the implication.

[kares]

the global setting for disabling peer verification Aws.config[:ssl_verify_peer] = false seems to get ignored but with the https://github.com/logstash-plugins/logstash-output-s3/pull/241 update, disabling at the plugin level works as a work-around (until the real cause is resolved upstream):

    additional_settings => {
      ssl_verify_peer => false
    }