SSL Verification Failure when using lumberjack to send to another logstash instance

[utilitynerd]

I'm running into an issue where the logstash-output-lumberjack plugin is failing to verify an SSL certificate. I believe this is a bug, since both logstash-forwarder and openssl successfully verify the SSL certificate using the same CA file as logstash.

Environment:

• RedHat 6.7
• Logstash 1.5.4 (all plugins updated)
• logstash-output-plugin 1.0.2
• Java 1.8.0.51

Logstash Sever config snippet:

lumberjack { 
    port => 5000
    ssl_certificate => "/etc/pki/tls/certs/sever.cert"
    ssl_key => "/etc/pki/tls/private/server.key"
}

Logstash Client config snippet

lumberjack {
    hosts => ['server.fqdn']
    port => 5000
    ssl_certificate => "/etc/pki/tls/certs/logstash.ca"
    codec => json { }
}

Here is the error I get in logstash.log:

All hosts unavailable, sleeping {:hosts=>["X.X.X.X"], :e=>#<OpenSSL::SSL::SSLError: certificate verify failed>,

As mentioned above, both logstash-forwarder and openssl have no issues verifying the certificate.

logstash-forwarder config snippet:

{
  "network": {
    "servers": [ "server.fqdn:5000"],
    "ssl ca": "logstash.ca",
    "timeout": 15
  },

and when run, I get this output:

$ /usr/local/bin/logstash-forwarder --config test.json
2015/09/01 14:03:02.579180 Setting trusted CA from file: logstash.ca
2015/09/01 14:03:02.583387 Connecting to [X.X.X.X]:5000 (server.fqdn)
2015/09/01 14:03:02.643339 Connected to X.X.X.X
2015/09/01 14:03:04.204200 Registrar: processing 1024 events
...

OpenSSL verification

$ openssl verify -CAfile logstash.ca server.cert
server.cert: OK

Note: server.crt is a SAN certificate and is valid for both the server.fqdn as well the server's IP address.

[plarivee]

having similar issue when update the output to logstash 1.5-6.

Lumberjack output => Lumberjack Input LS (1-4.5 ) => LS (1.4-5 ) Ok LS (1-4.5 ) => LS (1.5-6 ) OK LS (1-5.6 ) => LS (1.5-6 ) FAIL LS (1-5.6 ) => LS (1.4-5 ) FAIL

[plarivee]

issue seems to be related to Lumberjack/client

https://github.com/elastic/ruby-lumberjack/blob/master/lib/lumberjack/client.rb#L79-L97

[plarivee]

Changing to "VERIFY_NONE" makes it happy for now ( obviously )

https://github.com/elastic/ruby-lumberjack/blob/master/lib/lumberjack/client.rb#L91

I'm checking the Ruby Openssl to see what can be done

[m4ce]

I am also experiencing this issue

[asafm]

Same here

[paulczar]

We also experience this issue. need to be able to support self signed certs via https://github.com/logstash-plugins/logstash-output-lumberjack/pull/18 so that we don't have to hack around this.

[elisiano]

on the same boat. I was thinking to add our CA to the system CA certs bundle, but I'm not sure which one is used by JRuby

Did anyone find a solution?

my versions: OS: CentOS 7.2 Logstash: 5.0.2

[jordansissel]

This may be confusing, but my reading of the code (ruby-lumberjack, etc) makes me believe the ssl_certificate setting is the one that is used today, without code changes, for validating the server.

https://github.com/plarivee/ruby-lumberjack/blob/master/lib/lumberjack/client.rb#L90-L91

We set OpenSSL::SSL::SSLContext.cert_store, and this is used for certificate verification.

My guess is to try setting ssl_certificate to the path containing the necessary data to validate your server (logstash beats input?).

[elisiano]

@jordansissel I think I tried everything I could.

• use the client cert in ssl_certificate
• use the ca file in ssl_certificate
• bundle the ca file and client together and use that as ssl_certificate
• adding the ca to ${JAVA_HOME}/jre/lib/security/cacerts with keytool (on both "client" and "server")

And no, I'm not using lumberjack to receive files from filebeat because that doesn't work either (regardless of SSL). So I'm using filebeat to forward files to a local logstash which then communicates with a remote logstash via lumberjack (so I'm using the additional plugin logstash-lumberjack-output on the shipper).

EDIT: Just to be clear, I'm trying to do that but not being able to.

[plarivee]

I think that is why i implemented the ca cert since bundling ca cert + cert wasnt working

Pascal Larivée Photographe lariveephoto.com

On Dec 3, 2016 10:14, "Elisiano Petrini" notifications@github.com wrote:

@jordansissel https://github.com/jordansissel I think I tried everything I could.

• use the client cert in ssl_certificate
• use the ca file in ssl_certificate
• bundle the ca file and client together and use that as ssl_certificate
• adding the ca to ${JAVA_HOME}/jre/lib/security/cacerts with keytool (on both "client" and "server")

And no, I'm not using lumberjack to receive files from filebeat because that doesn't work either (regardless of SSL). So I'm using filebeat to forward files to a local logstash which then communicates with a remote logstash via lumberjack (so I'm using the additional plugin logstash-lumberjack-output on the shipper)

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/logstash-plugins/logstash-output-lumberjack/issues/11#issuecomment-264644853, or mute the thread https://github.com/notifications/unsubscribe-auth/AD0hpQ6DrtF87cE3WClc06oewmQnmzXwks5rEYc5gaJpZM4F2GBB .

[elisiano]

@plarivee can you clarify how you got it working (without modifying ruby code)? I'm also using a CA (I'm leveraging Puppet's CA for that).

[plarivee]

@elisiano Well i did modify code and submited a pull request (check earlier in the code ) this was to add support for the ca_cert option,

else you could set verify_none in the code also, but defeats security

[elisiano]

@plarivee thanks for the clarification, now I start to put all the pieces together.

In theory though, this setup should work even if we add our custom CA to the cacerts that logstash uses:

# rpm -ql logstash|grep cacert
/usr/share/logstash/logstash-core/lib/logstash/certs/cacert.pem
# (echo -e "\nPuppetCA\n========" && cat /var/lib/puppet/ssl/certs/ca.pem ) >> /usr/share/logstash/logstash-core/lib/logstash/certs/cacert.pem
# systemctl restart logstash.service

But even that didn't work for me (added on both logstash instances).

@jordansissel: just out of curiosity, is there any reason why the PR of @plarivee wasn't merged?

[plarivee]

Are you still having this issue with the latest logstash? we havent updated our log infra in the last year, thats why I havent pushed.more, thinking it might be solved now

[elisiano]

Yes, I'm running the latest:

# rpm -q logstash 
logstash-5.0.2-1.noarch

[mqamouser]

On line 84 of lib/logstash/outputs/lumberjack.rb, hostnames are converted to IP address before creating a new Lumberjack::Client, which then performs the connection and verifies the SSL cert.

Doesn't this guarantee an SSL failure unless you have an IP: entry in the SAN for the SSL cert?

[fatalglitch]

@mqa-mouser did you ever figure this out? I'm questioning the same thing, as I'm recently trying to get logstash -> logstash working with lumberjack output to beats input

[jhaprins]

I just found this bug doing a search after trying to debug this exact same issue the last couple of hours. We were also down to the source code to find out that it would probably never work because there is no way the correct CA can be selected.

@jordansissel can you verify that your pull request fixes this issue?

If that is the case, I'm going to create my own patched version.

[plarivee]

Are you using latest versions of logstash @jhaprins ?

[jhaprins]

I'm currently using 6.3.1. I would have to check where my colleagues picked up this version. I think they got it from some ElasticSearch repo.

[jhaprins]

/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-lumberjack-3.1.7/lib/logstash/outputs/lumberjack.rb

[plarivee]

have you tried including the CA in the cert? the lumberjack client is using https://docs.ruby-lang.org/en/2.4.0/OpenSSL/X509/Store.html to add the cert in the cert store.

I'm not using lumberjack anymore so cannot test but could be something to test easily @jhaprins

[jhaprins]

I have to check with my colleagues to find out what they want to use exactly and why. Could we have a quick chat? You say you don't use lumberjack anymore? What do you use to send data from one logstash server to an other?

We have tried adding the CA in the cert file, but this doesn't help.

[BobBlank12]

Here is a working solution for Logstash using Lumberjack output to BEATS input on a downstream logstash instance:

#Generate a CA, Key + Signed Cert
/opt/elasticsearch-6.4.2/bin/elasticsearch-certutil cert --pem
unzip certificate-bundle.zip

# Convert the key to PKCS8
openSSL pkcs8 -in ./instance/instance.key -topk8 -nocrypt -out ./instance/instance.pk8

###################################################################
#Upstream Logstash Server

# This input could be anything...
input {
  stdin {}
  }
}

# Send the output to a downstream server
output {
   lumberjack {
      codec => json
      hosts => [ "127.0.0.1" ]
      port => 5044
      ssl_certificate => "/Users/Downloads/lumberjack/certs/ca.crt"
   }
}

###################################################################
#Downstream Logstash Server
#
#  Using the BEATS input to receive data from the upsteam Logstash server
#  which is using the lumberjack output.
#
input {
  beats {
    id => "mylumberjack"
    codec => json
    port => 5044
    ssl_certificate => "/Users/Downloads/lumberjack/certs/instance.crt"
    ssl_key => "/Users/Downloads/lumberjack/certs/instance.pk8"
    ssl => true
  }

}

output {
   stdout { codec => rubydebug }
}

[garima098]

I am also facing the same issue in 6.4.1.

Is there any specific solution regarding this issue?

[garima098]

Also, i have raised a post in community.

https://discuss.elastic.co/t/getting-lumberjack-all-host-unavailable-sleeping-error-when-logstash-to-logstash-communication-is-configured/179903

Is there any workaround to solve this issue?