logstash input plugin TLS port hangs causes denial of service (DOS)

[jmschmaus]

We want to move https://github.com/elastic/logstash/issues/7650 to logstash-input-tcp, which I didn't know about when I opened the ticket.

Here is text from #7650:

Logstash 5.3 uses jruby-openssl 0.9.16 for SSL/TLS. jruby-openssl SSLSocket.java does SSL handshake. SSLSocket will not return until the SSL handshake is complete. Logstash will not do another accept on the TLS socket until SSL handshake is complete. If a non-TLS client connects to logstash's TLS port, the port is hung and unusable for any other clients.

For all general issues, please provide the following details for fast resolution:

Version: 5.3.0 Operating System: CentOS 6 Config File (if you have sensitive info, please remove it): Please see below for TLS configuration Sample Data: Steps to Reproduce: 1) Configure ssl for logshipper tcp input:

input {
...
tcp {
host => "0.0.0.0"
type => "tcp_json_event"
port => "10059"
ssl_enable => true # This needs to be true for the other ssl parameters to be considered
ssl_verify => false # Don't validate the cert against the CA. Useful for self signed certs
ssl_cert => "ssl cert" # SSL Cert
ssl_key => "ssl key"
ssl_extra_chain_certs => "trusted cert" # CA certs
codec => json_lines{
charset => "ISO-8859-1"
}
...

2) Verify TLS operation via openssl s_client -connect localhost:10059

3) Open non-TLS connection to port 10059, configured for TLS: nc localhost 10059 4) Verify SSL handshake no longer operational (step (2)).

[jdlourenco]

I am experiencing a similar issue, tough I am not sure it is exactly the same.

I have a tcp input with tls enabled which seems to work fine for some time. When I try to connect using openss I usually get this:

openssl s_client -connect machine:9400 -CAfile config/certs/ca.hidra-prd-logstash.pem -cert config/certs/las.hidra-prd-logstash.crt -key config/certs/las.hidra-prd-logstash.key
CONNECTED(00000003)
depth=1 C = PT, ST = Lisboa, L = Default City, O = XXX, OU = DCY-CSD, emailAddress = XXX@XXX.XX
verify return:1
depth=0 C = PT, ST = Lisboa, L = Default City, O = XXX, OU = DCY-CSD, CN = logstash, emailAddress = XXX@XXX.XX
verify return:1
---
Certificate chain
 0 s:/C=PT/ST=Lisboa/L=Default City/O=XXX/OU=DCY-CSD/CN=logstash/emailAddress=XXX@XXX.XX
   i:/C=PT/ST=Lisboa/L=Default City/O=XXX/OU=DCY-CSD/emailAddress=XXX@XXX.XX
 1 s:/C=PT/ST=Lisboa/L=Default City/O=XXX/OU=DCY-CSD/emailAddress=XXX@XXX.XX
   i:/C=PT/ST=Lisboa/L=Default City/O=XXX/OU=DCY-CSD/emailAddress=XXX@XXX.XX
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIDgjCCAmoCCQDnQz+vbbVU6TANBgkqhkiG9w0BAQsFADB5MQswCQYDVQQGEwJQ
VDEPMA0GA1UECAwGTGlzYm9hMRUwEwYDVQQHDAxEZWZhdWx0IENpdHkxDzANBgNV
BAoMBkFsdGljZTEQMA4GA1UECwwHRENZLUNTRDEfMB0GCSqGSIb3DQEJARYQcHVs
c29AdGVsZWNvbS5wdDAeFw0xNzA5MjUxNDU5NTdaFw0yMDA3MTUxNDU5NTdaMIGM
MQswCQYDVQQGEwJQVDEPMA0GA1UECAwGTGlzYm9hMRUwEwYDVQQHDAxEZWZhdWx0
IENpdHkxDzANBgNVBAoMBkFsdGljZTEQMA4GA1UECwwHRENZLUNTRDERMA8GA1UE
AwwIbG9nc3Rhc2gxHzAdBgkqhkiG9w0BCQEWEHB1bHNvQHRlbGVjb20ucHQwggEi
MA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDidCg1cp9Jp4dVbMlWd7f1XQ4b
B8IjNP+FbfKyakccopvAaA+PkRG9iZ+VQjs+RNwjazleDDXIF2BPiWS7tMT0s6hw
XEeWVZ02E8X3oKA3NXz5Nv0j8p7Pzc95kkTR1o3fCVSIc+z9+5WCAn0k4jyKkj82
FMIkGPbcMlYBFBbYUsiH3YUwhrj6Zsdvm3qrR8al7rscrkKYLawn0v3reXNiTIwY
Q0Hpj821SQk+OqGJB2OZbApFlb0jdycdkggoWtQjVzRPT+bYaqZYztprBVIatt5J
yjRNZ3WPsDKMtem+df26BHr2+WyI3fMoWYIBQm4WOsAcmDaGwmKF7XpPqK9XAgMB
AAEwDQYJKoZIhvcNAQELBQADggEBAFjKLjgQovkZWsW61IMdxQOy1VHIJS79X2Iz
FteAXf0s1kdkFY0YV+5TDisEn0P2mOIrjQ2tWfJbJEiKuiJ4rdRUNGwpbV3Il8gL
YRxbxfXOSnBMQ7MYAxqv6ALsVAcG3aqKcq32cwSjXSWXvKERNlvDYFm5MgYrbibj
IynQ7KYWDJqHDWC4WsMFDFQHR2gaYvVZ6YfnVAaOlzpzAQ1ZM48Spl8S3kYSxnRi
W1i5HcU0IuPOTqJ7lCHDMTmJNSaCS3amwWJU0UCADfxkzNRKe+Eq70CQtzS8ZdvV
pVQF7sdtMJFUfyzpw0B5agoKuREt0I70Vm7EhiTgLNVdQv5f8Ao=
-----END CERTIFICATE-----
subject=/C=PT/ST=Lisboa/L=Default City/O=XXX/OU=DCY-CSD/CN=logstash/emailAddress=XXX@XXX.XX
issuer=/C=PT/ST=Lisboa/L=Default City/O=XXX/OU=DCY-CSD/emailAddress=XXX@XXX.XX
---
No client certificate CA names sent
Server Temp Key: ECDH, secp521r1, 521 bits
---
SSL handshake has read 2500 bytes and written 2634 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES128-SHA256
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES128-SHA256
    Session-ID: 5A0C3B2DB5E145C12D67E8494675F1449F77073B1BB6DB7EBDCAC44322E21E8F
    Session-ID-ctx:
    Master-Key: 3C1DD3AABC322E3344B0D279B79B0A6579E83B9512005951303952A946F926BD7F1F25AAF13A3DCDB20DBD9D052512AE
    Key-Arg   : None
    Krb5 Principal: None
    PSK identity: None
    PSK identity hint: None
    Start Time: 1510751021
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---

However, after running for some time I am no longer able to establish a new connection as it hangs:

openssl  s_client -connect localhost:9400 -CAfile ca.pem -cert cert.crt -key cert.key
CONNECTED(00000003)

Listing all connections to my logstash instance I didn't find any non-TLS, and they all seemed to be working and sending data, so I'm not sure the issue is the same.

@jmschmaus were you able to connect again after closing the non-TLS connection or was it no longer able to establish any other new TLS connection after that?

[jmschmaus]

Hi José,

That’s exactly what we see when TLS is hung. CONNECTED and nothing else.

The logstash input plugin has send SYN,ACK back to the far end and the connection is ESTABLISHED, just like the healthy connections.

The logstash input plugin waits forever in SSLSocket.accept for a Client Hello, which never comes from a non-TLS client.

To find the offending connection (on CentOS 7):

- kill -3 <logstash input process pid>
- journalctl -u <logstash input process service name> >/tmp/out
- vi (or emacs…) /tmp/out.  Look for SSLSocket.accept.  Then search backwards for nid.  This is the hex thread ID of the hung logstash input thread.
- echo -e “ibase=16\n<nid value>” | bc  to translate it to decimal
- strace <thread id>    <— this should be in epoll_wait(<file descriptor>).  Remember the file descriptor.
- cat /proc/<logstash input pid>/fdinfo/<file descriptor, from above>
- One of the file descriptors shown is the hung connection.
- lsof -p <logstash input process> > /tmp/ls.lsof
- Search for both file descriptors in /tmp/ls.lsof.  One of them should show a connection to the TLS port and a remote IP address.

That’s the offending application. Log into the IP address, run netstat -pan | grep . Killing the process on the remote server will free the port, again.

John

On Nov 15, 2017, at 6:09 AM, José Lourenço notifications@github.com wrote:

I am experiencing a similar issue, tough I am not sure it is exactly the same.

I have a tcp input with tls enabled which seems to work fine for some time. When I try to connect using openss I usually get this:

openssl s_client -connect log2.sec.pulso.telecom.pt:9400 -CAfile config/certs/ca.hidra-prd-logstash.pem -cert config/certs/las.hidra-prd-logstash.crt -key config/certs/las.hidra-prd-logstash.key CONNECTED(00000003) depth=1 C = PT, ST = Lisboa, L = Default City, O = Altice, OU = DCY-CSD, emailAddress = pulso@telecom.pt verify return:1 depth=0 C = PT, ST = Lisboa, L = Default City, O = Altice, OU = DCY-CSD, CN = logstash, emailAddress = pulso@telecom.pt verify return:1

Certificate chain 0 s:/C=PT/ST=Lisboa/L=Default City/O=Altice/OU=DCY-CSD/CN=logstash/emailAddress=pulso@telecom.pt i:/C=PT/ST=Lisboa/L=Default City/O=Altice/OU=DCY-CSD/emailAddress=pulso@telecom.pt 1 s:/C=PT/ST=Lisboa/L=Default City/O=Altice/OU=DCY-CSD/emailAddress=pulso@telecom.pt i:/C=PT/ST=Lisboa/L=Default City/O=Altice/OU=DCY-CSD/emailAddress=pulso@telecom.pt

Server certificate -----BEGIN CERTIFICATE----- MIIDgjCCAmoCCQDnQz+vbbVU6TANBgkqhkiG9w0BAQsFADB5MQswCQYDVQQGEwJQ VDEPMA0GA1UECAwGTGlzYm9hMRUwEwYDVQQHDAxEZWZhdWx0IENpdHkxDzANBgNV BAoMBkFsdGljZTEQMA4GA1UECwwHRENZLUNTRDEfMB0GCSqGSIb3DQEJARYQcHVs c29AdGVsZWNvbS5wdDAeFw0xNzA5MjUxNDU5NTdaFw0yMDA3MTUxNDU5NTdaMIGM MQswCQYDVQQGEwJQVDEPMA0GA1UECAwGTGlzYm9hMRUwEwYDVQQHDAxEZWZhdWx0 IENpdHkxDzANBgNVBAoMBkFsdGljZTEQMA4GA1UECwwHRENZLUNTRDERMA8GA1UE AwwIbG9nc3Rhc2gxHzAdBgkqhkiG9w0BCQEWEHB1bHNvQHRlbGVjb20ucHQwggEi MA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDidCg1cp9Jp4dVbMlWd7f1XQ4b B8IjNP+FbfKyakccopvAaA+PkRG9iZ+VQjs+RNwjazleDDXIF2BPiWS7tMT0s6hw XEeWVZ02E8X3oKA3NXz5Nv0j8p7Pzc95kkTR1o3fCVSIc+z9+5WCAn0k4jyKkj82 FMIkGPbcMlYBFBbYUsiH3YUwhrj6Zsdvm3qrR8al7rscrkKYLawn0v3reXNiTIwY Q0Hpj821SQk+OqGJB2OZbApFlb0jdycdkggoWtQjVzRPT+bYaqZYztprBVIatt5J yjRNZ3WPsDKMtem+df26BHr2+WyI3fMoWYIBQm4WOsAcmDaGwmKF7XpPqK9XAgMB AAEwDQYJKoZIhvcNAQELBQADggEBAFjKLjgQovkZWsW61IMdxQOy1VHIJS79X2Iz FteAXf0s1kdkFY0YV+5TDisEn0P2mOIrjQ2tWfJbJEiKuiJ4rdRUNGwpbV3Il8gL YRxbxfXOSnBMQ7MYAxqv6ALsVAcG3aqKcq32cwSjXSWXvKERNlvDYFm5MgYrbibj IynQ7KYWDJqHDWC4WsMFDFQHR2gaYvVZ6YfnVAaOlzpzAQ1ZM48Spl8S3kYSxnRi W1i5HcU0IuPOTqJ7lCHDMTmJNSaCS3amwWJU0UCADfxkzNRKe+Eq70CQtzS8ZdvV pVQF7sdtMJFUfyzpw0B5agoKuREt0I70Vm7EhiTgLNVdQv5f8Ao= -----END CERTIFICATE----- subject=/C=PT/ST=Lisboa/L=Default City/O=Altice/OU=DCY-CSD/CN=logstash/emailAddress=pulso@telecom.pt issuer=/C=PT/ST=Lisboa/L=Default City/O=Altice/OU=DCY-CSD/emailAddress=pulso@telecom.pt

No client certificate CA names sent Server Temp Key: ECDH, secp521r1, 521 bits

SSL handshake has read 2500 bytes and written 2634 bytes

New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES128-SHA256 Server public key is 2048 bit Secure Renegotiation IS supported Compression: NONE Expansion: NONE SSL-Session: Protocol : TLSv1.2 Cipher : ECDHE-RSA-AES128-SHA256 Session-ID: 5A0C3B2DB5E145C12D67E8494675F1449F77073B1BB6DB7EBDCAC44322E21E8F Session-ID-ctx: Master-Key: 3C1DD3AABC322E3344B0D279B79B0A6579E83B9512005951303952A946F926BD7F1F25AAF13A3DCDB20DBD9D052512AE Key-Arg : None Krb5 Principal: None PSK identity: None PSK identity hint: None Start Time: 1510751021 Timeout : 300 (sec) Verify return code: 0 (ok)

However, after running for some time I am no longer able to establish a new connection as it hangs:

openssl s_client -connect localhost:9400 -CAfile ca.pem -cert cert.crt -key cert.key CONNECTED(00000003) Listing all connections to my logstash instance I didn't find any non-TLS, and they all seemed to be working and sending data, so I'm not sure the issue is the same.

@jmschmaus https://github.com/jmschmaus were you able to connect again after closing the non-TLS connection or was it no longer able to establish any other new TLS connection after that?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/logstash-plugins/logstash-input-tcp/issues/77#issuecomment-344587413, or mute the thread https://github.com/notifications/unsubscribe-auth/AX0t6XtoAw3SgPLdC2sVyxTi3hBTbz6Dks5s2uKGgaJpZM4OU-Es.

[jdlourenco]

Hi John,

That's exactly it! Thank you for the exhaustive description. It's really useful for finding the culprit connection

This has a huge impact in the reliability of our infrastructure since we are relying on a TCP input for ingesting all of our data.

Any prediction on when this will be fixed?

Best regards, José

[jmschmaus]

Hi José,

We are moving to filebeat, partially because it doesn’t suffer from this vulnerability.

I have no idea when this will be fixed. I opened the ticket a few months ago and your email is the first I’ve heard from it.

It has affected our logging pipeline significantly as well.

Thanks for the response!

John

On Nov 15, 2017, at 8:47 AM, José Lourenço notifications@github.com wrote:

Hi John,

That's exactly it! Thank you for the exhaustive description. It's really useful for finding the culprit connection

This has a huge impact in the reliability of our infrastructure since we are relying on a TCP input for ingesting all of our data.

Any prediction on when this will be fixed?

Best regards, José

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/logstash-plugins/logstash-input-tcp/issues/77#issuecomment-344634069, or mute the thread https://github.com/notifications/unsubscribe-auth/AX0t6U_9-Zjd1MPLfrhU4j-7gm0kw-31ks5s2weLgaJpZM4OU-Es.

[original-brownbear]

Looked over this a little today. There is no quick fix here it seems. This will have to wait until (and will be automatically fixed by moving the encrypted path to Netty like we did for the unencrypted path).

[jdlourenco]

@original-brownbear any prediction on when this will be fixed?

[original-brownbear]

@jdlourenco not for now no.

[ghost]

@original-brownbear any prediction on when this will be fixed?

[sunilchadha1973]

Hi @original-brownbear,

Is this issue resolved in Logstash 7.x version.