search

BroadSoft-Xtended / BroadWorks-Dashboards-and-Discovery

This repository contains the BroadWorks Dashboards and Discovery components to extend BroadWorks data mining, reporting, and analysis capabilities.
23 stars 4 forks source link

Strange SSL issue #69

Closed kdotson3263k closed 5 years ago

kdotson3263k commented 5 years ago

I have the same setup on two different bwlogrecievers, "senderreceiverusessl:true" on both receiver and sender. The sender can send to one of the receiver servers, but I am getting an error to the other. Am I missing an application?

I have tried to reinstall the receiver on the server that is getting the error with no luck.

Thanks for any assistance!

2019-03-06_16:26:20.132 [NetworkSocketListener] INFO c.b.eslogzipreceiver.NetworkSocket - Connection established from: /X.X.X.X:19791 2019-03-06_16:26:20.133 [Incoming connection from: ] INFO c.b.eslogzipreceiver.NetworkSocket - run method that is doing nothing 2019-03-06_16:26:20.133 [LogReceiver: ] INFO c.b.eslogzipreceiver.LogReceiver - LogReceiver running. 2019-03-06_16:26:38.050 [LogReceiver: ] INFO c.b.eslogzipreceiver.NetworkSocket - Got general Exception from BufferedReader().readLine() javax.net.ssl.SSLHandshakeException: no cipher suites in common at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1946) at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:316) at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:306) at sun.security.ssl.ServerHandshaker.chooseCipherSuite(ServerHandshaker.java:1127) at sun.security.ssl.ServerHandshaker.clientHello(ServerHandshaker.java:814) at sun.security.ssl.ServerHandshaker.processMessage(ServerHandshaker.java:221) at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1037) at sun.security.ssl.Handshaker.process_record(Handshaker.java:965) at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1064) at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1367) at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:931) at sun.security.ssl.AppInputStream.read(AppInputStream.java:105) at sun.security.ssl.AppInputStream.read(AppInputStream.java:71) at com.broadsoft.eslogzipshared.CompressedBlockInputStream.readAndDecompress(CompressedBlockInputStream.java:53) at com.broadsoft.eslogzipshared.CompressedBlockInputStream.read(CompressedBlockInputStream.java:142) at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284) at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326) at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178) at java.io.InputStreamReader.read(InputStreamReader.java:184) at java.io.BufferedReader.fill(BufferedReader.java:161) at java.io.BufferedReader.readLine(BufferedReader.java:324) at java.io.BufferedReader.readLine(BufferedReader.java:389) at com.broadsoft.eslogzipreceiver.NetworkSocket.Read(NetworkSocket.java:191) at com.broadsoft.eslogzipreceiver.LogReceiver$ElasticSearchHandler.run(LogReceiver.java:133) at java.lang.Thread.run(Thread.java:748) 2019-03-06_16:26:38.051 [LogReceiver: ] INFO c.b.eslogzipreceiver.LogReceiver - LogReceiver:Null read - cleaning up

kdotson3263k commented 5 years ago

Could this be caused by java versions not matching on the sender and receiver? If so, is it possible to point the sender to a different java version?

bchibbard commented 5 years ago

I don't know for sure, but it does sound to me like it difference of JDK could account for it. Out of curiosity, what is the JDK version used by the receiver that is giving you trouble?

In the configuration properties file for the receiver, you can specify which Java you want to use with JAVA_PATH. In the sender, I believe the Perl script that wraps the executable is hardcoded to use the Java that it expects BroadWorks to be using.

kdotson3263k commented 5 years ago

I think it may be a sender issue. I am going to try a different sender with high Java version and will let you know.

In this case receiver: openjdk version "1.8.0_191"

kdotson3263k commented 5 years ago

I am able to redirect the sender that is failing to a different receiver on the same java version and it works.

I have a tcpdump, from the failed receiver server and I see that the sender sends a "Client Hello" but the server does not respond with "Server Hello"

bchibbard commented 5 years ago

I'm no authority on TLS, but I believe "no cipher suites in common" should cause the server to refuse to send a ServerHello, and instead it will return to the server app a "handshake failure alert" and close the connection (https://tools.ietf.org/html/rfc5246#section-7.4.1.3). I don't know why both ends cannot agree on a ciphersuite, but I'm not shocked that different JDK platforms would have different answers to "what ciphersuites are available/supported". Not sure if it's related, but I've seen some search results mention that OpenJDK disables TLS 1.2 by default. If that's true, that sounds like it could be the problem.

kdotson3263k commented 5 years ago

Same OS and Java versions on both the good and bad receiver. The only change on the logsender side is the update to the IP in the props and restart.

kdotson3263k commented 5 years ago

Finally getting back to this....the sender is on java version "1.7.0_21" Java(TM) SE Runtime Environment (build 1.7.0_21-b11)

2019-03-26_23:42:00.493 [Sender Thread #0] INFO c.broadsoft.zipsender.NetworkSocket - Attempting to create socket to Receiver 2019-03-26_23:42:01.136 [Sender Thread #0] INFO c.broadsoft.zipsender.NetworkSocket - Connection seems successfull 2019-03-26_23:42:01.136 [Sender Thread #0] INFO c.broadsoft.zipsender.NetworkSocket - Connection (re)established.... Moving on... 2019-03-26_23:42:01.242 [Sender Thread #0] INFO c.broadsoft.zipsender.NetworkSocket - Generic Exception writing to Stream. Most likely network problem javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) at sun.security.ssl.Alerts.getSSLException(Alerts.java:154) at sun.security.ssl.SSLSocketImpl.recvAlert(SSLSocketImpl.java:1961) at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1077) at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1312) at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:702) at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:122) at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:136) at com.broadsoft.eslogzipshared.CompressedBlockOutputStream.compressAndSend(CompressedBlockOutputStream.java:78) at com.broadsoft.eslogzipshared.CompressedBlockOutputStream.write(CompressedBlockOutputStream.java:118) at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221) at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:282) at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125) at java.io.OutputStreamWriter.write(OutputStreamWriter.java:207) at java.io.BufferedWriter.flushBuffer(BufferedWriter.java:129) at java.io.BufferedWriter.write(BufferedWriter.java:230) at java.io.Writer.write(Writer.java:157) at com.broadsoft.zipsender.NetworkSocket.Write(NetworkSocket.java:338) at com.broadsoft.zipsender.NetworkSocket.run(NetworkSocket.java:185) at java.lang.Thread.run(Thread.java:722) 2019-03-26_23:42:01.243 [Sender Thread #0] INFO c.broadsoft.zipsender.NetworkSocket - failed to write log 2019-03-26_23:42:01.243 [Sender Thread #0] INFO c.broadsoft.zipsender.NetworkSocket - Connection lost.... Resetting...

kdotson3263k commented 5 years ago

I attempted to set java version the same on the receiver side and still getting the same errors. I don't know enough about java to take this much further.

ps -ef | grep bwlogr root 2729 1 0 14:58 pts/0 00:00:04 /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.211.x86_64/jre/bin//java -Dp=bwlogreceiver -Dcom.sun.management.jmxremote.port=4086 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -XX:CMSInitiatingOccupancyFraction=68 -XX:ParallelGCThreads=2 -Xms1024m -Xmx1024m -Dchannel.props=/bw/bwlogreceiver/bwlogreceiver_1021/logreceiver.props -Dapplogger.path=/bw/bwlogreceiver/bwlogreceiver_1021/log.txt -jar /bw/bwlogreceiver/bwlogreceiver_1021/eslogreceiver.jar

kdotson3263k commented 5 years ago

I performed a sslscan and found that the bwreceiver servers do not have any available ciphers while the working servers have the following.

[root@wdv-log09 bwlogreceiver]# sslscan XX.XX.XX.XX:9072 | grep Accepted Accepted TLSv1 256 bits AECDH-AES256-SHA Accepted TLSv1 256 bits ADH-AES256-SHA Accepted TLSv1 128 bits AECDH-AES128-SHA Accepted TLSv1 128 bits ADH-AES128-SHA Accepted TLSv1 0 bits AECDH-NULL-SHA Accepted TLS11 256 bits AECDH-AES256-SHA Accepted TLS11 256 bits ADH-AES256-SHA Accepted TLS11 128 bits AECDH-AES128-SHA Accepted TLS11 128 bits ADH-AES128-SHA Accepted TLS11 0 bits AECDH-NULL-SHA Accepted TLS12 256 bits AECDH-AES256-SHA Accepted TLS12 256 bits ADH-AES256-GCM-SHA384 Accepted TLS12 256 bits ADH-AES256-SHA256 Accepted TLS12 256 bits ADH-AES256-SHA Accepted TLS12 128 bits AECDH-AES128-SHA Accepted TLS12 128 bits ADH-AES128-GCM-SHA256 Accepted TLS12 128 bits ADH-AES128-SHA256 Accepted TLS12 128 bits ADH-AES128-SHA Accepted TLS12 0 bits AECDH-NULL-SHA

Any help on where I can go next will be very much appreciated.

kdotson3263k commented 5 years ago

Note: On the server that is not working, all of the "Accepted" ciphers from the working server (previous post) are being rejected with "sslv3 alert handshake failure".

This is from directly on the failing receiver server:

[root@wdv-log21 tmp]# openssl s_client -cipher AECDH-AES256-SHA -connect XX.XX.XX.XX:9072 CONNECTED(00000003) 140306023761736:error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure:s23_clnt.c:744:

no peer certificate available No client certificate CA names sent SSL handshake has read 7 bytes and written 119 bytes New, (NONE), Cipher is (NONE) Secure Renegotiation IS NOT supported Compression: NONE Expansion: NONE

output from a good receiver:

[root@wdv-log09 tmp]# openssl s_client -cipher AECDH-AES256-SHA -connect XX.XX.XX.XX:9072 CONNECTED(00000003) no peer certificate available No client certificate CA names sent Server Temp Key: ECDH, secp521r1, 521 bits SSL handshake has read 306 bytes and written 337 bytes New, TLSv1/SSLv3, Cipher is AECDH-AES256-SHA Secure Renegotiation IS supported Compression: NONE Expansion: NONE SSL-Session: Protocol : TLSv1.2 Cipher : AECDH-AES256-SHA Session-ID: 5CA2596778A7528E341A98683BF3811562468702B84E9B5C4BA2D154059F90B3 Session-ID-ctx: Master-Key: A2FB81D17D3906CA1C07533F53B7CF7195EC9CDB7A6F20632B17000C46E77ECEC10DDCA0DB6D74C48BF903011F30E969 Key-Arg : None Krb5 Principal: None PSK identity: None PSK identity hint: None Start Time: 1554143591 Timeout : 300 (sec) Verify return code: 0 (ok)

bchibbard commented 5 years ago

Are the contents of .../jre/lib/security/java.security, specifically the "security.provider.<x>" section identical in both of the receiver servers? It sounds like the ciphersuites are available on one of the servers but not both.

kdotson3263k commented 5 years ago

Thank you!!! This was the answer.

It made me look a little closer. The good host is on 1.8.0_191 the troubled host is on 1.8.0_201.

1.8.0_191 jdk.tls.disabledAlgorithms=SSLv3, RC4, DES, MD5withRSA, DH keySize < 1024, \ EC keySize < 224, 3DES_EDE_CBC

1.8.0_201 jdk.tls.disabledAlgorithms=SSLv3, RC4, DES, MD5withRSA, DH keySize < 1024, \ EC keySize < 224, 3DES_EDE_CBC, anon, NULL

I removed ", anon, NULL" from 1.8.0_201 and restarted, fixed.

bchibbard commented 5 years ago

You're welcome!