search

eclipse-openj9 / openj9

Eclipse OpenJ9: A Java Virtual Machine for OpenJDK that's optimized for small footprint, fast start-up, and high throughput. Builds on Eclipse OMR (https://github.com/eclipse/omr) and combines with the Extensions for OpenJDK for OpenJ9 repo.
Other
3.27k stars 721 forks source link

Is there a difference between openj9 and hotspot's interpretation of order in https.protocols #5629

Closed rubin55 closed 5 years ago

rubin55 commented 5 years ago

I noticed today, when running MuleSoft Anypoint Studio 7.3.2 with AdoptOpenJDK 8 /w OpenJ9 r212 that order matters in the system-wide https.protocols setting.

This is a potentially dangerous difference in behaviour compared to hotspot, which will select the highest protocol in the list regardless of ordering first.

I noticed this while adding an update site (in this case for markdown, here: https://nodeclipse.github.io/updates/markdown/) to Eclipse which is hosted by github.com. Github has [disabled insecure protocols some time ago]() and I got a protocol negotiation exception:

An error occurred while collecting items to be installed
session context was:(profile=DefaultProfile, phase=org.eclipse.equinox.internal.p2.engine.phases.Collect, operand=, action=).
Unable to read repository at https://nodeclipse.github.io/updates/markdown/features/markdown.editor.feature_1.2.0.201501260515.jar.
Received fatal alert: protocol_version

My https.protocols option was:

-Dhttps.protocols=TLSv1,TLSv1.1,TLSv1.2

The protocol negotiation seems to fail because TLSv1 was selected before TLSv1.2; Github gave an error response and I got an exception. When I change the ordering to:

-Dhttps.protocols=TLSv1.2,TLSv1.1,TLSv1

I don't get any exception, leading me to conclude the order is important, at least when running OpenJ9.

Anypoint Studio is based on Eclipse 4.7.2 btw.

rubin55 commented 5 years ago

By the way, before I forget, at least in one place IBM seems to be communicating an arbitrary and/or weak-to-strong list of protocols in its documentation.

rubin55 commented 5 years ago

I just tested to confirm, When I run with the hotspot variant of AdoptOpenJDK 8, it selects the highest protocol in the list instead of the first.

DanHeidinga commented 5 years ago

@theresa-m Can you take a look at this?

rubin55 commented 5 years ago

I've created a quick tryout reproducer (using javax.net.ssl.HttpsURLConnection). I cannot reproduce, see here: https://github.com/rubin55/https-protocol-order-reproducer

Both select TLSv1.2:

mvn package
java -Dhttps.protocols=TLSv1.2,TLSv1.1,TLSv1 -jar target/test-1.0.0-SNAPSHOT.jar
java -Dhttps.protocols=TLSv1,TLSv1.1,TLSv1.2 -jar target/test-1.0.0-SNAPSHOT.jar

Proof that https.protocols affect this code, won't work, throws exception:

java -Dhttps.protocols=TLSv1 -jar target/test-1.0.0-SNAPSHOT.jar

So OpenJ9 seems to do the right thing, but in combination with at least this version of Eclipse (4.7.2) the observed behaviour -Dhttps.protocols=TLSv1,TLSv1.1,TLSv1.2 with is:

rubin55 commented 5 years ago

I collected the backtrace from Eclipse:

!ENTRY org.eclipse.equinox.p2.engine 4 4 2019-04-29 15:39:00.753
!MESSAGE An error occurred while collecting items to be installed
!SUBENTRY 1 org.eclipse.equinox.p2.engine 4 0 2019-04-29 15:39:00.753
!MESSAGE session context was:(profile=DefaultProfile, phase=org.eclipse.equinox.internal.p2.engine.phases.Collect,      operand=, action=).
!SUBENTRY 1 org.eclipse.equinox.p2.transport.ecf 4 1002 2019-04-29 15:39:00.753
!MESSAGE Unable to read repository at https://nodeclipse.github.io/updates/markdown/features/markdown.editor.feature_1. 2.0.201501260515.jar.
!STACK 0
javax.net.ssl.SSLException: Received fatal alert: protocol_version
    at sun.security.ssl.Alerts.getSSLException(Alerts.java:208)
    at sun.security.ssl.Alerts.getSSLException(Alerts.java:154)
    at sun.security.ssl.SSLSocketImpl.recvAlert(SSLSocketImpl.java:2020)
    at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1127)
    at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1367)
    at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:750)
    at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:123)
    at org.apache.http.impl.io.AbstractSessionOutputBuffer.flushBuffer(AbstractSessionOutputBuffer.java:157)
    at org.apache.http.impl.io.AbstractSessionOutputBuffer.flush(AbstractSessionOutputBuffer.java:164)
    at org.apache.http.impl.AbstractHttpClientConnection.doFlush(AbstractHttpClientConnection.java:270)
    at org.apache.http.impl.AbstractHttpClientConnection.flush(AbstractHttpClientConnection.java:275)
    at org.apache.http.impl.conn.AbstractClientConnAdapter.flush(AbstractClientConnAdapter.java:207)
    at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:241)
    at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123)
    at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:686)
    at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:488)
    at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:884)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
    at org.eclipse.ecf.provider.filetransfer.httpclient4.HttpClientRetrieveFileTransfer.                                performConnect(HttpClientRetrieveFileTransfer.java:1084)
    at org.eclipse.ecf.provider.filetransfer.httpclient4.HttpClientRetrieveFileTransfer.                                access$0(HttpClientRetrieveFileTransfer.java:1075)
    at org.eclipse.ecf.provider.filetransfer.httpclient4.HttpClientRetrieveFileTransfer$1.                              performFileTransfer(HttpClientRetrieveFileTransfer.java:1071)
    at org.eclipse.ecf.filetransfer.FileTransferJob.run(FileTransferJob.java:74)
    at org.eclipse.core.internal.jobs.Worker.run(Worker.java:56)
rubin55 commented 5 years ago

I've collected the ssl debug information with -Djavax.net.debug=ssl:

Allow unsafe renegotiation: false
Allow legacy hello messages: true
Is initial handshake: true
Is secure renegotiation: false
Worker-9, setSoTimeout(120000) called
%% No cached client session
update handshake state: client_hello[1]
upcoming handshake states: server_hello[2]
*** ClientHello, TLSv1
RandomCookie:  GMT: 1539770977 bytes = { 140, 33, 127, 181, 171, 179, 137, 144, 178, 67, 48, 221, 38, 51, 135, 13, 153, 9, 160, 84, 158, 150, 197, 188, 116, 61, 18, 236 }
Session ID:  {}
Cipher Suites: [TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_CBC_SHA, TLS_EMPTY_RENEGOTIATION_INFO_SCSV]
Compression Methods:  { 0 }
Extension elliptic_curves, curve names: {secp256r1, secp384r1, secp521r1, sect283k1, sect283r1, sect409k1, sect409r1, sect571k1, sect571r1, secp256k1}
Extension ec_point_formats, formats: [uncompressed]
Extension extended_master_secret
Extension server_name, server_name: [type=host_name (0), value=nodeclipse.github.io]
***
Worker-9, WRITE: TLSv1 Handshake, length = 140
Worker-9, READ: TLSv1 Alert, length = 2
Worker-9, RECV TLSv1.2 ALERT:  fatal, protocol_version
Worker-9, called closeSocket()
Worker-9, handling exception: javax.net.ssl.SSLException: Received fatal alert: protocol_version
Worker-9, called close()
Worker-9, called closeInternal(true)
Allow unsafe renegotiation: false
Allow legacy hello messages: true
Is initial handshake: true
Is secure renegotiation: false
Worker-8, setSoTimeout(120000) called
%% No cached client session
update handshake state: client_hello[1]
upcoming handshake states: server_hello[2]
*** ClientHello, TLSv1
RandomCookie:  GMT: 1539770977 bytes = { 50, 31, 209, 247, 72, 83, 151, 246, 202, 28, 9, 227, 195, 244, 39, 142, 139, 175, 41, 65, 49, 217, 46, 60, 24, 146, 14, 202 }
Session ID:  {}
Cipher Suites: [TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_CBC_SHA, TLS_EMPTY_RENEGOTIATION_INFO_SCSV]
Compression Methods:  { 0 }
Extension elliptic_curves, curve names: {secp256r1, secp384r1, secp521r1, sect283k1, sect283r1, sect409k1, sect409r1, sect571k1, sect571r1, secp256k1}
Extension ec_point_formats, formats: [uncompressed]
Extension extended_master_secret
Extension server_name, server_name: [type=host_name (0), value=nodeclipse.github.io]
***
Worker-8, WRITE: TLSv1 Handshake, length = 140
Worker-8, READ: TLSv1 Alert, length = 2
Worker-8, RECV TLSv1.2 ALERT:  fatal, protocol_version
Worker-8, called closeSocket()
Worker-8, handling exception: javax.net.ssl.SSLException: Received fatal alert: protocol_version
Worker-8, called close()
Worker-8, called closeInternal(true)
Finalizer thread, called close()
Finalizer thread, called closeInternal(true)

This seems to be Eclipse specific, although I'm not sure yet what's going on. I've updated my reproducer to also use Apache HTTP Components' HTTP Client 4.5.2; It seems HTTP Client 4.5.2 does not use https.protocols setting at all (setting it to an invalid value or low protocol version does not prohibit the client from actually negotiating a TLSv1.2 connection)

rubin55 commented 5 years ago

Update from my side: I think i've isolated this issue to Eclipse ECF's httpclient4 filetransfer component. This component is the default HTTP client for for p2/Eclipse update. The Eclipse Wiki documents that you can turn off httpclient4 with the following option:

-Dorg.eclipse.ecf.provider.filetransfer.excludeContributors=org.eclipse.ecf.provider.filetransfer.httpclient4

When I run Eclipse 4.7.2 with that option in eclipse.ini under -vmargs, I see that regardless of the ordering of https.protocols, TLSv1.2 is selected, which leads me to assume that the issue originates within Eclipse ECF's httpclient4 component. I haven't looked in there yet, but I'm assuming some crazy iterator or some such..

rubin55 commented 5 years ago

Just a quick glance at Eclipse ECF's httpclient4 filetransfer component, I think I'm seeing a typo on line 56 (sslContext.init(..) instead of rtvContext.init(..)):

https://github.com/eclipse/ecf/blob/e2f3919166add4a8184b21ac9f358c9f4cf45756/providers/bundles/org.eclipse.ecf.provider.filetransfer.httpclient4/src/org/eclipse/ecf/provider/filetransfer/httpclient4/HttpClientDefaultSSLSocketFactoryModifier.java#L56

I'm not sure why this then would only occur on OpenJ9, but that might have to do with ordering/preference for the implemented interface (ISSLSocketFactoryModifier?) which then wouldn't get loaded on Hotspot for whatever reason mayhaps?

theresa-m commented 5 years ago

thanks for all your investigation work @rubin55 ! I'm attempting to reproduce the issue with AnypointStudio and find the cause

update: I tried downloading Eclipse 4.7.2 itself and was able to install the markdown package (using OpenJ9) without issue. Once I downloaded AnypointStudio the issue manifested

rubin55 commented 5 years ago

Update: ah, I need to read comments before posting :-) Actually surprised this is specific to Anypoint, wow :-) Hi @theresa-m :-) Cool, Note that you could most probably also reproduce the issue with a bog-standard Eclipse 4.7.2, maybe a bit easier (seeing as Anypoint Studio is a proprietary beast that requires access to a private download area afaik).

theresa-m commented 5 years ago

Okay the reason I was not able to reproduce in Eclipse before is because Anypoint has -Dhttps.protocols=TLSv1,TLSv1.1,TLSv1.2 in its .ini file but Eclipse does not. After this was added the failure was occuring in both Eclipse and Anypoint as expected.

However when I swap between using hotspot and openj9 I'm actually getting the same protocol_version failure for both vms. Can you confirm if thats the case for you as well? If thats the case the problem is likely an issue on the Eclipse side as you pointed out.

rubin55 commented 5 years ago

Hi @theresa-m,

Just tested with OpenJDK 1.8.0 r212 from AdoptOpenJDK, for macOS x64. You are right, the error happens in hotspot also, but at another moment: In OpenJ9, it happens as soon as I want to enumerate the updates site; in hotspot, it lets me enumerate the site, select the to-be-installed package and press finish, then finally I do get an error when it would actually start downloading.

This difference is what originally led me to conclude it was the OpenJ9/Hotspot difference that was causing the issue. I'm 100% certain this error is caused by the code in Eclipse ECF's httpclient4. I will create bugreports with them.

https://bugs.eclipse.org/bugs/show_bug.cgi?id=546894 https://bugs.eclipse.org/bugs/show_bug.cgi?id=546896

theresa-m commented 5 years ago

makes sense! Initially I was seeing that behavior as well. Now I'm seeing that the more times I restart eclipse the behavior is inconsistent. I've been seeing hotspot fail right away a few times and openj9 fail after markdown was selected as well as the other way around. Based on that I think we can safely hand over the issue to the eclipse side. thanks for all of your sleuthing :)

pshipton commented 5 years ago

Seems like the problem may be due to iteration order that changes due to hashcode generation.