neo4j-3.2.8 https with bolt is not working

[haridas]

Description

Running neo4j-3.2.8 with trusted CA signed certificates, the certificates are valid on browser and the pad-lock come properly when accessing the the neo4j browser via https. The neo4j browser works fine with bolt connection when access it via http.

The error comes on browser is:-

ServiceUnavailable: WebSocket connection failure. Due to security constraints in your web browser, the reason for the failure is not available to this Neo4j Driver. Please use your browsers development console to determine the root cause of the failure. Common reasons include the database being unavaila...

Your Environment

software name	version
OS	mac sierra
Web browser	62.0.3202.94
node.js	NA
npm	NA
yarn	NA
Neo4j Browser	NA
Neo4j	3.2.8

NOTE: I downloaded the tar.gz neo4j community edition for server, running on Redhat-7.4 64 bit server.

Additional Information

Strange thing is the same setup work on the Safari browser on "https" mode itself. Another intrigued me is, the bellow code snippet on browser client,

                        var e = "ws";
                        if (n.encrypted === !0 || n.encrypted === f.ENCRYPTION_ON) {
                            if (n.trust && "TRUST_CUSTOM_CA_SIGNED_CERTIFICATES" !== n.trust)
                                return void (this._error = (0,
                                a.newError)("The browser version of this driver only supports one trust strategy, 'TRUST_CUSTOM_CA_SIGNED_CERTIFICATES'. " + n.trust + ' is not supported. Please either use TRUST_CUSTOM_CA_SIGNED_CERTIFICATES or disable encryption by setting `encrypted:"' + f.ENCRYPTION_OFF + '"` in the driver configuration.'));
                            e = "wss"
                        }

Seems like, client only use wss when n.trust is set with TRUST_CUSTOM_CA_SIGNED_CERTIFICATES and the n.encrypted is enabled.

Please help me to fix this issue, I might be missing something simple.

[oskarhane]

Thanks for the report! I think we need some more info on this.

What are your server settings in neo4j.conf regarding the bolt tlsLevel? It works in Safari but not in Chrome (just to clarify)? That hints that Chrome is having some issues with the trusted certificate. Can you try other web browsers a well?

[haridas]

Yes, it works in Safari and firefox quantum. But not on chrome-62.0.3202.94.

Also I got the connection issue on neo4j-3.3.0 version initially, then was downgraded to 3.2.8 and trying.

conf/neo4j.conf

$ cat conf/neo4j.conf | grep \= | grep -v \#
dbms.directories.import=import
dbms.memory.heap.initial_size=2g
dbms.memory.heap.max_size=4g
dbms.connectors.default_listen_address=0.0.0.0
dbms.connectors.default_advertised_address=**********.com
dbms.connector.bolt.enabled=true
dbms.connector.bolt.tls_level=REQUIRED
dbms.connector.bolt.listen_address=0.0.0.0:7687
dbms.connector.http.enabled=true
dbms.connector.https.enabled=true
dbms.connector.https.listen_address=:7473
bolt.ssl_policy=default
https.ssl_policy=default
dbms.ssl.policy.default.base_directory=certificates/default
dbms.ssl.policy.default.trust_all=false
dbms.ssl.policy.default.client_auth=optional
dbms.logs.http.enabled=true
dbms.tx_log.rotation.retention_policy=1 days
dbms.shell.enabled=true
dbms.shell.port=1337
dbms.jvm.additional=-XX:+UseG1GC
dbms.jvm.additional=-XX:-OmitStackTraceInFastThrow
dbms.jvm.additional=-XX:+AlwaysPreTouch
dbms.jvm.additional=-XX:+UnlockExperimentalVMOptions
dbms.jvm.additional=-XX:+TrustFinalNonStaticFields
dbms.jvm.additional=-XX:+DisableExplicitGC
dbms.jvm.additional=-Djdk.tls.ephemeralDHKeySize=2048
dbms.windows_service_name=neo4j
dbms.jvm.additional=-Dunsupported.dbms.udc.source=tarball
browser.remote_content_hostname_whitelist=*

[oskarhane]

Thanks for the additional info.

Do the https url and bolt url have different hostnames / ip:s? I know some web browser can be picky about opening secure websockets from a different hostname.

[haridas]

Hostname used to access the "https" and "bolt" are same. Usually UI will show the default hostname from this property for bolt connection dbms.connectors.default_advertised_address.

Port is different thought, https using 7473 and bolt using 6787. Not sure the port change have any problem with security.

[oskarhane]

Alright, that's too bad. The ports should be different, so that's fine.

I'll do some research and try to reproduce this, but honestly I'm out of clues. I'm not sure if older versions of Chrome are available for download, but one idea to would be to try it out in different versions of Chrome.

[haridas]

Tested on chrome 58 and other different versions, the problem seems persisting.

We are using wildcard certificate for our domain(*.domain.com), will that have any thing todo with this ?

The bellow command resolving correctly and the Verify return code: 0 (ok).

openssl s_client -connect subdom.domain.com:7473

From wireshark it's clear that the handshake with backend is happening ( captured the packet when clicking on browser ), but browser closing the connection once it receives the certificate details from server.

I'm testing the scenario to confirm certificate causing this or not.

[haridas]

Now it worked,

changes done on the conf/neo4j.conf file.

bolt.ssl_policy=legacy
https.ssl_policy=legacy

I put our wild card certificate and key in legacy location ./certificates/neo4j.cert and ./certificates/neo4j.key.

The default ssl policy that I used has the bellow directory structure,

[ec2-user@neo4jneo4j]$ ls -al certificates/default/ -R
certificates/default/:
drwxr-xr-x. 4 ec2-user ec2-user  108 Nov 27 09:08 .
drwxr-xr-x. 3 ec2-user ec2-user   56 Nov 27 09:09 ..
-rw-------. 1 ec2-user ec2-user  981 Nov 22 11:03 neo4j.cert
-rw-------. 1 ec2-user ec2-user 1704 Nov 22 11:03 neo4j.key
-rwxr-xr-x. 1 ec2-user ec2-user 1679 Nov 22 11:03 private.key
-rwxr-xr-x. 1 ec2-user ec2-user 7531 Nov 22 11:03 public.crt
drwxr-xr-x. 2 ec2-user ec2-user    6 Nov 22 11:03 revoked
drwxr-xr-x. 2 ec2-user ec2-user   24 Nov 23 10:44 trusted

certificates/default/revoked:
total 0
drwxr-xr-x. 2 ec2-user ec2-user   6 Nov 22 11:03 .
drwxr-xr-x. 4 ec2-user ec2-user 108 Nov 27 09:08 ..

certificates/default/trusted:
total 8
drwxr-xr-x. 2 ec2-user ec2-user   24 Nov 23 10:44 .
drwxr-xr-x. 4 ec2-user ec2-user  108 Nov 27 09:08 ..
-rwxr-xr-x. 1 ec2-user ec2-user 7531 Nov 23 10:44 public.crt

Is there any issue with this directory structure ? Interesting thing is, this directory structure works with latest firefox quantum browser.

IMHO it's bit confusing the concept of trusted and revoked keys in terms of browser point of view. It works well for those backend client which uses client certificate also. May be I'm missing design decisions that lead to this policy based ssl handling.

[oskarhane]

Interesting, thanks for looking into this @haridas! This is out of my knowledge, but maybe @lutovich or @technige can chip in with more info and suggestions in the structure and bring light to the trusted and revoked keys.

[jboler]

I also have this issue on Ubuntu 16.04 with Chrome 66. Firefox Quantum works. This is with dbms.ssl.policy.default.client_auth=OPTIONAL.

If I use dbms.ssl.policy.default.client_auth=NONE it works in Chrome.

If I set dbms.ssl.policy.default.client_auth=REQUIRE the neo4j browser page won't load at all in any browser and with curl I get curl: (35) gnutls_handshake() failed: The TLS connection was non-properly terminated.

[oskarhane]

@jboler Are you loading neo4j-browser over the https protocol when trying that? https://localhost:7473

Neo4j Browser connects with unencrypted bolt if it's loaded unencrypted in your web browser. And vice versa.

[jboler]

Yes, the initial page (https://server.mydomain.com:7473) loads fine then when I enter the correct user/pass and try to log in I get the security constraints error. That's with client_auth=OPTIONAL. It works just fine in Firefox or with clienth_auth=NONE in Chrome.

[mikeblum]

I've also encountered this in 3.4 using Let's Encrypt. The client_auth flag fixes the error when using the Python driver as well. Otherwise I get a BAD_CERTIFICATE_ERROR in Firefox and using the python neo4j-driver. dbms.ssl.policy.default.client_auth=OPTIONAL and NONE worked for me but REQUIRE causes the bad cert error.

[oskarhane]

This sounds like the issue lies somewhere in neo4j-driver land rather than neo4j-browser. Did you run into this or heard about it @lutovich @technige?