search

trinodb / trino

Official repository of Trino, the distributed SQL query engine for big data, formerly known as PrestoSQL (https://trino.io)
https://trino.io
Apache License 2.0
10.41k stars 3k forks source link

Problem with securing internal communication #8883

Closed hungnt61h closed 3 years ago

hungnt61h commented 3 years ago

Hi, I'm just beginner in setting up Trino cluster! I deployed Trino (version 360) cluster on vmware machines and got stuck at setting up internal TLS communication. Enabling TLS connection between client and coordinator works fine with me. I am able to log on Trino with my LDAP account and all workers are discovered. (Sensitive fields are ** or xx)
My first configuration here (no internal TLS communication) Coordinator**

coordinator=true
node-scheduler.include-coordinator=false

http-server.http.port=8442
query.max-memory=50GB
query.max-memory-per-node=1GB
query.max-total-memory-per-node=2GB
discovery.uri=http://node2.xx.xx:8442

http-server.authentication.type=PASSWORD
http-server.https.enabled=true
http-server.https.port=8443
http-server.https.keystore.path=/path/to/keystore.jks
http-server.https.keystore.key=****

log.path=/var/log/trino/server.log
http-server.log.path=/var/log/trino/http-request.log

Worker

coordinator=false

http-server.http.port=8442
query.max-memory=50GB
query.max-memory-per-node=1GB
query.max-total-memory-per-node=2GB
discovery.uri=http://node2.xx.xx:8442

log.path=/var/log/trino/server.log
http-server.log.path=/var/log/trino/http-request.log

With these configuration, Trino cluster works fine and smoothly. But when I configured internal TLS communication, the problem occured like this:

2021-08-14T17:42:33.887+0700 ERROR Announcer-0 io.airlift.discovery.client.Announcer Service announcement failed after 66.07ms. Next request will happen within 0.00s 2021-08-14T17:42:33.911+0700 ERROR Announcer-0 io.airlift.discovery.client.Announcer Service announcement failed after 17.11ms. Next request will happen within 1.00ms 2021-08-14T17:42:33.927+0700 ERROR Announcer-4 io.airlift.discovery.client.Announcer Service announcement failed after 13.21ms. Next request will happen within 2.00ms 2021-08-14T17:42:33.945+0700 ERROR Announcer-2 io.airlift.discovery.client.Announcer Service announcement failed after 13.28ms. Next request will happen within 4.00ms 2021-08-14T17:42:33.970+0700 ERROR Announcer-0 io.airlift.discovery.client.Announcer Service announcement failed after 16.54ms. Next request will happen within 8.00ms 2021-08-14T17:42:34.005+0700 ERROR Announcer-4 io.airlift.discovery.client.Announcer Service announcement failed after 17.71ms. Next request will happen within 16.00ms 2021-08-14T17:42:34.054+0700 ERROR Announcer-2 io.airlift.discovery.client.Announcer Service announcement failed after 16.39ms. Next request will happen within 32.00ms 2021-08-14T17:42:34.132+0700 ERROR Announcer-0 io.airlift.discovery.client.Announcer Service announcement failed after 14.19ms. Next request will happen within 64.00ms 2021-08-14T17:42:34.275+0700 ERROR Announcer-4 io.airlift.discovery.client.Announcer Service announcement failed after 14.14ms. Next request will happen within 128.00ms 2021-08-14T17:42:34.542+0700 ERROR Announcer-2 io.airlift.discovery.client.Announcer Service announcement failed after 9.37ms. Next request will happen within 256.00ms

 - In worker:
```text
2021-08-14T17:46:24.410+0700    INFO    main    io.trino.security.AccessControlManager  Using system access control default
2021-08-14T17:46:24.446+0700    INFO    main    io.trino.server.Server  ======== SERVER STARTED ========
2021-08-14T17:46:24.446+0700    ERROR   Announcer-0     io.airlift.discovery.client.Announcer   Cannot connect to discovery server for announce
io.airlift.discovery.client.DiscoveryException: Announcement failed for https://node2.vmware.com:8443
        at io.airlift.discovery.client.HttpDiscoveryAnnouncementClient$DiscoveryResponseHandler.handleException(HttpDiscoveryAnnouncementClient.java:195)
        at io.airlift.http.client.jetty.JettyResponseFuture.failed(JettyResponseFuture.java:120)
        at io.airlift.http.client.jetty.BufferingResponseListener.onComplete(BufferingResponseListener.java:85)
        at org.eclipse.jetty.client.ResponseNotifier.notifyComplete(ResponseNotifier.java:218)
        at org.eclipse.jetty.client.ResponseNotifier.notifyComplete(ResponseNotifier.java:210)
        at org.eclipse.jetty.client.HttpSender.terminateRequest(HttpSender.java:411)
        at org.eclipse.jetty.client.HttpSender.abort(HttpSender.java:574)
        at org.eclipse.jetty.client.HttpSender.lambda$executeAbort$0(HttpSender.java:361)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:882)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1036)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:349)
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:292)
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:287)
        at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1357)
        at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1232)
        at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1175)
        at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392)
        at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:443)
        at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1074)
        at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1061)
        at java.base/java.security.AccessController.doPrivileged(Native Method)
        at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1008)
        at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.fill(SslConnection.java:654)
        at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.process(HttpReceiverOverHTTP.java:164)
        at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.receive(HttpReceiverOverHTTP.java:79)
        at org.eclipse.jetty.client.http.HttpChannelOverHTTP.receive(HttpChannelOverHTTP.java:131)
        at org.eclipse.jetty.client.http.HttpConnectionOverHTTP.onFillable(HttpConnectionOverHTTP.java:172)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
        at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint$IncompleteWriteCallback.succeeded(SslConnection.java:1577)
        at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:293)
        at org.eclipse.jetty.io.AbstractEndPoint.write(AbstractEndPoint.java:381)
        at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.needsFillInterest(SslConnection.java:920)
        at org.eclipse.jetty.io.AbstractEndPoint$1.needsFillInterest(AbstractEndPoint.java:45)
        at org.eclipse.jetty.io.FillInterest.tryRegister(FillInterest.java:83)
        at org.eclipse.jetty.io.FillInterest.register(FillInterest.java:55)
        at org.eclipse.jetty.io.AbstractEndPoint.fillInterested(AbstractEndPoint.java:362)
        at org.eclipse.jetty.io.AbstractConnection.fillInterested(AbstractConnection.java:134)
        at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.fillInterested(HttpReceiverOverHTTP.java:243)
        at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.process(HttpReceiverOverHTTP.java:175)
        at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.receive(HttpReceiverOverHTTP.java:79)
        at org.eclipse.jetty.client.http.HttpChannelOverHTTP.receive(HttpChannelOverHTTP.java:131)
        at org.eclipse.jetty.client.http.HttpConnectionOverHTTP.onFillable(HttpConnectionOverHTTP.java:172)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
        at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:555)
        at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:410)
        at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:164)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
        at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:383)
        ... 3 more
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:439)
        at java.base/sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:306)
        at java.base/sun.security.validator.Validator.validate(Validator.java:264)
        at java.base/sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:313)
        at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:276)
        at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:141)
        at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1335)
        ... 44 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at java.base/sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
        at java.base/sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
        at java.base/java.security.cert.CertPathBuilder.build(CertPathBuilder.java:297)
        at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:434)
        ... 50 more

2021-08-14T17:46:24.447+0700    ERROR   Announcer-0     io.airlift.discovery.client.Announcer   Service announcement failed after 36.14ms. Next request will happen within 0.00s
2021-08-14T17:46:24.460+0700    ERROR   Announcer-1     io.airlift.discovery.client.Announcer   Service announcement failed after 10.67ms. Next request will happen within 1.00ms
2021-08-14T17:46:24.473+0700    ERROR   Announcer-2     io.airlift.discovery.client.Announcer   Service announcement failed after 11.24ms. Next request will happen within 2.00ms

with my configuration in
Coordinator

coordinator=true
node-scheduler.include-coordinator=false

http-server.http.port=8442
query.max-memory=50GB
query.max-memory-per-node=1GB
query.max-total-memory-per-node=2GB
discovery.uri=https://node2.xx.xx:8443

http-server.authentication.type=PASSWORD
http-server.https.enabled=true
http-server.https.port=8443
http-server.https.keystore.path=/path/to/keystore.jks
http-server.https.keystore.key=****

log.path=/var/log/trino/server.log
http-server.log.path=/var/log/trino/http-request.log

internal-communication.shared-secret=vwdcd6wdIx14KQdMIw/228WFjuKpWzie+v4xRDeMcGUmmzciccxbVHnfFeQFBfb5UyXg7OBDE9rgpaDLiMf6CcKgErQ+HFTAk611KHZ08Xvt9CFhIOTShawp7PlPFMXGZMRoHggihroa2+WzUHcla9bRiADnGbTQ6WR1UQm989OeCqZQDZtugPqZ1820uXrvTW/QDylG+vHwrYr48P+TcNUKON5ObHatjsA0t4N1kuQJ2NPkyjWqcBzVYbN1RHI9PuWZbNMkaxYYBru1Jl+Hg1mbMZRXaqFjfT76F2yT9bnwmC8UiTidDxkSsuN/OCaPheX0FXyIv91+h1eifQ/P//f7BA1cpJ2k5Oag9ywyAcrZphLB2briecJicioAtOXaGUhW5mxQlSc2+mbf+xaZc7lyI/Pe5S81seRGWUN/GOVidz31e7sSXIFgOIoU05zR+uWFbAGIhF7x2c2cS+BIfcEiXM/+bf161QqDuXvZ95zUJVnENi8VjMnmCNPGeg/fVis315t+MCC9joSOGf4f6FSxUP5yeQ2fGLbHdfBNcweGTDbSZXzOJBjVQ+yrK9zEuKU0DqZPTyuC3x9c881BP2RswLW+3eyPT2tRGHpPQgjv/6RdWHZJb1dPl+9xdyXPJgSN9JWKVJJ4Wsla6avZtlQq7K9512CjftODCo5Ee20=
internal-communication.https.required=true

Worker

coordinator=false

http-server.http.port=8442
query.max-memory=50GB
query.max-memory-per-node=1GB
query.max-total-memory-per-node=2GB
discovery.uri=https://node2.xx.xx:8443

log.path=/var/log/trino/server.log
http-server.log.path=/var/log/trino/http-request.log

http-server.https.enabled=true
http-server.https.port=8443

internal-communication.shared-secret=vwdcd6wdIx14KQdMIw/228WFjuKpWzie+v4xRDeMcGUmmzciccxbVHnfFeQFBfb5UyXg7OBDE9rgpaDLiMf6CcKgErQ+HFTAk611KHZ08Xvt9CFhIOTShawp7PlPFMXGZMRoHggihroa2+WzUHcla9bRiADnGbTQ6WR1UQm989OeCqZQDZtugPqZ1820uXrvTW/QDylG+vHwrYr48P+TcNUKON5ObHatjsA0t4N1kuQJ2NPkyjWqcBzVYbN1RHI9PuWZbNMkaxYYBru1Jl+Hg1mbMZRXaqFjfT76F2yT9bnwmC8UiTidDxkSsuN/OCaPheX0FXyIv91+h1eifQ/P//f7BA1cpJ2k5Oag9ywyAcrZphLB2briecJicioAtOXaGUhW5mxQlSc2+mbf+xaZc7lyI/Pe5S81seRGWUN/GOVidz31e7sSXIFgOIoU05zR+uWFbAGIhF7x2c2cS+BIfcEiXM/+bf161QqDuXvZ95zUJVnENi8VjMnmCNPGeg/fVis315t+MCC9joSOGf4f6FSxUP5yeQ2fGLbHdfBNcweGTDbSZXzOJBjVQ+yrK9zEuKU0DqZPTyuC3x9c881BP2RswLW+3eyPT2tRGHpPQgjv/6RdWHZJb1dPl+9xdyXPJgSN9JWKVJJ4Wsla6avZtlQq7K9512CjftODCo5Ee20=
internal-communication.https.required=true

I followed this tutorial. Can anyone help me figure out what I missed or what's wrong, etc. Thank you guys in advance 😊

hungnt61h commented 3 years ago

I tried to configure trino like previous version then it works now. These properties are required:

hashhar commented 3 years ago

@hungnt61h This new mechanism requires the workers to use the IP address of the co-ordinator in the discovery service uri. It doesn't work if the IP address associated with the co-ordinator changes for some reason.

See the 3rd step under https://trino.io/docs/current/security/internal-communication.html#internal-tls-configuration where it mentions using an IP address for the discover service URI.

cc: @mosabua @dain Do you have more information?

@mosabua Maybe we add the point about using IP address as a NOTE too in https://trino.io/docs/current/security/internal-communication.html#internal-tls-configuration (even though it's mentioned in step 3 of the instructions).

hungnt61h commented 3 years ago

@hashhar My cluster has internal DNS servers, so this discovery.uri property must still require ip address instead of fqdn?

mosabua commented 3 years ago

It does NOT work with fqdn .. IP address usage is required. Do you really think we need to point that out even more in the docs @hashhar ?

hashhar commented 3 years ago

@mosabua with the existing wording it doesn't sound like it's required. It seems natural to assume that you can replace IP with FQDN since that's generally true.

mosabua commented 3 years ago

This is what we decided in the doc PR with @dain and @electrum .. I can send an update though..

mosabua commented 3 years ago

How about https://github.com/trinodb/trino/pull/8902

hungnt61h commented 3 years ago

I’m clear for certain now! Thank you for supporting me! 😃

jiangjialiang918 commented 2 years ago

I tried to configure trino like previous version then it works now. These properties are required:

  • internal-communication.https.keystore.path
  • internal-communication.https.truststore.path
  • node.internal-address-source Why this tutorial said that they must be removed?

I also encountered this problem, what are these values?

jiangjialiang918 commented 2 years ago

In fact, I just want to limit the client to use HTTP and force the use of HTTPS plus account passwords to limit its authority, but when I use HTTPS, there is a problem with the internal communication of the cluster. This is embarrassing.