search

Graylog2 / graylog2-server

Free and open log management
https://www.graylog.org
Other
7.21k stars 1.05k forks source link

[bug] Data node does not start up successfully after importing custom CA. #19759

Open mako42 opened 6 days ago

mako42 commented 6 days ago

This is the bug Dan experienced and reported in Slack.

Dan used his own windows CA. He had no issues importing it, but afterwards the data node does not start successfully.

Expected Behavior

Current Behavior

The data node does not start successfully after the CA import, instead it throws errors:

Index cds_4 migration failed after 0 seconds: GetTaskResponse[completed=true, task=Task[node=jMDF5RCbRSKzHnzcv4i1eA, id=8234, type=transport, action=indices:data/write/reindex, status=TaskStatus[total=0, updated=0, created=0, deleted=0, batches=0, versionConflicts=0, noops=0, failures=null], description=reindex from [scheme=https host=glos01.eclipsenetwork.org port=9200 pathPrefix=/ query={ "match_all" : { "boost" : 1.0 } } username=elastic password=<<>>][cds_4] to [cds_4], startTimeInMillis=1719421000765, runningTimeInNanos=350832478, cancellable=true, cancelled=false, headers={X-Opaque-Id=667c47ce66e1a566a1b983a9}], error=type='s_s_l_handshake_exception', reason='PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target', causedBy='{type=validator_exception, reason=PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target, caused_by={type=sun_cert_path_builder_exception, reason=unable to find valid certification path to requested target}}'].

Seems like the certificate is the culprit, there's a certificate_unknown in its output:

2024-06-26T18:02:25.450Z INFO [OpensearchProcessImpl] [2024-06-26T18:02:25,448][ERROR][o.o.s.s.h.n.SecuritySSLNettyHttpServerTransport] [gldn03.lab.eclipsenetwork.org] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown 2024-06-26T18:02:25.450Z INFO [OpensearchProcessImpl] javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown 2024-06-26T18:02:25.450Z INFO [OpensearchProcessImpl] at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:130) ~[?:?] 2024-06-26T18:02:25.450Z INFO [OpensearchProcessImpl] at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:117) ~[?:?] 2024-06-26T18:02:25.450Z INFO [OpensearchProcessImpl] at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:365) ~[?:?] 2024-06-26T18:02:25.450Z INFO [OpensearchProcessImpl] at java.base/sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:287) ~[?:?] 2024-06-26T18:02:25.451Z INFO [OpensearchProcessImpl] at java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:204) ~[?:?] 2024-06-26T18:02:25.451Z INFO [OpensearchProcessImpl] at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:172) ~[?:?] 2024-06-26T18:02:25.451Z INFO [OpensearchProcessImpl] at java.base/sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:736) ~[?:?] 2024-06-26T18:02:25.451Z INFO [OpensearchProcessImpl] at java.base/sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:691) ~[?:?] 2024-06-26T18:02:25.451Z INFO [OpensearchProcessImpl] at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:506) ~[?:?] 2024-06-26T18:02:25.451Z INFO [OpensearchProcessImpl] at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:482) ~[?:?] 2024-06-26T18:02:25.451Z INFO [OpensearchProcessImpl] at java.base/javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:679) ~[?:?] 2024-06-26T18:02:25.451Z INFO [OpensearchProcessImpl] at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:310) ~[netty-handler-4.1.106.Final.jar:4.1.106.Final] 2024-06-26T18:02:25.452Z INFO [OpensearchProcessImpl] at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1445) ~[netty-handler-4.1.106.Final.jar:4.1.106.Final] 2024-06-26T18:02:25.452Z INFO [OpensearchProcessImpl] at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1338) ~[netty-handler-4.1.106.Final.jar:4.1.106.Final] 2024-06-26T18:02:25.452Z INFO [OpensearchProcessImpl] at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1387) ~[netty-handler-4.1.106.Final.jar:4.1.106.Final] 2024-06-26T18:02:25.452Z INFO [OpensearchProcessImpl] at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) ~[netty-codec-4.1.106.Final.jar:4.1.106.Final] 2024-06-26T18:02:25.452Z INFO [OpensearchProcessImpl] at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) ~[netty-codec-4.1.106.Final.jar:4.1.106.Final] 2024-06-26T18:02:25.452Z INFO [OpensearchProcessImpl] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) ~[netty-codec-4.1.106.Final.jar:4.1.106.Final] 2024-06-26T18:02:25.452Z INFO [OpensearchProcessImpl] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.106.Final.jar:4.1.106.Final] 2024-06-26T18:02:25.452Z INFO [OpensearchProcessImpl] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.106.Final.jar:4.1.106.Final] 2024-06-26T18:02:25.453Z INFO [OpensearchProcessImpl] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.106.Final.jar:4.1.106.Final] 2024-06-26T18:02:25.453Z INFO [OpensearchProcessImpl] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.106.Final.jar:4.1.106.Final] 2024-06-26T18:02:25.453Z INFO [OpensearchProcessImpl] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) [netty-transport-4.1.106.Final.jar:4.1.106.Final] 2024-06-26T18:02:25.453Z INFO [OpensearchProcessImpl] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.106.Final.jar:4.1.106.Final] 2024-06-26T18:02:25.453Z INFO [OpensearchProcessImpl] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.106.Final.jar:4.1.106.Final] 2024-06-26T18:02:25.453Z INFO [OpensearchProcessImpl] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.106.Final.jar:4.1.106.Final] 2024-06-26T18:02:25.453Z INFO [OpensearchProcessImpl] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) [netty-transport-4.1.106.Final.jar:4.1.106.Final] 2024-06-26T18:02:25.453Z INFO [OpensearchProcessImpl] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) [netty-transport-4.1.106.Final.jar:4.1.106.Final] 2024-06-26T18:02:25.453Z INFO [OpensearchProcessImpl] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) [netty-transport-4.1.106.Final.jar:4.1.106.Final] 2024-06-26T18:02:25.453Z INFO [OpensearchProcessImpl] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.106.Final.jar:4.1.106.Final] 2024-06-26T18:02:25.454Z INFO [OpensearchProcessImpl] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.106.Final.jar:4.1.106.Final] 2024-06-26T18:02:25.454Z INFO [OpensearchProcessImpl] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.106.Final.jar:4.1.106.Final] 2024-06-26T18:02:25.454Z INFO [OpensearchProcessImpl] at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 2024-06-26T18:02:25.454Z INFO [OpensearchProcessImpl] [2024-06-26T18:02:25,450][WARN ][o.o.h.AbstractHttpServerTransport] [gldn03.lab.eclipsenetwork.org] caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=/192.168.99.92:9200, remoteAddress=/192.168.1.231:40352}

Possible Solution

Steps to Reproduce (for bugs)

  1. Start migration (remote reindexing)
  2. Upload own CA
  3. Data node won't start up successfully.

Context

Migration testing.

Your Environment

mako42 commented 6 days ago

Linking @mcdowellster: for any questions and missing info, ask him and not me :D