[BUG] Window compatibility test failing java.lang.IllegalStateException: Unsupported transport.type

[Gaganjuneja]

What is the bug? Possibly due to race condition windows backward compatibility test is failing -

[ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [securityBwcCluster1-0] uncaught exception in thread [main]
org.opensearch.bootstrap.StartupException: java.lang.IllegalStateException: Unsupported transport.type [org.opensearch.security.ssl.http.netty.SecuritySSLNettyTransport]
    at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:184) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
    at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:171) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
    at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
    at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) ~[opensearch-cli-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
    at org.opensearch.cli.Command.main(Command.java:101) ~[opensearch-cli-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
    at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:137) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
    at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:103) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
 Caused by: java.lang.IllegalStateException: Unsupported transport.type [org.opensearch.security.ssl.http.netty.SecuritySSLNettyTransport]
    at org.opensearch.common.network.NetworkModule.getTransportSupplier(NetworkModule.java:257) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
    at org.opensearch.node.Node.<init>(Node.java:897) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
    at org.opensearch.node.Node.<init>(Node.java:401) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
    at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
    at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
    at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
    at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
    ... 6 more

How can one reproduce the bug? https://github.com/opensearch-project/security/pull/3463

build request - https://github.com/opensearch-project/security/actions/runs/6414322099/job/17414527728?pr=3463

What is the expected behavior? A clear and concise description of what you expected to happen.

What is your host/environment?

• OS: [e.g. iOS] Windows 11 & 17
• Version [e.g. 22]
• Plugins org.opensearch.security.ssl.http.netty.SecuritySSLNettyTransport

Do you have any screenshots? If applicable, add screenshots to help explain your problem.

Do you have any additional context? Add any other context about the problem.

[peternied]

@peternied I was trying to reproduce the issue on Windows box locally, sadly no luck, BUT it looks to me this is the issue with distribution: the OS 3.0.0-SNAPSHOT was not yet updated (so no changes in transport) but security plugin tried to use the modified signature - the instantiation fails (which will explain the error)

Originally posted by @reta in https://github.com/opensearch-project/security/issues/3463#issuecomment-1748838217

[reta]

I suspect we may see it with https://github.com/opensearch-project/security/pull/3469 or/and https://github.com/opensearch-project/security/pull/3468 as well :-)

[stephen-crawford]

[Triage] Hi @Gaganjuneja, thank you for filing this issue. Right now the action item from this issue is to make sure we are not running with the feature flag on. That being said, the other problem this relates to appears to be something outside the scope of the security plugin and more focused on the traceability team.

[cwperks]

I've seen a similar issue when the experimental feature flag (opensearch.experimental.feature.telemetry.enabled) is set to true and the security plugin installed.

I get this error on startup:

Caused by: org.opensearch.core.common.io.stream.NotSerializableExceptionWrapper: class_cast_exception: class org.opensearch.telemetry.tracing.channels.TraceableTcpTransportChannel cannot be cast to class org.opensearch.transport.TcpTransportChannel (org.opensearch.telemetry.tracing.channels.TraceableTcpTransportChannel and org.opensearch.transport.TcpTransportChannel are in unnamed module of loader 'app')
    at org.opensearch.security.ssl.transport.SecuritySSLRequestHandler.messageReceived(SecuritySSLRequestHandler.java:116) ~[?:?]
    at org.opensearch.security.OpenSearchSecurityPlugin$6$1.messageReceived(OpenSearchSecurityPlugin.java:790) ~[?:?]
    at org.opensearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:106) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
    at org.opensearch.transport.InboundHandler.handleRequest(InboundHandler.java:262) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
    at org.opensearch.transport.InboundHandler.messageReceived(InboundHandler.java:140) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
    at org.opensearch.transport.InboundHandler.inboundMessage(InboundHandler.java:123) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
    at org.opensearch.transport.TcpTransport.inboundMessage(TcpTransport.java:770) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
    at org.opensearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:175) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
    at org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:150) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
    at org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:115) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
    at org.opensearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:95) ~[?:?]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[?:?]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
    at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280) ~[?:?]