search

opensearch-project / security

🔐 Secure your cluster with TLS, numerous authentication backends, data masking, audit logging as well as role-based access control on indices, documents, and fields
https://opensearch.org/docs/latest/security-plugin/index/
Apache License 2.0
189 stars 271 forks source link

[BUG] Caused by: java.lang.IllegalArgumentException: 'opensearch-http-channel' is already in use #3664

Closed xiaodizi closed 10 months ago

xiaodizi commented 10 months ago

It started with an error after installation. My Opensearch, pulled the main branch on github for building. The security plugin was also pulled from the main branch on github.

Use sh. /plugins/opensearch-security/tools/install_demo_configuration.sh to generate the certificate file. It's all in the config directory.

The following is the generated configuration：

`######## Start OpenSearch Security Demo Configuration ########

WARNING: revise all the lines below before you go into production

plugins.security.ssl.transport.pemcert_filepath: esnode.pem plugins.security.ssl.transport.pemkey_filepath: esnode-key.pem plugins.security.ssl.transport.pemtrustedcas_filepath: root-ca.pem plugins.security.ssl.transport.enforce_hostname_verification: false plugins.security.ssl.http.enabled: true plugins.security.ssl.http.pemcert_filepath: esnode.pem plugins.security.ssl.http.pemkey_filepath: esnode-key.pem plugins.security.ssl.http.pemtrustedcas_filepath: root-ca.pem plugins.security.allow_unsafe_democertificates: true plugins.security.allow_default_init_securityindex: true plugins.security.authcz.admin_dn:

plugins.security.audit.type: internal_opensearch plugins.security.enable_snapshot_restore_privilege: true plugins.security.check_snapshot_restore_write_privileges: true plugins.security.restapi.roles_enabled: ["all_access", "security_rest_api_access"] plugins.security.system_indices.enabled: true plugins.security.system_indices.indices: [".plugins-ml-config", ".plugins-ml-connector", ".plugins-ml-model-group", ".plugins-ml-model", ".plugins-ml-task", ".plugins-ml-conversation-meta", ".plugins-ml-conversation-interactions", ".opendistro-alerting-config", ".opendistro-alerting-alert", ".opendistro-anomaly-results", ".opendistro-anomaly-detector", ".opendistro-anomaly-checkpoints", ".opendistro-anomaly-detection-state", ".opendistro-reports-", ".opensearch-notifications-", ".opensearch-notebooks", ".opensearch-observability", ".ql-datasources", ".opendistro-asynchronous-search-response", ".replication-metadata-store", ".opensearch-knn-models", ".geospatial-ip2geo-data*"] node.max_local_storage_nodes: 3 ######## End OpenSearch Security Demo Configuration ########`

The plug-in is installed using the built zip package. ./bin/opensearch-plugin install file:///Users/lei.fu/opensearch-security-3.0.0.0-SNAPSHOT.zip

This worked fine, but there was a problem starting the Opensearch server.

[2023-11-07T15:27:13,294][INFO ][o.o.n.Node ] [lei.local] version[3.0.0], pid[41490], build[tar/unknown/unknown], OS[Mac OS X/13.5/x86_64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/11.0.18/11.0.18+9-LTS-195] [2023-11-07T15:27:13,297][INFO ][o.o.n.Node ] [lei.local] JVM home [/Users/lei.fu/java/jdk-11.0.18.jdk/Contents/Home], using bundled JDK/JRE [false] [2023-11-07T15:27:13,298][INFO ][o.o.n.Node ] [lei.local] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+ExitOnOutOfMemoryError, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djdk.attach.allowAttachSelf=true, --add-exports=java.base/jdk.internal.misc=ALL-UNNAMED, --add-exports=java.base/jdk.internal.ref=ALL-UNNAMED, --add-exports=java.base/sun.nio.ch=ALL-UNNAMED, --add-exports=java.management.rmi/com.sun.jmx.remote.internal.rmi=ALL-UNNAMED, --add-exports=java.rmi/sun.rmi.registry=ALL-UNNAMED, --add-exports=java.rmi/sun.rmi.server=ALL-UNNAMED, --add-exports=java.sql/java.sql=ALL-UNNAMED, --add-opens=java.base/java.lang.module=ALL-UNNAMED, --add-opens=java.base/jdk.internal.loader=ALL-UNNAMED, --add-opens=java.base/jdk.internal.ref=ALL-UNNAMED, --add-opens=java.base/jdk.internal.reflect=ALL-UNNAMED, --add-opens=java.base/jdk.internal.math=ALL-UNNAMED, --add-opens=java.base/jdk.internal.module=ALL-UNNAMED, --add-opens=java.base/jdk.internal.util.jar=ALL-UNNAMED, --add-opens=jdk.management/com.sun.management.internal=ALL-UNNAMED, -Djava.io.tmpdir=/var/folders/10/j8h9mmqd6mdb_04_tyql5p7r0000gn/T/opensearch-17567297936889535354, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -XX:MaxDirectMemorySize=536870912, -Dopensearch.path.home=/Users/lei.fu/work/RatuDB-2.1.0, -Dopensearch.path.conf=/Users/lei.fu/work/RatuDB-2.1.0/config, -Dopensearch.distribution.type=tar, -Dopensearch.bundled_jdk=true] [2023-11-07T15:27:13,940][INFO ][o.o.s.s.t.SSLConfig ] [lei.local] SSL dual mode is disabled [2023-11-07T15:27:13,940][INFO ][o.o.s.OpenSearchSecurityPlugin] [lei.local] OpenSearch Config path is /Users/lei.fu/work/RatuDB-2.1.0/config [2023-11-07T15:27:14,121][INFO ][o.o.s.s.DefaultSecurityKeyStore] [lei.local] JVM supports TLSv1.3 [2023-11-07T15:27:14,122][INFO ][o.o.s.s.DefaultSecurityKeyStore] [lei.local] Config directory is /Users/lei.fu/work/RatuDB-2.1.0/config/, from there the key- and truststore files are resolved relatively [2023-11-07T15:27:14,318][INFO ][o.o.s.s.DefaultSecurityKeyStore] [lei.local] TLS Transport Client Provider : JDK [2023-11-07T15:27:14,318][INFO ][o.o.s.s.DefaultSecurityKeyStore] [lei.local] TLS Transport Server Provider : JDK [2023-11-07T15:27:14,319][INFO ][o.o.s.s.DefaultSecurityKeyStore] [lei.local] TLS HTTP Provider : JDK [2023-11-07T15:27:14,319][INFO ][o.o.s.s.DefaultSecurityKeyStore] [lei.local] Enabled TLS protocols for transport layer : [TLSv1.3, TLSv1.2] [2023-11-07T15:27:14,320][INFO ][o.o.s.s.DefaultSecurityKeyStore] [lei.local] Enabled TLS protocols for HTTP layer : [TLSv1.3, TLSv1.2] [2023-11-07T15:27:14,474][INFO ][o.o.s.OpenSearchSecurityPlugin] [lei.local] Clustername: RatuDB_Cluster [2023-11-07T15:27:14,479][WARN ][o.o.s.OpenSearchSecurityPlugin] [lei.local] Directory /Users/lei.fu/work/RatuDB-2.1.0/config has insecure file permissions (should be 0700) [2023-11-07T15:27:14,480][WARN ][o.o.s.OpenSearchSecurityPlugin] [lei.local] File /Users/lei.fu/work/RatuDB-2.1.0/config/kirk.pem has insecure file permissions (should be 0600) [2023-11-07T15:27:14,480][WARN ][o.o.s.OpenSearchSecurityPlugin] [lei.local] File /Users/lei.fu/work/RatuDB-2.1.0/config/esnode-key.pem has insecure file permissions (should be 0600) [2023-11-07T15:27:14,480][WARN ][o.o.s.OpenSearchSecurityPlugin] [lei.local] File /Users/lei.fu/work/RatuDB-2.1.0/config/root-ca.pem has insecure file permissions (should be 0600) [2023-11-07T15:27:14,481][WARN ][o.o.s.OpenSearchSecurityPlugin] [lei.local] File /Users/lei.fu/work/RatuDB-2.1.0/config/kirk-key.pem has insecure file permissions (should be 0600) [2023-11-07T15:27:14,481][WARN ][o.o.s.OpenSearchSecurityPlugin] [lei.local] File /Users/lei.fu/work/RatuDB-2.1.0/config/opensearch-security/internal_users.yml has insecure file permissions (should be 0600) [2023-11-07T15:27:14,481][WARN ][o.o.s.OpenSearchSecurityPlugin] [lei.local] File /Users/lei.fu/work/RatuDB-2.1.0/config/esnode.pem has insecure file permissions (should be 0600) [2023-11-07T15:27:14,483][INFO ][o.o.i.r.ReindexModulePlugin] [lei.local] ReindexPlugin reloadSPI called [2023-11-07T15:27:14,485][INFO ][o.o.i.r.ReindexModulePlugin] [lei.local] Unable to find any implementation for RemoteReindexExtension [2023-11-07T15:27:14,494][INFO ][o.o.p.PluginsService ] [lei.local] loaded module [aggs-matrix-stats] [2023-11-07T15:27:14,495][INFO ][o.o.p.PluginsService ] [lei.local] loaded module [analysis-common] [2023-11-07T15:27:14,495][INFO ][o.o.p.PluginsService ] [lei.local] loaded module [geo] [2023-11-07T15:27:14,495][INFO ][o.o.p.PluginsService ] [lei.local] loaded module [ingest-common] [2023-11-07T15:27:14,495][INFO ][o.o.p.PluginsService ] [lei.local] loaded module [ingest-geoip] [2023-11-07T15:27:14,495][INFO ][o.o.p.PluginsService ] [lei.local] loaded module [ingest-user-agent] [2023-11-07T15:27:14,495][INFO ][o.o.p.PluginsService ] [lei.local] loaded module [lang-expression] [2023-11-07T15:27:14,496][INFO ][o.o.p.PluginsService ] [lei.local] loaded module [lang-mustache] [2023-11-07T15:27:14,496][INFO ][o.o.p.PluginsService ] [lei.local] loaded module [lang-painless] [2023-11-07T15:27:14,496][INFO ][o.o.p.PluginsService ] [lei.local] loaded module [mapper-extras] [2023-11-07T15:27:14,496][INFO ][o.o.p.PluginsService ] [lei.local] loaded module [opensearch-dashboards] [2023-11-07T15:27:14,496][INFO ][o.o.p.PluginsService ] [lei.local] loaded module [parent-join] [2023-11-07T15:27:14,496][INFO ][o.o.p.PluginsService ] [lei.local] loaded module [percolator] [2023-11-07T15:27:14,497][INFO ][o.o.p.PluginsService ] [lei.local] loaded module [rank-eval] [2023-11-07T15:27:14,497][INFO ][o.o.p.PluginsService ] [lei.local] loaded module [reindex] [2023-11-07T15:27:14,497][INFO ][o.o.p.PluginsService ] [lei.local] loaded module [repository-url] [2023-11-07T15:27:14,497][INFO ][o.o.p.PluginsService ] [lei.local] loaded module [search-pipeline-common] [2023-11-07T15:27:14,497][INFO ][o.o.p.PluginsService ] [lei.local] loaded module [test-delayed-aggs] [2023-11-07T15:27:14,497][INFO ][o.o.p.PluginsService ] [lei.local] loaded module [transport-netty4] [2023-11-07T15:27:14,498][INFO ][o.o.p.PluginsService ] [lei.local] loaded plugin [opensearch-security] [2023-11-07T15:27:14,512][INFO ][o.o.s.OpenSearchSecurityPlugin] [lei.local] Disabled https compression by default to mitigate BREACH attacks. You can enable it by setting 'http.compression: true' in opensearch.yml [2023-11-07T15:27:14,515][INFO ][o.o.e.ExtensionsManager ] [lei.local] ExtensionsManager initialized [2023-11-07T15:27:14,537][INFO ][o.o.e.NodeEnvironment ] [lei.local] using [1] data paths, mounts [[/System/Volumes/Data (/dev/disk1s2)]], net usable_space [465.8gb], net total_space [931.5gb], types [apfs] [2023-11-07T15:27:14,540][INFO ][o.o.e.NodeEnvironment ] [lei.local] heap size [1gb], compressed ordinary object pointers [true] [2023-11-07T15:27:14,578][INFO ][o.o.n.Node ] [lei.local] node name [localhost], node ID [YSnUvLASQFWbXysNXv-m0Q], cluster name [RatuDB_Cluster], roles [ingest, remote_cluster_client, data, cluster_manager] [2023-11-07T15:27:14,675][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [lei.local] fatal error in thread [main], exiting java.lang.ExceptionInInitializerError: null at org.opensearch.transport.Netty4ModulePlugin.getSettings(Netty4ModulePlugin.java:70) ~[?:?] at org.opensearch.plugins.PluginsService.lambda$getPluginSettings$0(PluginsService.java:112) ~[RatuDB-2.1.0.jar:2.1.0] at java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:271) ~[?:?] at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655) ~[?:?] at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) ~[?:?] at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) ~[?:?] at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) ~[?:?] at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?] at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578) ~[?:?] at org.opensearch.plugins.PluginsService.getPluginSettings(PluginsService.java:112) ~[RatuDB-2.1.0.jar:2.1.0] at org.opensearch.node.Node.<init>(Node.java:570) ~[RatuDB-2.1.0.jar:2.1.0] at org.opensearch.node.Node.<init>(Node.java:410) ~[RatuDB-2.1.0.jar:2.1.0] at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[RatuDB-2.1.0.jar:2.1.0] at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[RatuDB-2.1.0.jar:2.1.0] at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[RatuDB-2.1.0.jar:2.1.0] at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:212) ~[RatuDB-2.1.0.jar:2.1.0] at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:203) ~[RatuDB-2.1.0.jar:2.1.0] at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) ~[RatuDB-2.1.0.jar:2.1.0] at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) ~[opensearch-cli-2.1.0.jar:2.1.0] at org.opensearch.cli.Command.main(Command.java:101) ~[opensearch-cli-2.1.0.jar:2.1.0] at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:169) ~[RatuDB-2.1.0.jar:2.1.0] at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:113) ~[RatuDB-2.1.0.jar:2.1.0] Caused by: java.lang.IllegalArgumentException: 'opensearch-http-channel' is already in use at io.netty.util.ConstantPool.createOrThrow(ConstantPool.java:108) ~[netty-common-4.1.100.Final.jar:4.1.100.Final] at io.netty.util.ConstantPool.newInstance(ConstantPool.java:90) ~[netty-common-4.1.100.Final.jar:4.1.100.Final] at io.netty.util.AttributeKey.newInstance(AttributeKey.java:55) ~[netty-common-4.1.100.Final.jar:4.1.100.Final] at org.opensearch.http.netty4.Netty4HttpServerTransport.<clinit>(Netty4HttpServerTransport.java:337) ~[?:?] ... 22 more fatal error in thread [main], exiting java.lang.ExceptionInInitializerError at org.opensearch.transport.Netty4ModulePlugin.getSettings(Netty4ModulePlugin.java:70) at org.opensearch.plugins.PluginsService.lambda$getPluginSettings$0(PluginsService.java:112) at java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:271) at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655) at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578) at org.opensearch.plugins.PluginsService.getPluginSettings(PluginsService.java:112) at org.opensearch.node.Node.<init>(Node.java:570) at org.opensearch.node.Node.<init>(Node.java:410) at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:212) at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:203) at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) at org.opensearch.cli.Command.main(Command.java:101) at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:169) at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:113) Caused by: java.lang.IllegalArgumentException: 'opensearch-http-channel' is already in use at io.netty.util.ConstantPool.createOrThrow(ConstantPool.java:108) at io.netty.util.ConstantPool.newInstance(ConstantPool.java:90) at io.netty.util.AttributeKey.newInstance(AttributeKey.java:55) at org.opensearch.http.netty4.Netty4HttpServerTransport.<clinit>(Netty4HttpServerTransport.java:337) ... 22 more

Why is this a problem?

xiaodizi commented 10 months ago

After investigation, I found that there was a jar package conflict, because I added Cassandra jar package under the server module, so I needed netty jar package. But the plugin has the netty client jar which causes a conflict.

I would like to ask whether the plugin jar package can be written into the Opensearch lib folder? This will solve the problem of conflicts.

cwperks commented 10 months ago

@xiaodizi thank you for an update. server/ dependencies are added sparingly in OpenSearch and usually if functionality is needed to extend the core its done through plugins. Core has some native plugins (called modules/) and other optional plugins that cluster admins can install to extend the features of the core of OpenSearch.

Would you be able to accomplish your goals through plugins?

stephen-crawford commented 10 months ago

[Triage] Hi @xiaodizi, thank you for opening this issue. This seems like you are trying to configure a special set up for your cluster. For support with that I would recommend filing an issue/question over on the OpenSearch forums. There you should be able to get advice on setting up your configuration effectively.

Alternatively, you can always visit the OpenSearch slack channel to get real-time support.

xiaodizi commented 10 months ago

@xiaodizi thank you for an update. server/ dependencies are added sparingly in OpenSearch and usually if functionality is needed to extend the core its done through plugins. Core has some native plugins (called modules/) and other optional plugins that cluster admins can install to extend the features of the core of OpenSearch.

Would you be able to accomplish your goals through plugins?

Thanks for the reply, I'll try.