opensearch-project / helm-charts

:wheel_of_dharma: A community repository for Helm Charts of OpenSearch Project.
https://opensearch.org/docs/latest/opensearch/install/helm/
Apache License 2.0
173 stars 235 forks source link

[BUG][Opensearch] It is not possible to deploy the chart with the default settings. #617

Closed LemonDouble closed 1 day ago

LemonDouble commented 2 weeks ago

Describe the bug When deploying the OpenSearch chart with default settings, an error occurs in the Security plugin, causing the deployment to fail.

To Reproduce

When deploying the default chart with only the OPENSEARCH_INITIAL_ADMIN_PASSWORD environment variable set, the deployment fails.

Error Log ``` Enabling OpenSearch Security Plugin Enabling execution of install_demo_configuration.sh for OpenSearch Security Plugin OpenSearch 2.12.0 onwards, the OpenSearch Security Plugin a change that requires an initial password for 'admin' user. Please define an environment variable 'OPENSEARCH_INITIAL_ADMIN_PASSWORD' with a strong password string. If a password is not provided, the setup will quit. For more details, please visit: https://opensearch.org/docs/latest/install-and-configure/install-opensearch/docker/ ### OpenSearch Security Demo Installer ### ** Warning: Do not use on production or public reachable systems ** OpenSearch install type: rpm/deb on Linux 5.15.0-124-generic amd64 OpenSearch config dir: /usr/share/opensearch/config/ OpenSearch config file: /usr/share/opensearch/config/opensearch.yml OpenSearch bin dir: /usr/share/opensearch/bin/ OpenSearch plugins dir: /usr/share/opensearch/plugins/ OpenSearch lib dir: /usr/share/opensearch/lib/ Detected OpenSearch Version: 2.18.0 Detected OpenSearch Security Version: 2.18.0.0 /usr/share/opensearch/config/opensearch.yml seems to be already configured for Security. Quit. Enabling execution of OPENSEARCH_HOME/bin/opensearch-performance-analyzer/performance-analyzer-agent-cli for OpenSearch Performance Analyzer Plugin WARNING: Using incubator modules: jdk.incubator.vector WARNING: A terminally deprecated method in java.lang.System has been called WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/opensearch/lib/opensearch-2.18.0.jar) WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch WARNING: System::setSecurityManager will be removed in a future release Nov 08, 2024 11:43:26 PM sun.util.locale.provider.LocaleProviderAdapter WARNING: COMPAT locale provider will be removed in a future release WARNING: A terminally deprecated method in java.lang.System has been called WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/opensearch/lib/opensearch-2.18.0.jar) WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security WARNING: System::setSecurityManager will be removed in a future release [2024-11-08T23:43:26,441][INFO ][o.o.n.Node ] [opensearch-cluster-master-0] version[2.18.0], pid[1], build[tar/99a9a81da366173b0c2b963b26ea92e15ef34547/2024-10-31T19:08:39.157471098Z], OS[Linux/5.15.0-124-generic/amd64], JVM[Eclipse Adoptium/OpenJDK 64-Bit Server VM/21.0.5/21.0.5+11-LTS] [2024-11-08T23:43:26,442][INFO ][o.o.n.Node ] [opensearch-cluster-master-0] JVM home [/usr/share/opensearch/jdk], using bundled JDK/JRE [true] [2024-11-08T23:43:26,442][INFO ][o.o.n.Node ] [opensearch-cluster-master-0] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.security.manager=allow, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/opensearch-9408234510311292452, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.security.manager=allow, --add-modules=jdk.incubator.vector, -Djava.util.concurrent.ForkJoinPool.common.threadFactory=org.opensearch.secure_sm.SecuredForkJoinWorkerThreadFactory, -Dclk.tck=100, -Djdk.attach.allowAttachSelf=true, -Djava.security.policy=/usr/share/opensearch/config/opensearch-performance-analyzer/opensearch_security.policy, --add-opens=jdk.attach/sun.tools.attach=ALL-UNNAMED, -Dopensearch.cgroups.hierarchy.override=/, -Xmx512M, -Xms512M, -XX:MaxDirectMemorySize=268435456, -Dopensearch.path.home=/usr/share/opensearch, -Dopensearch.path.conf=/usr/share/opensearch/config, -Dopensearch.distribution.type=tar, -Dopensearch.bundled_jdk=true] [2024-11-08T23:43:26,561][INFO ][o.a.l.i.v.PanamaVectorizationProvider] [opensearch-cluster-master-0] Java vector incubator API enabled; uses preferredBitSize=256; FMA enabled [2024-11-08T23:43:27,108][INFO ][o.o.s.s.t.SSLConfig ] [opensearch-cluster-master-0] SSL dual mode is disabled [2024-11-08T23:43:27,108][INFO ][o.o.s.OpenSearchSecurityPlugin] [opensearch-cluster-master-0] OpenSearch Config path is /usr/share/opensearch/config [2024-11-08T23:43:27,195][WARN ][o.o.s.s.SslSettingsManager] [opensearch-cluster-master-0] OpenSSL not available (this is not an error, we simply fallback to built-in JDK SSL) because of {} java.lang.ClassNotFoundException: io.netty.internal.tcnative.SSLContext at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:445) ~[?:?] at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:593) ~[?:?] at java.base/java.net.FactoryURLClassLoader.loadClass(URLClassLoader.java:872) ~[?:?] at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:526) ~[?:?] at java.base/java.lang.Class.forName0(Native Method) ~[?:?] at java.base/java.lang.Class.forName(Class.java:534) ~[?:?] at java.base/java.lang.Class.forName(Class.java:513) ~[?:?] at io.netty.handler.ssl.OpenSsl.(OpenSsl.java:95) ~[netty-handler-4.1.114.Final.jar:4.1.114.Final] at org.opensearch.security.ssl.OpenSearchSecuritySSLPlugin$4.run(OpenSearchSecuritySSLPlugin.java:218) ~[opensearch-security-2.18.0.0.jar:2.18.0.0] at java.base/java.security.AccessController.doPrivileged(AccessController.java:319) ~[?:?] at org.opensearch.security.ssl.OpenSearchSecuritySSLPlugin.(OpenSearchSecuritySSLPlugin.java:213) [opensearch-security-2.18.0.0.jar:2.18.0.0] at org.opensearch.security.OpenSearchSecurityPlugin.(OpenSearchSecurityPlugin.java:318) [opensearch-security-2.18.0.0.jar:2.18.0.0] at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62) ~[?:?] at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502) ~[?:?] at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486) ~[?:?] at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:796) [opensearch-2.18.0.jar:2.18.0] at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:744) [opensearch-2.18.0.jar:2.18.0] at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:545) [opensearch-2.18.0.jar:2.18.0] at org.opensearch.plugins.PluginsService.(PluginsService.java:197) [opensearch-2.18.0.jar:2.18.0] at org.opensearch.node.Node.(Node.java:523) [opensearch-2.18.0.jar:2.18.0] at org.opensearch.node.Node.(Node.java:450) [opensearch-2.18.0.jar:2.18.0] at org.opensearch.bootstrap.Bootstrap$5.(Bootstrap.java:242) [opensearch-2.18.0.jar:2.18.0] at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) [opensearch-2.18.0.jar:2.18.0] at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) [opensearch-2.18.0.jar:2.18.0] at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) [opensearch-2.18.0.jar:2.18.0] at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:172) [opensearch-2.18.0.jar:2.18.0] at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) [opensearch-2.18.0.jar:2.18.0] at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) [opensearch-cli-2.18.0.jar:2.18.0] at org.opensearch.cli.Command.main(Command.java:101) [opensearch-cli-2.18.0.jar:2.18.0] at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:138) [opensearch-2.18.0.jar:2.18.0] at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:104) [opensearch-2.18.0.jar:2.18.0] [2024-11-08T23:43:27,288][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [opensearch-cluster-master-0] uncaught exception in thread [main] org.opensearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to load plugin class [org.opensearch.security.OpenSearchSecurityPlugin] at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:185) ~[opensearch-2.18.0.jar:2.18.0] at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:172) ~[opensearch-2.18.0.jar:2.18.0] at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) ~[opensearch-2.18.0.jar:2.18.0] at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) ~[opensearch-cli-2.18.0.jar:2.18.0] at org.opensearch.cli.Command.main(Command.java:101) ~[opensearch-cli-2.18.0.jar:2.18.0] at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:138) ~[opensearch-2.18.0.jar:2.18.0] at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:104) ~[opensearch-2.18.0.jar:2.18.0] Caused by: java.lang.IllegalStateException: failed to load plugin class [org.opensearch.security.OpenSearchSecurityPlugin] at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:805) ~[opensearch-2.18.0.jar:2.18.0] at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:744) ~[opensearch-2.18.0.jar:2.18.0] at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:545) ~[opensearch-2.18.0.jar:2.18.0] at org.opensearch.plugins.PluginsService.(PluginsService.java:197) ~[opensearch-2.18.0.jar:2.18.0] at org.opensearch.node.Node.(Node.java:523) ~[opensearch-2.18.0.jar:2.18.0] at org.opensearch.node.Node.(Node.java:450) ~[opensearch-2.18.0.jar:2.18.0] at org.opensearch.bootstrap.Bootstrap$5.(Bootstrap.java:242) ~[opensearch-2.18.0.jar:2.18.0] at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.18.0.jar:2.18.0] at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.18.0.jar:2.18.0] at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) ~[opensearch-2.18.0.jar:2.18.0] ... 6 more Caused by: java.lang.reflect.InvocationTargetException at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:74) ~[?:?] at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502) ~[?:?] at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486) ~[?:?] at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:796) ~[opensearch-2.18.0.jar:2.18.0] at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:744) ~[opensearch-2.18.0.jar:2.18.0] at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:545) ~[opensearch-2.18.0.jar:2.18.0] at org.opensearch.plugins.PluginsService.(PluginsService.java:197) ~[opensearch-2.18.0.jar:2.18.0] at org.opensearch.node.Node.(Node.java:523) ~[opensearch-2.18.0.jar:2.18.0] at org.opensearch.node.Node.(Node.java:450) ~[opensearch-2.18.0.jar:2.18.0] at org.opensearch.bootstrap.Bootstrap$5.(Bootstrap.java:242) ~[opensearch-2.18.0.jar:2.18.0] uncaught exception in thread [main] at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.18.0.jar:2.18.0] at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.18.0.jar:2.18.0] at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) ~[opensearch-2.18.0.jar:2.18.0] ... 6 more Caused by: org.opensearch.OpenSearchException: Unable to read the file root-ca.pem. Please make sure this files exists and is readable regarding to permissions at org.opensearch.security.ssl.config.SslCertificatesLoader.resolvePath(SslCertificatesLoader.java:165) ~[?:?] at org.opensearch.security.ssl.config.SslCertificatesLoader.loadConfiguration(SslCertificatesLoader.java:85) ~[?:?] at org.opensearch.security.ssl.SslSettingsManager.loadConfigurations(SslSettingsManager.java:137) ~[?:?] at org.opensearch.security.ssl.SslSettingsManager.buildSslContexts(SslSettingsManager.java:93) ~[?:?] at org.opensearch.security.ssl.SslSettingsManager.(SslSettingsManager.java:80) ~[?:?] at org.opensearch.security.ssl.OpenSearchSecuritySSLPlugin.(OpenSearchSecuritySSLPlugin.java:249) ~[?:?] at org.opensearch.security.OpenSearchSecurityPlugin.(OpenSearchSecurityPlugin.java:318) ~[?:?] at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62) ~[?:?] at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502) ~[?:?] at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486) ~[?:?] at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:796) ~[opensearch-2.18.0.jar:2.18.0] at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:744) ~[opensearch-2.18.0.jar:2.18.0] at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:545) ~[opensearch-2.18.0.jar:2.18.0] at org.opensearch.plugins.PluginsService.(PluginsService.java:197) ~[opensearch-2.18.0.jar:2.18.0] at org.opensearch.node.Node.(Node.java:523) ~[opensearch-2.18.0.jar:2.18.0] at org.opensearch.node.Node.(Node.java:450) ~[opensearch-2.18.0.jar:2.18.0] at org.opensearch.bootstrap.Bootstrap$5.(Bootstrap.java:242) ~[opensearch-2.18.0.jar:2.18.0] at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.18.0.jar:2.18.0] at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.18.0.jar:2.18.0] at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) ~[opensearch-2.18.0.jar:2.18.0] ... 6 more java.lang.IllegalStateException: failed to load plugin class [org.opensearch.security.OpenSearchSecurityPlugin] Likely root cause: OpenSearchException[Unable to read the file root-ca.pem. Please make sure this files exists and is readable regarding to permissions] at org.opensearch.security.ssl.config.SslCertificatesLoader.resolvePath(SslCertificatesLoader.java:165) at org.opensearch.security.ssl.config.SslCertificatesLoader.loadConfiguration(SslCertificatesLoader.java:85) at org.opensearch.security.ssl.SslSettingsManager.loadConfigurations(SslSettingsManager.java:137) at org.opensearch.security.ssl.SslSettingsManager.buildSslContexts(SslSettingsManager.java:93) at org.opensearch.security.ssl.SslSettingsManager.(SslSettingsManager.java:80) at org.opensearch.security.ssl.OpenSearchSecuritySSLPlugin.(OpenSearchSecuritySSLPlugin.java:249) at org.opensearch.security.OpenSearchSecurityPlugin.(OpenSearchSecurityPlugin.java:318) at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62) at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502) at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486) at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:796) at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:744) at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:545) at org.opensearch.plugins.PluginsService.(PluginsService.java:197) at org.opensearch.node.Node.(Node.java:523) at org.opensearch.node.Node.(Node.java:450) at org.opensearch.bootstrap.Bootstrap$5.(Bootstrap.java:242) at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:172) at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) at org.opensearch.cli.Command.main(Command.java:101) at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:138) at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:104) For complete error details, refer to the log at /usr/share/opensearch/logs/opensearch-cluster.log ```

Expected behavior

The chart should be deployed successfully with the default settings.

Chart Name opensearch

Host/Environment (please complete the following information):

Additional context

"It seems that the Security Plugin's demo configuration is not working because the opensearch.yml file is declared in the values.yaml."

In the Security plugin, if the opensearch.yml file is missing, it generates demo certificates. However, since the values.yaml file creates an opensearch.yml file, the plugin skips generating demo certificates, leading to this issue. (See Link)

Previously, the Security Plugin could not detect nested YAML, resulting in the demo configuration of the Security Plugin always being executed. However, it appears that this bug has been fixed with this PR(from Security Plugin release 2.18.0 onwards).

Therefore, it appears that this issue started occurring from the Opensearch chart version 2.27.0.

Would it be okay if I submit a PR for this issue? It seems that commenting out lines 49 to 99 in values.yaml would resolve the problem.

siennathesane commented 5 days ago

I ran into this problem and fixed it with:

singleNode: true
rbac:
  create: true
  automountServiceAccountToken: true
ingress:
  enabled: true
  annotations:
    traefik.ingress.kubernetes.io/router.entrypoints: web
    kubernetes.io/ingress.class: traefik
  hosts:
    - opensearch.127.0.0.1.sslip.io
config:
  opensearch.yml: "" #disable the default due to https://github.com/opensearch-project/helm-charts/issues/617
extraEnvs:
  - name: OPENSEARCH_INITIAL_ADMIN_PASSWORD
    value: <some-password>

for Rancher Desktop

StefanSchuhart commented 3 days ago

Can someone give me a working values.yaml configuration? I am facing the same problem with chart versions 2.26.1 and 2.27.0 . I tried with different combinations of:

config:
  opensearch.yml: "" 

# or the default one

and with or without

extraEnvs:
  - name: DISABLE_INSTALL_DEMO_CONFIG
    value: "true"

(Also tried version 2.25.0 and got a different error: java.lang.IllegalArgumentException: Could not load codec 'Lucene95'. Did you forget to add lucene-backward-codecs.jar?)

ms-semarchy commented 3 days ago

It works for me emptying opensearch.yml:

config:
  opensearch.yml: ""
singleNode: true
LemonDouble commented 3 days ago

@StefanSchuhart Could you share the error message that occurs when opensearch.yml is left as an empty string?