[BUG][Chart Name] opensearch not staring with master only role

[ibrahimjelliti]

Describe the bug I am deploying Opensearch within EKS cluster using the helm chat, I want to achieve 3 master noes with 3 @data nodes. when I install the cluster I got an error

{"type": "server", "timestamp": "2023-07-05T04:04:04,413Z", "level": "INFO", "component": "o.o.s.OpenSearchSecurityPlugin", "cluster.name": "opensearch-cluster", "node.name": "opensearch-cluster-master-0", "message": "Disabled https compression by default to mitigate BREACH attacks. You can enable it by setting 'http.compression: true' in opensearch.yml" } {"type": "server", "timestamp": "2023-07-05T04:04:04,416Z", "level": "INFO", "component": "o.o.e.ExtensionsManager", "cluster.name": "opensearch-cluster", "node.name": "opensearch-cluster-master-0", "message": "ExtensionsManager initialized" } {"type": "server", "timestamp": "2023-07-05T04:04:04,433Z", "level": "INFO", "component": "o.o.e.NodeEnvironment", "cluster.name": "opensearch-cluster", "node.name": "opensearch-cluster-master-0", "message": "using [1] data paths, mounts [[/usr/share/opensearch/data (/dev/nvme2n1)]], net usable_space [7.9gb], net total_space [7.9gb], types [xfs]" } {"type": "server", "timestamp": "2023-07-05T04:04:04,434Z", "level": "INFO", "component": "o.o.e.NodeEnvironment", "cluster.name": "opensearch-cluster", "node.name": "opensearch-cluster-master-0", "message": "heap size [2gb], compressed ordinary object pointers [true]" } {"type": "server", "timestamp": "2023-07-05T04:04:04,449Z", "level": "ERROR", "component": "o.o.b.OpenSearchUncaughtExceptionHandler", "cluster.name": "opensearch-cluster", "node.name": "opensearch-cluster-master-0", "message": "uncaught exception in thread [main]", "stacktrace": ["org.opensearch.bootstrap.StartupException: java.lang.IllegalStateException: node does not have the data and cluster_manager roles but has index metadata: [/usr/share/opensearch/data/nodes/0/indices/1E4WoMDsTEGj_Ce4qIm0bg/_state, /usr/share/opensearch/data/nodes/0/indices/aGUuCYRzStGtsUGbJrlrZQ/_state, /usr/share/opensearch/data/nodes/0/indices/mJT2Jq_qSFW6DgsfcHKIaA/_state, /usr/share/opensearch/data/nodes/0/indices/i9wChBVBTE2i1msQH1Q6Qw/_state]. Use 'opensearch-node repurpose' tool to clean up", "at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:184) ~[opensearch-2.8.0.jar:2.8.0]", "at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:171) ~[opensearch-2.8.0.jar:2.8.0]", "at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) ~[opensearch-2.8.0.jar:2.8.0]", "at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) ~[opensearch-cli-2.8.0.jar:2.8.0]", "at org.opensearch.cli.Command.main(Command.java:101) ~[opensearch-cli-2.8.0.jar:2.8.0]", "at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:137) ~[opensearch-2.8.0.jar:2.8.0]", "at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:103) ~[opensearch-2.8.0.jar:2.8.0]", "Caused by: java.lang.IllegalStateException: node does not have the data and cluster_manager roles but has index metadata: [/usr/share/opensearch/data/nodes/0/indices/1E4WoMDsTEGj_Ce4qIm0bg/_state, /usr/share/opensearch/data/nodes/0/indices/aGUuCYRzStGtsUGbJrlrZQ/_state, /usr/share/opensearch/data/nodes/0/indices/mJT2Jq_qSFW6DgsfcHKIaA/_state, /usr/share/opensearch/data/nodes/0/indices/i9wChBVBTE2i1msQH1Q6Qw/_state]. Use 'opensearch-node repurpose' tool to clean up", "at org.opensearch.env.NodeEnvironment.ensureNoIndexMetadata(NodeEnvironment.java:1198) ~[opensearch-2.8.0.jar:2.8.0]", "at org.opensearch.env.NodeEnvironment.(NodeEnvironment.java:374) ~[opensearch-2.8.0.jar:2.8.0]", "at org.opensearch.node.Node.(Node.java:498) ~[opensearch-2.8.0.jar:2.8.0]", "at org.opensearch.node.Node.(Node.java:381) ~[opensearch-2.8.0.jar:2.8.0]", "at org.opensearch.bootstrap.Bootstrap$5.(Bootstrap.java:242) ~[opensearch-2.8.0.jar:2.8.0]", "at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.8.0.jar:2.8.0]", "at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.8.0.jar:2.8.0]", "at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180) ~[opensearch-2.8.0.jar:2.8.0]", "... 6 more"] }

To Reproduce Steps to reproduce the behavior: with values.yml have this: roles:

• master: "true"
• ingest: "false"
• data: "false"
• remote_cluster_client: "false" or roles:
  • cluster_manager Expected behavior stateful set logs show an error

Chart Name OpenSearch chart in this repo

Screenshots If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

• Helm Version: [e.g. 3.12.0]
• Kubernetes Version: [e.g. 1.27.0]

Additional context Add any other context about the problem here.

[smlx]

Looks like you have re-used a volume in kubernetes. Try deleting the entire namespace and reinstalling.

[ibrahimjelliti]

Yeah thats right I have storage class with retain policy i deleted the pvc and works now