When we are upgraded our cluster from 2.15 to 2.17, we observed that multiple system indices were having different IndexMetadata on Cluster Manager node and remaining follower nodes. Where system flag is true on cluster manager but on followers it is false with same version.
This breaks the invariance guarantee of IndexMetadata being same as cluster manager on all nodes.
This has started happening because we have marked various index as system index as part of https://github.com/opensearch-project/security/issues/4439
Related component
Plugins
To Reproduce
Create a multinode 2.15 cluster with plugins such as observability, which create a system index.
Upgrade the cluster to 2.16/2.17
Active Cluster Manager have system flag true for .opensearch-observability index metadata, but all follower show false
Expected behavior
Cluster State Index Metadata should be same for all the nodes.
Additional Details
This is happening in both the causes, with remote_store.publication.enabled: true/false
Describe the bug
When we are upgraded our cluster from 2.15 to 2.17, we observed that multiple system indices were having different IndexMetadata on Cluster Manager node and remaining follower nodes. Where system flag is
trueon cluster manager but on followers it isfalsewith same version. This breaks the invariance guarantee of IndexMetadata being same as cluster manager on all nodes. This has started happening because we have marked various index as system index as part of https://github.com/opensearch-project/security/issues/4439Related component
Plugins
To Reproduce
truefor.opensearch-observabilityindex metadata, but all follower showfalseExpected behavior
Cluster State Index Metadata should be same for all the nodes.
Additional Details
This is happening in both the causes, with
remote_store.publication.enabled:true/false