[BUG] Updating Security config in a mixed cluster can result in Unrecognized field "<new_security_config_value>"

[cwperks]

What is the bug?

A new security config value on_behalf_of was introduced in OpenSearch 2.12.0. When performing a rolling upgrade from 2.11 -> 2.12 the cluster will be in a mixed state until all nodes in the old version have been rebooted. While in a mixed state if a security config update request is performed against a 2.12 node, it will add the new security config values to the security index and fan out an internal transport config update request to instruct all nodes to re-read the security configuration from the index. Below is an example of a config update request to enable HTTP Basic auth that encounters this error:

➜  latest curl -XPATCH "https://admin:admin@localhost:9201/_plugins/_security/api/securityconfig" -k -H 'Content-Type: application/json' -d'[
{
    "op": "replace",
    "path": "/config/dynamic/authc/basic_internal_auth_domain/http_enabled",
    "value": "true"
  }
]'
{"status":"INTERNAL_SERVER_ERROR","message":"Error Failed node [mJV1aP-QSEiHzkFxe_CQEA]"}

The OpenSearch logs show:

opensearch-node1-1       | com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "on_behalf_of" (class org.opensearch.security.securityconf.impl.v7.ConfigV7$Dynamic), not marked as ignorable (15 known properties: "license", "disable_intertransport_auth", "http", "kibana", "authz", "auth_failure_listeners", "transport_userrname_attribute", "filtered_alias_mode", "authc", "disable_rest_auth", "respect_request_indices_options", "multi_rolespan_enabled", "do_not_fail_on_forbidden", "hosts_resolver_mode", "do_not_fail_on_forbidden_empty"])

What is the expected behavior?

The security config update should be successful regardless of which node receives the request.

Do you have any additional context?

The request to update the security config does not need to actually change the config. Any security config request will encounter this error. If HTTP Basic auth is already enabled and you submit a request to enable HTTP Basic auth, it will encounter this error.

There is an opportunity for an optimization to prevent fanning out an internal transport config update if nothing in the security index actually changes.