Anonymous token loses policies during big upgrades

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request. Searching for pre-existing feature requests helps us consolidate datapoints for identical requirements into a single place, thank you!
Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.
If you are interested in working on this issue or have submitted a pull request, please leave a comment.

Overview of the Issue

An upgrade of Consul-k8s 0.26.0 to 1.1.6 in my primary datacenter caused the anonymous token to lose the custom policy we had linked to it (in our case called Anonymous). After the upgrade, the only token policy linked to the anonymous token was the anonymous-token-policy created by the server-acl-init process. This caused an outage for certain customers of ours because our tooling relies on certain anonymous privileges for KV reads that we had granted to the anonymous token via our policy.

Upgrades to a deployment that already contain an anonymous-token-policy will skip altering the token policies as of consul-k8s 1.1.4 thanks to an existence check added in https://github.com/hashicorp/consul-k8s/pull/2790. Based on the code in https://github.com/hashicorp/consul-k8s/blob/2feff9f2cb36f4ee818874c4d657ede2acbc074a/control-plane/subcommand/server-acl-init/anonymous_token.go#L49 any policies linked to the Anonymous Token will not be persisted through an upgrade, replaced only with the managed policy if the managed token policy does not already exist.

I believe this is undesirable behavior because user configuration data is thrown away (the linked policies they had configured prior to upgrade). Note that the policies themselves will still exist, and re-linking them is trival to accomplish, but it requires manual intervention.

Reproduction Steps

Upgrade consul-k8s from a very old version (something <~0.49.0) to a new version (such as >=1.1.6) in a primary datacenter.

Logs

Logs from the run of my server-acl-init are not available.

Expected behavior

Any user-defined token policies that are linked to well-known tokens (specifically the anonymous token) should remain linked through an upgrade.

Environment details

Old Consul-K8s version: 0.26.0 New Consul-K8s version: 1.1.6 Kubernetes version: 1.27.3 Consul Server version: 1.15.2-ent Relevant values:

global:
  acls:
    manageSystemACLs: true

Additional Context

In the 1.2.1 breaking changes there is mention that all policies managed by consul-k8s will now be updated on upgrade. This is not true after the implementation of https://github.com/hashicorp/consul-k8s/pull/2790. An existing anonymous-token-policy will not be updated on upgrade. Notes in the documentation should reflect this.

Also the GH-2790 improvement notes in the changelog do not appear in any 1.1.x minor release. They exist only for 1.0.10 and 1.2.1, neither of which are minor versions that would have applied to me. 1.1.4 appears to be the first version with the backported change. Knowing there was a change in behavior with 1.1.4+ would have likely led to quicker resolution of my original problem.

hashicorp / consul-k8s