Ingress Gateway init lack permissions 'service:write'

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request. Searching for pre-existing feature requests helps us consolidate datapoints for identical requirements into a single place, thank you!
Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.
If you are interested in working on this issue or have submitted a pull request, please leave a comment.

Overview of the Issue

Hello, I have encountered an issue with starting up ingress-gateway services in my consul client cluster that was deployed in the AWS EKS.

Reproduction Steps

The issue was managed to reproduce on the consul helm chart version starting from v0.42 - v0.44. Below was the config.yaml that I used for consul helm installation.

global:
  name: dc1
  enabled: false
  datacenter: igw-test
  acls:
    manageSystemACLs: true
    bootstrapToken:
      secretName: consul-bootstrap-token
      secretKey: token
  gossipEncryption:
    secretName: consul-gossip-key
    secretKey: key
  tls:
    enabled: true
    enableAutoEncrypt: true
    caCert:
      secretName: consul-ca-cert
      secretKey: tls.crt
externalServers:
  enabled: true
  hosts: ["${HOST}"]
  httpsPort: 443
  useSystemRoots: true
  k8sAuthMethodHost: ${AUTH}

client:
  enabled: true
  join: ["${HOST}"]

connectInject:
  enabled: true
  default: false
  transparentProxy:
    defaultEnabled: false
controller:
  enabled: true
ui:
  enabled: true
dns:
  enabled: true

ingressGateways:
  enabled: true
  defaults:
    replicas: 1
  gateways:
    - name: igw-1
      service:
        ports:
          - port: 9090

I did manage to resolve it if I specify the use of consul enterprise image (hashicorp/consul-enterprise:1.12.0-ent) in the config.yaml. Otherwise, the startup of ingress gateway will have this issue. It is possible that there are some problems on the default specified consul image in the helm chart.

Hope someone could have a look. Thanks.

Logs

Below shows the logs captured from the ingress-gateway-init

2022-06-15T07:47:41.420Z [INFO]  Consul login complete
2022-06-15T07:47:41.420Z [INFO]  Checking that the ACL token exists when reading it in the stale consistency mode
2022-06-15T07:47:41.476Z [INFO]  Successfully read ACL token from the server
2022-06-15T07:47:41.477Z [INFO]  Successfully read ACL token from the server
Address "172.20.195.90" written to /tmp/address.txt successfully
Error registering service "igw-1": Unexpected response code: 403 (Permission denied: token with AccessorID 'ad71aa34-4f19-7a67-91e5-24f5beaefe5a' lacks permission 'service:write' on "igw-1")

hashicorp / consul-k8s