Service identities from roles are ignored when checking Consul ACL's

[sorenisanerd]

I think a bit of Terraform demonstrates the problem best:

# mysvc.tf

resource "consul_acl_token" "mysvc" {
    roles = [consul_acl_role.mysvc.name]
}

resource "consul_acl_role" "mysvc" {
    name = "mysvc"
    service_identities {
        service_name = "mysvc"
    }
}

resource "nomad_job" "mysvc" {
    jobspec = file("${path.module}/mysvc.nomad")
    consul_token = data.consul_acl_token_secret_id.mysvc.secret_id
}

data "consul_acl_token_secret_id" "mysvc" {
    accessor_id = consul_acl_token.mysvc.id
}

# mysvc.nomad
job "mysvc" {
  group "mysvc" {
    network {
      mode = "bridge"
      port "http" {}
    }
    service {
      provider = "consul"
      name     = "mysvc"
      port     = "http"
    }

    task "mysvc" {
      [...]
    }
  }
}

Attempting to apply that Terraform code yields:

Error submitting job: Unexpected response code: 500 (rpc error: job-submitter consul token denied: insufficient Consul ACL permissions to write service "mysvc")

nomad.consulACLsAPI.canWriteService goes through:

• service identities in the token: https://github.com/hashicorp/nomad/blob/a8e68e64793670f0d19ae5659a8b78606b24aa27/nomad/consul_policy.go#L168-L172
• service or service_prefix policies in the token: https://github.com/hashicorp/nomad/blob/a8e68e64793670f0d19ae5659a8b78606b24aa27/nomad/consul_policy.go#L175-L181
• and finally service or service_prefix policies in the roles assigned to the token: https://github.com/hashicorp/nomad/blob/a8e68e64793670f0d19ae5659a8b78606b24aa27/nomad/consul_policy.go#L184-L200

Notably absent is a check for service identities in the roles assigned to the token.

[tgross]

Hi @sorenisanerd I agree this seems straightforwardly a bug. We're also in the middle of deprecating this whole process for Nomad 1.7.0 so that we can use Workload Identity for jobs that need service identities, and this code will be removed outright in Nomad 1.9.0.

That being said, I don't love leaving a known bug around until 1.9.0. I'll mark this for roadmapping but it's also likely a small enough fix that we could take a community contribution, so I'll mark it as such for that as well.

[sorenisanerd]

I've moved entirely to the workload identity approach, so this does not affect me anymore. As such, I'm not investing more effort here.