ACLs filter allocation resource usage on Node API

[kobzonera]

If filing a bug please include the following:

Nomad version

Nomad v1.0.0 (cfca6405ad9b5f66dffc8843e3d16f92f3bedb43)

Operating system and Environment details

Operating system:

• Architecture.......: "x86_64"
• CPU Cores..........: "8"
• Memory.............: "16046" Mb
• OS.................: "Ubuntu"
• Release............: "16.04"
• Kernel.............: "4.4.0-131-generic"

Nomad ACL Enabled.

Issue

API Call /v1/node/:node_id/allocations return empty list even allocations presented on node

Reproduction steps

Simple curl request on node with allocations curl -v --header "X-Nomad-Token: <token>" "http://<ip>:4646/v1/node/<node_id>/allocations" return empty list

< HTTP/1.1 200 OK
< Content-Type: application/json
< Vary: Accept-Encoding
< X-Nomad-Index: 159510
< X-Nomad-Knownleader: true
< X-Nomad-Lastcontact: 0
< Date: Wed, 16 Dec 2020 21:56:13 GMT
< Content-Length: 2
< 
[]

Second request with index param curl --header "X-Nomad-Token: <token>" "http://<ip>:4646/v1/node/<node_id>/allocations?index=159510" return empty list after wait about minute.

> GET /v1/node/<node_id>/allocations?index=159510 HTTP/1.1
> User-Agent: curl/7.47.0
> Accept: */*
> X-Nomad-Token: <token>
> 
< HTTP/1.1 200 OK
< Content-Type: application/json
< Vary: Accept-Encoding
< X-Nomad-Index: 159510
< X-Nomad-Knownleader: true
< X-Nomad-Lastcontact: 0
< Date: Wed, 16 Dec 2020 22:25:44 GMT
< Content-Length: 2
< 
[]

Token has policies described in docs

[tgross]

Hi @kobzonera I just checked this locally with the following read-only policy and wasn't able to verify the behavior:

namespace "*" {
  policy       = "read"
  capabilities = ["alloc-node-exec"]
}

agent {
  policy = "read"
}

operator {
  policy = "read"
}

quota {
  policy = "read"
}

node {
  policy = "read"
}

host_volume "*" {
  policy = "read"
}

Can you share the redacted token info and policy document associated with the ACL token you're using?

nomad acl policy info <policy-name>
nomad acl token self

[kobzonera]

Ow, it seems my policies wasn't complete and accurate. Definitely I forgot about namespace policies as don't use this feature. In my opinion policy rule with node read should have been enough.

node {
  policy = "read"
}

So why response code was 200 (not 403 forbidden)? Is it correct logic with necessarily namespace acl policy even though there not uses?

[tgross]

So why response code was 200 (not 403 forbidden)?

A node can have allocations from multiple namespaces. So it's filtering out the allocations that you don't have access to and returning the list on ones you do have access to. (In this case, none.)

Imagine if you had these three allocations on the same node:

• allocation ab12cd in namespace development
• allocation 56ef78 in namespace development
• allocation dc34ba in namespace production

If your ACL only gives you access to the development namespace, then we would return the allocations ab12cd and 56ef78. If we were to return a 403 then you wouldn't be able to view the allocations you're supposed to have access to.

[tgross]

An unfortunate side-effect of this problem is that nomad node status does not display the total of allocated resources (CPU, Memory, Disk) at the node level when an accessor doesn't have the correct ACLs to see all namespaces on a box. This is because we have to hit this same endpoint to find out which allocations are running, and that happens entirely client-side in the CLI (ref command/node_status.go#L782-L795)

[tgross]

Related: https://github.com/hashicorp/nomad/issues/9899