search

hashicorp / nomad

Nomad is an easy-to-use, flexible, and performant workload orchestrator that can deploy a mix of microservice, batch, containerized, and non-containerized applications. Nomad is easy to operate and scale and has native Consul and Vault integrations.
https://www.nomadproject.io/
Other
14.85k stars 1.95k forks source link

UI doesn't report reserved resources correctly (cpu and memory) #8694

Open scalp42 opened 4 years ago

scalp42 commented 4 years ago

Hi folks,

Related to our mistake in #8689, after fixing the memory reserved to be 50% of the node, the client in the UI will still report the full memory and CPU.

The VM has 985 MiB of memory, using the following reserved params:

    "reserved": {
      "memory": 493,
      "cpu": 259
    },

Nomad UI:

Screen Shot 2020-08-19 at 11 08 57

Trying to schedule a job with 512 MiB of memory fails as expected (so we got the reserved logic right 🤟 ):

      ==> Evaluation "f4037bba" finished with status "complete" but failed to place all allocations:
           Task Group "httpbin" (failed to place 1 allocation):
             * Resources exhausted on 1 nodes
             * Dimension "memory" exhausted on 1 nodes
           Evaluation "19e8bfe9" waiting for additional capacity to place remainder

@jrasell I'm fairly confident this is misleading.

Let us know, cheers!

MorphBonehunter commented 3 years ago

While i want open an enhancement issue for the UI, i stumble across this issue and it seem that this one is similar to my thoughts. So as @scalp42 already mentiond, at the moment the "reserved" in the node view is in my eyes what the cli shows under the Host Resource Utilization point (ex.):

nomad node status -stats 767e701c-d576-834a-8d50-eb94ed657843
...
Allocated Resources
CPU            Memory           Disk
2128/2094 MHz  800 MiB/947 MiB  1.8 GiB/2.4 GiB

Allocation Resource Utilization
CPU          Memory
90/2094 MHz  215 MiB/947 MiB

Host Resource Utilization
CPU           Memory           Disk
563/2194 MHz  556 MiB/980 MiB  (/dev/mapper/vg0-root)
...

So i would guess what it should be display ist the Allocation Resource Utilization (it's only the node view, the allocation view is fine). So with this it would shows the actual use and the max. useable stuff.

In contrast to the mentioned "total amount available" (https://github.com/hashicorp/nomad/pull/8696#issuecomment-676783600) in @backspace PR #8696, i would think it is just a nice to have what the system ressources are as nomad was told to use only an smaller ammount and so this is the key fact. This is also an point for the new topology view, as here also the Host Resource Utilization is shown, not the Allocation Resource Utilization.

But what would be an good idea i think, is the Allocated Resources stuff, maybe that could be inserted as an horizontal bar in both graphs? In the topology view this could be displayed if someone hovers over the "free" / grey bars which is diplayed after the green bars.

If it is unclear what i mean, i can try to take some screenshots and draw my thought ;)

DingoEatingFuzz commented 3 years ago

Hi @scalp42 I believe #10208 clears up these charts as they relate to reserved numbers.

  1. Show totals in the figures under the chart
  2. Chart is still total host utilization (with a tooltip to better explain that)
  3. Reserved numbers are shown in the chart as annotations to quickly see what proportion of the total is reserved for host processes.