search

hashicorp / nomad

Nomad is an easy-to-use, flexible, and performant workload orchestrator that can deploy a mix of microservice, batch, containerized, and non-containerized applications. Nomad is easy to operate and scale and has native Consul and Vault integrations.
https://www.nomadproject.io/
Other
14.91k stars 1.95k forks source link

Docker IPAM IP address resource utilization #2756

Open rickardrosen opened 7 years ago

rickardrosen commented 7 years ago

Using the docker MACVLAN driver we hand over IP addressing of containers to the built-in IPAM driver of docker.

Nomad have no notion of these addresses in the form of resources utilisation which means we can end up in a scenario where there are no available IP-addresses to assign on a host, but Nomad insist on allocation placement on this nomad agent.

Due to the bin packing nature of Nomad we are unable to successfully schedule containers at all in a cluster with many available IP addresses.

There are a few solutions to this, which all requires enhancements to Nomad:

  1. Ideally Nomad should be able to include the number of available IP addresses in the evaluation process for tasks using docker driver when docker IPAM is configured.
  2. An option to use a spread scheduling algorithm.
  3. Set a maximum number of tasks for an agent.
  4. The ability to disable a node i.e a "nomad disable" command option that excludes this host from receiving further allocations.

Are any of these features in the works or are they perhaps already available and I'm just being ignorant? =)

schmichael commented 7 years ago

Since the network bandwidth resource isn't enforced you can use it to limit the number of tasks on a node. For example if you have 1000Mbps NICs give each task mbits = 100 to limit the node to having 10 tasks. If you have a heterogenous set of NICs you could override their speeds in the client configs to make the cluster appear homogenous: https://www.nomadproject.io/docs/agent/configuration/client.html#network_speed

Obviously this is just a hacky workaround while you wait on official support (I doubt even a better workaround will make it into 0.6).

rickardrosen commented 7 years ago

@schmichael that is actually the kind of hacky workaround I was looking for, but could not figure out during the day! A workaround that might just work for the time being... Thank you!

I believe suggestions 2-4 would be great general improvements, while the first perhaps will be redundant with 0.6.0 depending on how we can handle the IP assignments.

Again, thank you.

maxramqvist commented 7 years ago

@schmichael what would be the easiest way of getting the allocated network speed from each node? We'd like to graph and measure the stats. I can't find any place for it thorugh CLI or API.

schmichael commented 7 years ago

@maxramqvist You must enable telemtry.publish_node_metrics: https://www.nomadproject.io/docs/agent/configuration/telemetry.html#publish_node_metrics