search

hashicorp / nomad

Nomad is an easy-to-use, flexible, and performant workload orchestrator that can deploy a mix of microservice, batch, containerized, and non-containerized applications. Nomad is easy to operate and scale and has native Consul and Vault integrations.
https://www.nomadproject.io/
Other
14.84k stars 1.95k forks source link

Using the new "spread" stanza doesn't work (0.9.0) #5578

Closed zonnie closed 5 years ago

zonnie commented 5 years ago

Nomad version

Nomad v0.9.0 (18dd59056ee1d7b2df51256fe900a98460d3d6b9)

Operating system and Environment details

Linux prod_1 3.16.0-77-generic #99~14.04.1-Ubuntu SMP Tue Jun 28 19:17:10 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

Issue

Unable to use the new spread stanza. I tried adding it in all levels (job, group and task) and it seems to have no effect. To verify I use the nomad inspect command and get Spreads: null in return 😢 Am I missing something ?

Reproduction steps

  1. save job below in a.txt
  2. Run curl -XPOST http://localhost:4646/v1/jobs -H "Content-Type: application/json" -d @a.txt
  3. Run gc-nomad job inspect worker1 | grep Spread -C 2 and get: 
        "Region": "global",
    "Reschedule": null,
    "Spreads": null,
    "Stable": false,
    "Status": "running",
--
                  "Mode": "fail"
            },
            "Spreads": null,
            "Tasks": [
                {

    I.e. seems like the spread is not effecting the job

Job file

{ 
    "job":{
        "id": "worker1",
        "name": "worker1",
        "datacenters": [
            "gc"
        ],
        "reigon": "prod",
        "type": "service",
        "priority": 1,
        "taskgroups": [
            {
                "name": "worker1",
                "spreads": [
                    {
                        "attribute": "${meta.slave}",
                        "weight": 100
                    }
                ],
                "count": 2,
                "constraints": [
                    {
                        "ltarget": "${meta.slave}",
                        "rtarget": "true",
                        "operand": "=="
                    }
                ],
                "tasks": [
                    {
                        "name": "worker1",
                        "config": {
                            "command": "/var/lib/worker/main.pyc",
                            "image": "172.16.0.1:5000/worker:97770593648268",
                            "entrypoint": null,
                            "hostname": null,
                            "ulimit": [],
                            "args": [],
                            "privileged": false,
                            "labels": [
                                {
                                    "image.name": "worker"
                                },
                                {
                                    "service.name": "worker1"
                                }
                            ],
                            "cap_add": [
                                "SYS_PTRACE"
                            ],
                            "network_mode": "host",
                            "volumes": [],
                            "logging": null
                        },
                        "resources": {
                            "cpu": 20,
                            "memorymb": 2000,
                            "networks": []
                        },
                        "services": [
                            {
                                "name": "worker1-w-${NOMAD_ALLOC_INDEX}",
                                "tags": [
                                    "worker1"
                                ],
                                "checks": []
                            }
                        ],
                        "driver": "docker",
                        "env": {
                            "service_name": "worker1"
                        },
                        "killtimeout": 15000000000
                    }
                ],
                "reschedulepolicy": {
                    "attempts": 0,
                    "interval": 0,
                    "delay": 10000000000,
                    "delayfunction": "exponential",
                    "maxdelay": 300000000000,
                    "unlimited": true
                },
                "restartpolicy": {
                    "attempts": 5,
                    "interval": 300000000000,
                    "delay": 15000000000,
                    "mode": "fail"
                }
            }
        ]
    }
}

Nomad Client logs

    2019/04/18 11:54:38.984961 [DEBUG] client: updated allocations at index 2210 (total 99) (pulled 10) (filtered 89)
    2019/04/18 11:54:38.985985 [DEBUG] client: allocs: (added 2) (removed 0) (updated 8) (ignore 89)
    2019/04/18 11:54:39.001430 [DEBUG] client: starting task runners for alloc 'e4e6c35c-af0e-8114-2af5-6d870e58df6a'
    2019/04/18 11:54:39.001846 [DEBUG] client: starting task context for 'worker1' (alloc 'e4e6c35c-af0e-8114-2af5-6d870e58df6a')
    2019/04/18 11:54:39.001890 [DEBUG] client.alloc_watcher: deadline for alloc "e4e6c35c-af0e-8114-2af5-6d870e58df6a" is at 2019-04-18 11:59:39.001839291 +0000 UTC m=+69055.264535649 (deploy=false checks=true)
    2019/04/18 11:54:39.017792 [DEBUG] client: starting task runners for alloc '596d14cc-6b1b-0748-b4e9-db06e6b7e50a'
    2019/04/18 11:54:39.018153 [INFO] client.gc: garbage collection due to number of allocations (83) is over the limit (50) skipped because no terminal allocations
    2019/04/18 11:54:39.018596 [DEBUG] client.alloc_watcher: deadline for alloc "596d14cc-6b1b-0748-b4e9-db06e6b7e50a" is at 2019-04-18 11:59:39.018536731 +0000 UTC m=+69055.281233096 (deploy=false checks=true)
    2019/04/18 11:54:39.018856 [DEBUG] client: starting task context for 'worker1' (alloc '596d14cc-6b1b-0748-b4e9-db06e6b7e50a')
2019-04-18T11:54:39.024Z [DEBUG] plugin: starting plugin: path=/usr/local/bin/nomad args="[/usr/local/bin/nomad executor {"LogFile":"/storage/nomad/alloc/e4e6c35c-af0e-8114-2af5-6d870e58df6a/worker1/executor.out","LogLevel":"DEBUG"}]"
2019-04-18T11:54:39.029Z [DEBUG] plugin: waiting for RPC address: path=/usr/local/bin/nomad
2019-04-18T11:54:39.039Z [DEBUG] plugin: starting plugin: path=/usr/local/bin/nomad args="[/usr/local/bin/nomad executor {"LogFile":"/storage/nomad/alloc/596d14cc-6b1b-0748-b4e9-db06e6b7e50a/worker1/executor.out","LogLevel":"DEBUG"}]"
2019-04-18T11:54:39.041Z [DEBUG] plugin: waiting for RPC address: path=/usr/local/bin/nomad
2019-04-18T11:54:39.131Z [DEBUG] plugin.nomad: plugin address: timestamp=2019-04-18T11:54:39.130Z network=unix address=/tmp/plugin803957093
    2019/04/18 11:54:39.144349 [DEBUG] driver.docker: Setting default logging options to syslog and unix:///tmp/plugin172446080
    2019/04/18 11:54:39.144431 [DEBUG] driver.docker: Using config for logging: {Type:syslog ConfigRaw:[] Config:map[syslog-address:unix:///tmp/plugin172446080]}
    2019/04/18 11:54:39.144455 [DEBUG] driver.docker: using 2097152000 bytes memory for worker1
    2019/04/18 11:54:39.144464 [DEBUG] driver.docker: using 20 cpu shares for worker1
    2019/04/18 11:54:39.144539 [DEBUG] driver.docker: No network interfaces are available
    2019/04/18 11:54:39.144558 [DEBUG] driver.docker: setting container startup command to: /var/lib/worker/main.pyc
    2019/04/18 11:54:39.144583 [DEBUG] driver.docker: applied labels on the container: map[service.name:worker1 image.name:worker]
    2019/04/18 11:54:39.144624 [DEBUG] driver.docker: setting container name to: worker1-e4e6c35c-af0e-8114-2af5-6d870e58df6a
2019-04-18T11:54:39.154Z [DEBUG] plugin.nomad: plugin address: timestamp=2019-04-18T11:54:39.154Z address=/tmp/plugin525984002 network=unix
    2019/04/18 11:54:39.177085 [DEBUG] driver.docker: Setting default logging options to syslog and unix:///tmp/plugin170946169
    2019/04/18 11:54:39.177425 [DEBUG] driver.docker: Using config for logging: {Type:syslog ConfigRaw:[] Config:map[syslog-address:unix:///tmp/plugin170946169]}
    2019/04/18 11:54:39.177453 [DEBUG] driver.docker: using 2097152000 bytes memory for worker1
    2019/04/18 11:54:39.177469 [DEBUG] driver.docker: using 20 cpu shares for worker1
    2019/04/18 11:54:39.177950 [DEBUG] driver.docker: No network interfaces are available
    2019/04/18 11:54:39.177974 [DEBUG] driver.docker: setting container startup command to: /var/lib/worker/main.pyc
    2019/04/18 11:54:39.178017 [DEBUG] driver.docker: applied labels on the container: map[image.name:worker service.name:worker1]
    2019/04/18 11:54:39.178176 [DEBUG] driver.docker: setting container name to: worker1-596d14cc-6b1b-0748-b4e9-db06e6b7e50a
    2019/04/18 11:54:39.186462 [DEBUG] client: updated allocations at index 2212 (total 99) (pulled 8) (filtered 91)
    2019/04/18 11:54:39.187084 [DEBUG] client: allocs: (added 0) (removed 0) (updated 8) (ignore 91)
    2019/04/18 11:54:39.189715 [INFO] client.gc: garbage collection due to number of allocations (83) is over the limit (50) skipped because no terminal allocations
    2019/04/18 11:54:39.329677 [INFO] driver.docker: created container 34bf9c1fc58038b79522efc3c2d43053d9139eabf1fc943655a4780200d68471
    2019/04/18 11:54:39.361346 [INFO] driver.docker: created container 5651ae0dfd58265f9d6e412cdd1309646d50f4509df9ae2ef28078b1b1522561
    2019/04/18 11:54:39.838854 [INFO] driver.docker: started container 34bf9c1fc58038b79522efc3c2d43053d9139eabf1fc943655a4780200d68471
    2019/04/18 11:54:39.875117 [DEBUG] client: error fetching stats of task worker1: stats collection hasn't started yet
    2019/04/18 11:54:39.886893 [DEBUG] consul.sync: registered 1 services, 0 checks; deregistered 0 services, 0 checks
    2019/04/18 11:54:39.940280 [INFO] driver.docker: started container 5651ae0dfd58265f9d6e412cdd1309646d50f4509df9ae2ef28078b1b1522561
    2019/04/18 11:54:39.953351 [DEBUG] client: error fetching stats of task worker1: stats collection hasn't started yet
    2019/04/18 11:54:39.994717 [DEBUG] consul.sync: registered 1 services, 0 checks; deregistered 0 services, 0 checks
    2019/04/18 11:54:40.001094 [DEBUG] client: updated allocations at index 2213 (total 99) (pulled 8) (filtered 91)
    2019/04/18 11:54:40.001541 [DEBUG] client: allocs: (added 0) (removed 0) (updated 8) (ignore 91)
    2019/04/18 11:54:40.012763 [INFO] client.gc: garbage collection due to number of allocations (83) is over the limit (50) skipped because no terminal allocations
    2019/04/18 11:54:40.197639 [DEBUG] client: updated allocations at index 2214 (total 99) (pulled 8) (filtered 91)
    2019/04/18 11:54:40.198182 [DEBUG] client: allocs: (added 0) (removed 0) (updated 8) (ignore 91)
    2019/04/18 11:54:40.200945 [INFO] client.gc: garbage collection due to number of allocations (83) is over the limit (50) skipped because no terminal allocations

Nomad Server logs

    2019/04/18 11:52:41.983183 [DEBUG] http: Request POST /v1/jobs (55.016824ms)
    2019/04/18 11:52:42.011522 [DEBUG] worker1: submitted plan at index 2194 for evaluation d7f628ee-4979-0891-323f-8f4fffe40ddc
    2019/04/18 11:52:42.011597 [DEBUG] sched: <Eval "d7f628ee-4979-0891-323f-8f4fffe40ddc" JobID: "worker1" Namespace: "default">: setting status to complete
    2019/04/18 11:52:42.043825 [DEBUG] worker1: updated evaluation <Eval "d7f628ee-4979-0891-323f-8f4fffe40ddc" JobID: "worker1" Namespace: "default">
    2019/04/18 11:52:42.043946 [DEBUG] worker1: ack for evaluation d7f628ee-4979-0891-323f-8f4fffe40ddc
    2019/04/18 11:52:44 [DEBUG] memberlist: TCP connection from=172.16.0.1:52726
preetapan commented 5 years ago

@zonnie Spreads can be specified at the job/task group or task level. In your json file, you have specified it at the task group level. I ran your example and checked the output of nomad inspect and I do see the Spreads attribute at the task group worker1.

Can you run nomad alloc status -verbose <allocid> on one of the allocations for this job and paste the output of the "Placement Metadata" section? That should show whether spread criteria was used in scoring placements.

zonnie commented 5 years ago

@preetapan Here you go:

ID                  = 596d14cc-6b1b-0748-b4e9-db06e6b7e50a
Eval ID             = 275ba214-83f3-c487-d22f-65d70f9a273e
Name                = worker1[1]
Node ID             = 3a58975d-39e5-42c8-f893-06ba1ab1a917
Job ID              = worker1
Job Version         = 0
Client Status       = running
Client Description  = <none>
Desired Status      = run
Desired Description = <none>
Created             = 2019-04-18T11:54:38Z
Modified            = 2019-04-18T11:54:50Z
Evaluated Nodes     = 1
Filtered Nodes      = 0
Exhausted Nodes     = 0
Allocation Time     = 177.06µs
Failures            = 0

Task "worker1" is "running"
Task Resources
CPU        Memory          Disk     Addresses
49/20 MHz  76 MiB/2.0 GiB  300 MiB

Task Events:
Started At     = 2019-04-18T11:54:39Z
Finished At    = N/A
Total Restarts = 0
Last Restart   = N/A

Recent Events:
Time                  Type        Description
2019-04-18T11:54:39Z  Started     Task started by client
2019-04-18T11:54:39Z  Task Setup  Building Task Directory
2019-04-18T11:54:39Z  Received    Task received by client

Placement Metrics
  * Score "3a58975d-39e5-42c8-f893-06ba1ab1a917.binpack" = 2.074423
  * Score "3a58975d-39e5-42c8-f893-06ba1ab1a917.job-anti-affinity" = -20.000000

Also double checked and my Spread is null on both job and task group levels.

preetapan commented 5 years ago

@zonnie That looks like you are running a 0.8.7 CLI against a 0.9 server? When I ran your example I used 0.9 everywhere. The placement metrics in the CLI output should look like this example, with a column for spread score.

zonnie commented 5 years ago

@preetapan Is there a way to check the version of the client itself ? Even though that is very weird because I use the same binary and running nomad version gets this:

Nomad v0.9.0 (18dd59056ee1d7b2df51256fe900a98460d3d6b9)
preetapan commented 5 years ago

Can you run curl on http;//127.0.0.1:4646/v1/agent/self" and report the Version section. It should look something like: 

"Version": {
      "Revision": "",
      "Version": "0.9.0",
      "VersionMetadata": "",
      "VersionPrerelease": ""
    }

Also check what nomad server members returns to confirm that your nomad server is also on 0.9.

I tried your example again on the official binaries downloaded from releases.hashicorp.com and I still see the Spreads attribute defined at the taskgroup level

zonnie commented 5 years ago

@preetapan - You are correct, not sure how...but that's on me:

        "Version": {
            "Revision": "21a2d93eecf018ad2209a5eab6aae6c359267933+CHANGES",
            "Version": "0.8.7",
            "VersionMetadata": "",
            "VersionPrerelease": ""
        }

Thanks a lot ! PS - Nomad is awesome (coming from someone who spent a painful vacation in Mesos/Marathon land)

preetapan commented 5 years ago

You are welcome and thanks for using Nomad!

Another thing I wanted to point out that I noticed, you have a constraint on meta.slave=true, which means only nodes that have that set to true are eligible for placement. You also have a spread stanza on the same attribute. The spread stanza does useful things only when you have more than one unique value for the attribute (e.g datacenter or rack). In your case, because you already have a constraint that limits the set of nodes, the distinct_hosts constraint is likely what you need rather than spread.

See https://www.nomadproject.io/docs/job-specification/constraint.html#quot-distinct_hosts-quot- for more info.

github-actions[bot] commented 1 year ago

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.