Wrong allocation report when launch nomad status

[tantra35]

Nomad version

Nomad v0.6.0-dev (b5f687df1dfc71f484c9bdfe51190ec124b3d3c3)

Issue

For one of jobs we see that it doiens't have any allocation:

root@vol-h-docker-01:/home/ruslan# nomad status teamcity
ID            = teamcity
Name          = teamcity
Type          = service
Priority      = 80
Datacenters   = test
Status        = running
Periodic      = false
Parameterized = false

Summary
Task Group  Queued  Starting  Running  Failed  Complete  Lost
teamcity    0       0         0        0       0         0

Allocations
No allocations placed

but allocations exists and present on some of our nodes(for this case this is node with id a6026d01(vol-h-docker-02))

root@vol-h-docker-01:/home/ruslan# nomad node-status
ID        DC    Name             Class   Drain  Status
04eb99a2  test  rbdmonitor       ceph    true   ready
e9981b62  ptz   ptz-v-docker-01  branch  false  ready
de4cfe25  test  vol-v-docker-03  ceph    false  ready
a6026d01  test  vol-h-docker-02  ceph    false  ready
439a2f5a  test  graphite         ceph    false  ready
41b521c8  test  vol-h-docker-01  ceph    false  ready
ec475f0a  test  social           ceph    false  ready

and we can see that when launch nomad node-status a6026d01

root@vol-h-docker-01:/home/ruslan# nomad node-status a6026d01
ID      = a6026d01
Name    = vol-h-docker-02
Class   = ceph
DC      = test
Drain   = false
Status  = ready
Drivers = docker,exec,raw_exec
Uptime  = 798h27m51s

Allocated Resources
CPU              Memory         Disk             IOPS
13400/13488 MHz  25 GiB/31 GiB  6.7 GiB/849 GiB  0/0

Allocation Resource Utilization
CPU             Memory
2060/13488 MHz  9.2 GiB/31 GiB

Host Resource Utilization
CPU             Memory         Disk
8209/14000 MHz  10 GiB/31 GiB  21 GiB/917 GiB

Allocations
ID        Eval ID   Job ID                       Task Group                   Desired Status  Client Status
7f610326  ac93d39c  zooMMMDynamo                 zooMMMDynamo                 run             running
72f1eafd  307b4657  skycharmsDynamo              skycharmsDynamo              run             running
caa0d94a  da2bb995  gardenscapesMacDynamo        gardenscapesMacDynamo        run             running
60ab8d39  18c6572f  gardenscapesFBEntDynamoNode  gardenscapesFBEntDynamoNode  run             running
988a9d0a  b92b2535  gardenscapesFBDynamoNode     gardenscapesFBDynamoNode     run             running
a78b49eb  b1a4a55f  fishdomDDMac                 fishdomDDMac                 run             running
c053d6a4  a18e1d6e  fishdomDDDynamoNode          fishdomDDDynamoNode          run             running
96ed6a9a  5f6775f0  townshipMacDynamoNode        townshipMacDynamoNode        run             running
a4410703  dacd4bef  townshipDynamoNode           townshipDynamoNode           run             running
7b2d6617  4afeb057  webmanage                    webmanage                    run             running
089fe159  33d2660b  githubproxy                  githubproxy                  run             running
d628f667  6cb451df  webmysql                     webmysql                     run             running
828e86aa  8d46e5f3  cephmon                      cephmon                      run             running
82638f74  2d2c1e25  redis                        redis                        run             running
240ca78e  185a733a  gardenscapesFBPhp            gardenscapesFBPhp            run             running
5c2fb6b6  fce34448  dynamoDBLocal                dynamoDBLocal                run             running
4e3966ad  54d96e62  radosgw                      radosgw                      run             running
fe9ee9b3  7f10a9ed  CheckTeamcityTasks           CheckTeamcityTasks           run             running
59c82cdb  08e74189  svn                          svn                          run             running
8a1e1ef3  18c1ad1b  ElasticSearchCurator         escurator                    run             running
4c846385  455f3dae  teamcity-mysql               teamcity-mysql               run             running
d7fb35a4  a54be6e6  cassandraLocal               cassandraLocal               run             running
93587b10  2631f2d1  teamcity                     teamcity                     run             running

As you can see allocation for job teamcity exists with id 93587b10

root@vol-h-docker-01:/home/ruslan# nomad alloc-status 93587b10
ID                  = 93587b10
Eval ID             = 2631f2d1
Name                = teamcity.teamcity[0]
Node ID             = a6026d01
Job ID              = teamcity
Client Status       = running
Client Description  = <none>
Desired Status      = run
Desired Description = <none>
Created At          = 05/12/17 16:12:33 MSK

Task "teamcity" is "running"
Task Resources
CPU            Memory           Disk     IOPS  Addresses
1249/4000 MHz  3.2 GiB/3.9 GiB  300 MiB  0     https: 172.16.9.199:8443

Recent Events:
Time                   Type                   Description
05/12/17 16:14:16 MSK  Started                Task started by client
05/12/17 16:12:39 MSK  Downloading Artifacts  Client is downloading artifacts
05/12/17 16:12:39 MSK  Task Setup             Building Task Directory
05/12/17 16:12:39 MSK  Received               Task received by client

restart of nomad service on vol-h-docker-02(a6026d01) node which placed "hidden" allocations does nothing

[dadgar]

Please run nomad status -all-allocs teamcity. The status only shows allocs by default for the currently launched job, not older versions. You probably ran, stop and then re-ran.

[github-actions[bot]]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.