search

SUSE / teuthology

Ceph test suite
MIT License
5 stars 4 forks source link

ceph_cm_salt: pinging by "job_id" grain not working as expected #89

Open smithfarm opened 7 years ago

smithfarm commented 7 years ago

The idea behind pinging minions matching a certain "job_id" was to only ping those minions that are running the current job (test). Surprisingly, this is picking up other minions at the same time:

2017-06-21T22:01:28.954 INFO:teuthology.orchestra.run.target167114236223:Running: 'sudo sh -c \'salt -C "G@job_id:61" test.ping || true\''
2017-06-21T22:01:29.456 INFO:teuthology.orchestra.run.target167114236223.stderr:[WARNING ] Key 'file_ignore_glob' with value None has an invalid type of NoneType, a list is required for this value
2017-06-21T22:01:29.456 INFO:teuthology.orchestra.run.target167114236223.stderr:[WARNING ] Key 'file_ignore_glob' with value None has an invalid type of NoneType, a list is required for this value
2017-06-21T22:01:29.535 INFO:teuthology.orchestra.run.target167114236223.stderr:[WARNING ] Key 'file_ignore_glob' with value None has an invalid type of NoneType, a list is required for this value
2017-06-21T22:01:29.536 INFO:teuthology.orchestra.run.target167114236223.stderr:[WARNING ] Key 'file_ignore_glob' with value None has an invalid type of NoneType, a list is required for this value
2017-06-21T22:01:49.036 INFO:teuthology.orchestra.run.target167114236223.stdout:target167114225180.teuthology:
2017-06-21T22:01:49.037 INFO:teuthology.orchestra.run.target167114236223.stdout:    True
2017-06-21T22:01:49.037 INFO:teuthology.orchestra.run.target167114236223.stdout:target167114243175.teuthology:
2017-06-21T22:01:49.037 INFO:teuthology.orchestra.run.target167114236223.stdout:    Minion did not return. [Not connected]
2017-06-21T22:01:49.038 INFO:teuthology.orchestra.run.target167114236223.stdout:target167114253140.teuthology:
2017-06-21T22:01:49.038 INFO:teuthology.orchestra.run.target167114236223.stdout:    Minion did not return. [Not connected]
2017-06-21T22:01:49.038 INFO:teuthology.orchestra.run.target167114236223.stdout:target167114250038.teuthology:
2017-06-21T22:01:49.038 INFO:teuthology.orchestra.run.target167114236223.stdout:    Minion did not return. [Not connected]
2017-06-21T22:01:49.038 INFO:teuthology.orchestra.run.target167114236223.stdout:target167114243177.teuthology:
2017-06-21T22:01:49.038 INFO:teuthology.orchestra.run.target167114236223.stdout:    Minion did not return. [Not connected]
2017-06-21T22:01:49.038 INFO:teuthology.orchestra.run.target167114236223.stdout:target167114243010.teuthology:
2017-06-21T22:01:49.039 INFO:teuthology.orchestra.run.target167114236223.stdout:    Minion did not return. [Not connected]
2017-06-21T22:01:49.039 INFO:teuthology.orchestra.run.target167114236223.stdout:target167114250008.teuthology:
2017-06-21T22:01:49.039 INFO:teuthology.orchestra.run.target167114236223.stdout:    Minion did not return. [Not connected]
2017-06-21T22:01:49.039 INFO:teuthology.orchestra.run.target167114236223.stdout:target167114253162.teuthology:
2017-06-21T22:01:49.039 INFO:teuthology.orchestra.run.target167114236223.stdout:    Minion did not return. [Not connected]
2017-06-21T22:01:49.039 INFO:teuthology.orchestra.run.target167114236223.stdout:target167114253139.teuthology:
2017-06-21T22:01:49.039 INFO:teuthology.orchestra.run.target167114236223.stdout:    Minion did not return. [Not connected]
2017-06-21T22:01:49.041 DEBUG:teuthology.salt:1 minion(s) responded

There should only be one minion with job_id:61 - not sure what is going on here.

jan--f commented 7 years ago

I think what's going on is that the salt master needs to contact all minions to query for their job_id grain. It's just logging that it couldn't reach some minions.

A fix would be to clean up minions on the master after they stop existing.

smithfarm commented 7 years ago

It does seem harmless. I've been running 6 tests at a time with no issues. So it's just log spam.