[SES5] Check that all OSDs are "in" during restart

[swiftgist]

Signed-off-by: Eric Jackson swiftgist@gmail.com bnc:#1153310

Checklist:

• [ ] Added unittests and or functional tests
• [ ] Adapted documentation
• [x] Referenced issues or internal bugtracker
• [ ] Ran integration tests successfully (trigger with "@susebot run teuthology" in a GitHub comment; see the wiki for more information)

[swiftgist]

@susebot run teuthology

[susebot]

Commit d065329db297430e08e3e944bdf4936c12460221 is NOT OK for suite deepsea:tier2. Check tests results in the Jenkins job: http://ci.ses.suse.de:8080/job/pr-deepsea/327/

[swiftgist]

Running deepsea stage run ceph.stage.0 for me is working... not sure why the failure in teuthology

[swiftgist]

@susebot run teuthology

[susebot]

Commit c68c32df93c7bdbd60d7032d12754e226d840e46 is NOT OK for suite deepsea:tier2. Check tests results in the Jenkins job: http://ci.ses.suse.de:8080/job/pr-deepsea/332/

[jschmid1]

That test failure looks like a genuine issue (however probably not connected to this PR, atleast I don't see how)

2019-10-10T20:58:09.693 INFO:teuthology.orchestra.run.target192168000101:Running: "sudo bash -c 'DEV_ENV=true timeout 60m deepsea --log-file=/var/log/salt/deepsea.log --log-level=debug salt-run state.orch ceph.stage.3 --simple-output'"
2019-10-10T20:58:10.133 INFO:teuthology.orchestra.run.target192168000101.stdout:Starting stage: ceph.stage.3
2019-10-10T20:58:37.100 INFO:teuthology.orchestra.run.target192168000101.stdout:Parsing stage ceph.stage.3 steps... fail
2019-10-10T20:58:37.101 INFO:teuthology.orchestra.run.target192168000101.stdout:
2019-10-10T20:58:37.101 INFO:teuthology.orchestra.run.target192168000101.stdout:An error occurred when rendering one of the following states:
2019-10-10T20:58:37.137 INFO:teuthology.orchestra.run.target192168000101.stderr:Traceback (most recent call last):
2019-10-10T20:58:37.138 INFO:teuthology.orchestra.run.target192168000101.stderr:  File "/usr/bin/deepsea", line 11, in <module>
2019-10-10T20:58:37.139 INFO:teuthology.orchestra.run.target192168000101.stderr:    load_entry_point('deepsea==0.8.13+2.g3f43be5b', 'console_scripts', 'deepsea')()
2019-10-10T20:58:37.139 INFO:teuthology.orchestra.run.target192168000101.stderr:  File "/usr/lib/python2.7/site-packages/deepsea/deepsea.py", line 371, in main
2019-10-10T20:58:37.140 INFO:teuthology.orchestra.run.target192168000101.stderr:    cli(prog_name='deepsea')
2019-10-10T20:58:37.140 INFO:teuthology.orchestra.run.target192168000101.stderr:  File "/usr/lib/python2.7/site-packages/click/core.py", line 722, in __call__
2019-10-10T20:58:37.140 INFO:teuthology.orchestra.run.target192168000101.stderr:    return self.main(*args, **kwargs)
2019-10-10T20:58:37.140 INFO:teuthology.orchestra.run.target192168000101.stderr:  File "/usr/lib/python2.7/site-packages/click/core.py", line 697, in main
2019-10-10T20:58:37.145 INFO:teuthology.orchestra.run.target192168000101.stderr:    rv = self.invoke(ctx)
2019-10-10T20:58:37.145 INFO:teuthology.orchestra.run.target192168000101.stderr:  File "/usr/lib/python2.7/site-packages/click/core.py", line 1066, in invoke
2019-10-10T20:58:37.145 INFO:teuthology.orchestra.run.target192168000101.stderr:    return _process_result(sub_ctx.command.invoke(sub_ctx))
2019-10-10T20:58:37.145 INFO:teuthology.orchestra.run.target192168000101.stderr:  File "/usr/lib/python2.7/site-packages/click/core.py", line 1066, in invoke
2019-10-10T20:58:37.145 INFO:teuthology.orchestra.run.target192168000101.stderr:    return _process_result(sub_ctx.command.invoke(sub_ctx))
2019-10-10T20:58:37.146 INFO:teuthology.orchestra.run.target192168000101.stderr:  File "/usr/lib/python2.7/site-packages/click/core.py", line 895, in invoke
2019-10-10T20:58:37.146 INFO:teuthology.orchestra.run.target192168000101.stderr:    return ctx.invoke(self.callback, **ctx.params)
2019-10-10T20:58:37.146 INFO:teuthology.orchestra.run.target192168000101.stderr:  File "/usr/lib/python2.7/site-packages/click/core.py", line 535, in invoke
2019-10-10T20:58:37.146 INFO:teuthology.orchestra.run.target192168000101.stderr:    return callback(*args, **kwargs)
2019-10-10T20:58:37.146 INFO:teuthology.orchestra.run.target192168000101.stderr:  File "/usr/lib/python2.7/site-packages/deepsea/common.py", line 66, in func_wraper
2019-10-10T20:58:37.146 INFO:teuthology.orchestra.run.target192168000101.stderr:    return func(*args, **kwargs)
2019-10-10T20:58:37.147 INFO:teuthology.orchestra.run.target192168000101.stderr:  File "/usr/lib/python2.7/site-packages/deepsea/deepsea.py", line 356, in state_orch
2019-10-10T20:58:37.147 INFO:teuthology.orchestra.run.target192168000101.stderr:    ret = run_stage(stage_name, hide_state_steps, hide_dynamic_steps, simple_output)
2019-10-10T20:58:37.147 INFO:teuthology.orchestra.run.target192168000101.stderr:  File "/usr/lib/python2.7/site-packages/deepsea/stage_executor.py", line 79, in run_stage
2019-10-10T20:58:37.147 INFO:teuthology.orchestra.run.target192168000101.stderr:    mon.parse_stage(stage_name)
2019-10-10T20:58:37.147 INFO:teuthology.orchestra.run.target192168000101.stderr:  File "/usr/lib/python2.7/site-packages/deepsea/monitor.py", line 538, in parse_stage
2019-10-10T20:58:37.148 INFO:teuthology.orchestra.run.target192168000101.stderr:    self._fire_event('stage_parsing_finished', None, None, ex)
2019-10-10T20:58:37.148 INFO:teuthology.orchestra.run.target192168000101.stderr:  File "/usr/lib/python2.7/site-packages/deepsea/monitor.py", line 601, in _fire_event
2019-10-10T20:58:37.148 INFO:teuthology.orchestra.run.target192168000101.stderr:    getattr(listener, event)(*args)
2019-10-10T20:58:37.148 INFO:teuthology.orchestra.run.target192168000101.stderr:  File "/usr/lib/python2.7/site-packages/deepsea/monitors/terminal_outputter.py", line 45, in stage_parsing_finished
2019-10-10T20:58:37.148 INFO:teuthology.orchestra.run.target192168000101.stderr:    for state in exception.states:
2019-10-10T20:58:37.148 INFO:teuthology.orchestra.run.target192168000101.stderr:AttributeError: 'StateRenderingException' object has no attribute 'states'

[swiftgist]

@susebot run teuthology

[swiftgist]

Checking whether this run will fail on Stage 3 again.... the original run failed on Stage 0, but works with the same basic command in my environment. If Stage 0 succeeded with the calls to restart and Stage 3 is failing, I start suspecting OOM issues.

I have added one more state.... maybe that was enough.

[susebot]

Commit c68c32df93c7bdbd60d7032d12754e226d840e46 is NOT OK for suite deepsea:tier2. Check tests results in the Jenkins job: http://ci.ses.suse.de:8080/job/pr-deepsea/334/

[swiftgist]

@susebot run teuthology

[susebot]

Commit e74e745d276ff245fbd0f89602098ddfe6064c3c is OK for suite deepsea:tier2. Check tests results in the Jenkins job: http://ci.ses.suse.de:8080/job/pr-deepsea/335/