search

juju-solutions / matrix

Automatic testing of big software deployments under various failure conditions
Other
8 stars 9 forks source link

matrix is too green #123

Closed kwmonroe closed 7 years ago

kwmonroe commented 7 years ago

Matrix is failing, yet showing up green in my CWR reports. See matrix.log under the Azure heading:

http://bigtop.charm.qa/cwr_charm_release_ubuntu_devenv_in_cs__kwmonroe_java_devenv/25/report.html

The first error is a timeout, but I thought gating=True would at least result in a yellow triangle:

matrix:185:execute: Error in Rule(task=Task(command='matrix.tasks.health', args={}, gating=True), conditions=[Condition(mode='after', statement='deploy'), Condition(mode='until', statement='health.status.healthy'), Condition(mode='periodic', statement=5)])'s task Task(command='matrix.tasks.health', args={}, gating=True)

It seems like i should not get a "check mark" for that test. Pip versions:

# pip3 list | grep -E 'matrix|juju'
juju (0.3.0)
jujubundlelib (0.5.3)
jujumatrix (0.9.0)
pengale commented 7 years ago

@kwmonroe Thanks for filing this. It looks like something went wrong on the error handling for the timeout.

It might be that my new handling for connection timeouts doesn't work as nicely in the wild as it does in my tests.

And it might be that I've just screwed up the case where the websocket connection stays healthy, but we time out for other reasons.

Will dig into this and see what I can do to fix it. Regardless of the cause, I agree that this should have caused a test failure!

abentley commented 7 years ago

Copy of the relevant log: https://pastebin.canonical.com/185841/

pengale commented 7 years ago

Fixed by https://github.com/juju-solutions/matrix/pull/129