Why does Scrutinizer not run sometimes?

mila-iqia / blocks

A Theano framework for building and training neural networks

Other

1.16k stars 351 forks source link

Why does Scrutinizer not run sometimes? #1010

Open dwf opened 8 years ago

dwf commented 8 years ago

I just noticed some flake errors that Scrutinizer raised about #976 but they got flagged on a subsequent pull request and not that one. Does it just "flake out" (no pun intended) after it's run a couple of times on a given PR?

bartvm commented 8 years ago

There seems to be something wrong with the actual status returned to ~~Travis~~GitHub. Scrutinizer's tests ran and failed for every commit (see e.g. the one for the first commit) but then Travis flagged them as successful anyway.

dwf commented 8 years ago

Er, you mean GitHub rather than Travis? Don't think Scrutinizer reports to Travis.

bartvm commented 8 years ago

Yup, sorry.

bartvm commented 8 years ago

All the commits in e.g. https://github.com/mila-udem/blocks/pull/1007 also failed in Scrutinizer but had a success status. Looking at the statuses through the GitHub API it looks as if GitHub receives 7 status changes from Scrutinizer: The first 4 are "pending", then it informs GitHub that the tests failed, after that it gets two identical updates with "state": "success" and the number of new issues. GitHub uses the last state received, so it marks things as successful...

dwf commented 8 years ago

Well that's useful.