search

mozilla / mozci

A library for inspecting push and task results in Mozilla's CI
https://mozci.readthedocs.io/
Mozilla Public License 2.0
10 stars 22 forks source link

Handle 'intermittent' classification #6

Open marco-c opened 4 years ago

marco-c commented 4 years ago

For now, we only take into account 'not classified' and 'fixed by commit', https://github.com/mozilla/mozci/blob/f9ca3bee3ea1cd822e88aa1858b542c80ef6e5ac/mozci/push.py#L218. What should we do when the classification is 'intermittent'? Can we know from the classification if it was introduced in this push or a previous one? We could load the linked bug from Bugzilla and check when it was filed, or is there a faster way?

marco-c commented 4 years ago

@jmaher do you know the answer to the main question: Given an "intermittent" classification for a failure, how can we tell if it was a "known intermittent" (already happened before) or a "new intermittent" (first time it was seen)? Do sheriffs classify these differently?

jmaher commented 4 years ago

they are not classified differently- we can cross reference bug filing times and determine if it was filed within 12 hours of the job completing. It often happens that after the bugs are filed we find they are duplicated to an existing bug.

marco-c commented 4 years ago

An idea to speed things up: if the classification is "intermittent CLASSIFICATION_BUG_ID" and CLASSIFICATION_BUG_ID is lower than the bug ID in the commit message, we can consider the intermittent to be pre-existing. If it is higher, we fallback to querying Bugzilla.

marco-c commented 4 years ago

A good test case for this, test-linux1804-64-asan/opt-web-platform-tests-e10s-15 on https://treeherder.mozilla.org/#/jobs?repo=autoland&tochange=f67d2901bafdfb476860e7d6ca5bd4b7f8a71e80&fromchange=d025442980452a4250f7192ef1408cb93c8bd934&searchStr=test-linux1804-64-asan%2Fopt-web-platform-tests-e10s-15. It passed on the parent of the parent of 39cc23a7483ce4f73ac20e195b7cbd3f7d49851e, and failed in the 7th child. It didn't run on any of the pushes between them. It is thus considered as a possible regression of 39cc23a7483ce4f73ac20e195b7cbd3f7d49851e, with distance 9.

marco-c commented 4 years ago

It looks like unfortunately this information is currently missing in ActiveData, so we have to wait for https://github.com/klahnakoski/ActiveData-ETL/issues/75 before being able to do this.

jmaher commented 4 years ago

I think this is a good idea, and probably 95% complete.