Add a 'check' function to the models

[marco-c]

Each model will have a few 'checks' that we will run periodically to ensure everything is OK.

The base check will be a check on the accuracy, if it's too low we need to be notified so we can figure out what's wrong.

Other models will have their own checks. E.g. the component model will have to check that the components it was trained on still exist and are still the most important.

[marco-c]

Another check should be on the feature importances (if a feature was really important and is no longer important => notify, if a feature was not important and is suddenly really important => notify).

[Lothiraldan]

Those checks will run somewhere else than the http_service, right?

From what I understand of this ticket, we need to check each model periodically (every day at midnight for example) against a recent set of data (latest bugs for example). Then the model 'check' will say if it's still relevant or not and if not we need to get notified.

Do you want me to focus on the scheduling + notification part or the implementing the check part?

[marco-c]

Yes, we can run them as Taskcluster tasks similarly to how we run the pipeline, but on a different schedule.

Do you want me to focus on the scheduling + notification part or the implementing the check part?

You could start with a simple check (e.g. the one for the components), then work on the scheduling/notification.

For the checks that need history (e.g. the accuracy), we could either use some sort of DB or, since the data is simple, we could use the Taskcluster index (see for example the static analysis usage of the index: https://tools.taskcluster.net/index/project.releng.services.project.production.static_analysis_bot.phabricator/15243).

[marco-c]

https://bugzilla.mozilla.org/report.cgi?y_axis_field=&cumulate=0&z_axis_field=&format=bar&x_axis_field=component&query_format=report-graph&short_desc_type=allwordssubstr&short_desc=&resolution=---&longdesc_type=allwordssubstr&longdesc=&bug_file_loc_type=allwordssubstr&bug_file_loc=&status_whiteboard_type=allwordssubstr&status_whiteboard=&keywords_type=allwords&keywords=&bug_id=&bug_id_type=anyexact&votes=&votes_type=greaterthaneq&emailassigned_to1=1&emailtype1=exact&email1=&emailassigned_to2=1&emailreporter2=1&emailqa_contact2=1&emailtype2=exact&email2=&emailtype3=substring&email3=&chfieldvalue=&chfieldfrom=&chfieldto=Now&j_top=AND&f1=creation_ts&o1=greaterthan&v1=2019-04-01&action=wrap

We can use something like this (CSV link at the bottom) to get all components and the number of bugs for each component, so we make one query only instead of one per product/component (and, we can also check that the most used components are in the list of components we have considered during training).

[Lothiraldan]

The CSV is missing the product, are the components unique outside their product namespace?

[marco-c]

No, there might be the same component names used across products. We should see if we can do something similar including the product too.

[marco-c]

Something like this should contain both product and component: https://bugzilla.mozilla.org/report.cgi?x_axis_field=product&y_axis_field=component&z_axis_field=&query_format=report-table&short_desc_type=allwordssubstr&short_desc=&resolution=---&longdesc_type=allwordssubstr&longdesc=&bug_file_loc_type=allwordssubstr&bug_file_loc=&status_whiteboard_type=allwordssubstr&status_whiteboard=&keywords_type=allwords&keywords=&bug_id=&bug_id_type=anyexact&votes=&votes_type=greaterthaneq&emailassigned_to1=1&emailtype1=exact&email1=&emailassigned_to2=1&emailreporter2=1&emailqa_contact2=1&emailtype2=exact&email2=&emailtype3=substring&email3=&chfieldvalue=&chfieldfrom=&chfieldto=Now&j_top=AND&f1=creation_ts&o1=greaterthan&v1=2019-04-01&f2=noop&o2=noop&v2=&format=table&action=wrap

[Lothiraldan]

(The current implementation takes 3.5 minutes but does multiple queries)

[Lothiraldan]

There seems to be an issue with the csv, some combinations of product /components are not returned, for example component Dom: Geolocation of product Core doesn't appears at all.

I'm checking if the report can be "fixed"

[Lothiraldan]

Here is the full list of missing components / product combination:

Component 'Printing' of product 'Toolkit' have 0 bugs in it, failure
Component 'WebIDE' of product 'DevTools' have 0 bugs in it, failure
Component 'Audio/Video: GMP' of product 'Core' have 0 bugs in it, failure
Component 'DOM: Geolocation' of product 'Core' have 0 bugs in it, failure
Component 'Layout: Floats' of product 'Core' have 0 bugs in it, failure
Component 'Firefox Accounts' of product 'Firefox for Android' have 0 bugs in it, failure
Component 'Metrics' of product 'Firefox for Android' have 0 bugs in it, failure
Component 'Toolbar' of product 'Firefox for Android' have 0 bugs in it, failure
Component 'Application Panel' of product 'DevTools' have 0 bugs in it, failure
Component 'Print Preview' of product 'Core' have 0 bugs in it, failure
Component 'JavaScript: Standard Library' of product 'Core' have 0 bugs in it, failure
Component 'Activity Streams: Application Servers' of product 'Firefox' have 0 bugs in it, failure
Component 'Device Permissions' of product 'Firefox' have 0 bugs in it, failure
Component 'Memory' of product 'DevTools' have 0 bugs in it, failure
Component 'Compatibility' of product 'WebExtensions' have 0 bugs in it, failure

[Lothiraldan]

When I try to limit the report to one component (one of the above list), the report is blank: https://bugzilla.mozilla.org/report.cgi?x_axis_field=product&y_axis_field=component&z_axis_field=&query_format=report-table&short_desc_type=allwordssubstr&short_desc=&component=Printing&resolution=---&longdesc_type=allwordssubstr&longdesc=&bug_file_loc_type=allwordssubstr&bug_file_loc=&status_whiteboard_type=allwordssubstr&status_whiteboard=&keywords_type=allwords&keywords=&bug_id=&bug_id_type=anyexact&votes=&votes_type=greaterthaneq&emailtype1=substring&email1=&emailtype2=substring&email2=&emailtype3=substring&email3=&chfieldvalue=&chfieldfrom=&chfieldto=Now&j_top=AND&f1=creation_ts&o1=greaterthan&v1=2019-04-01&f2=noop&o2=noop&v2=&f3=noop&o3=noop&v3=&format=table&action=wrap

[Lothiraldan]

Ok there was a date filter to I have missed while reading the URL.

When removing the date filter, all components are checking now, https://github.com/mozilla/bugbug/pull/341 is up-to-date