Allow flaky tests to be manually changed to not flaky

[whesse]

@alexmarkov said:

... it would be useful to have an ability to manually override (reset) the flakiness status of particular test(s). This could be an annotation on the CL, or we could keep these "flakiness reset" commands along with approvals and integrate such ability to the test results feed, both for CI and per CL in Gerrit.

@athomas @whesse What do you think? Is there a workaround or maybe a command-line tool for that?

I want to get this feature added as soon as possible. Proposed implementation:

• Add flakiness status to the new current_status service
• Add an API to write a changed modified_flaky.json to the latest build for that configuration.
  • Builds are still using the per-build latest results.json and flaky.json from latest, not per-configuration
• Get a tool (command line?) for that API asap.

Relevant to #53 and #4.

[alexmarkov]

We can also recognize special lines in CL description, something like

Fixes language_2/foo/bar_test
Fixes language_2/foo/bar_test in dartk-linux-release-x64
Fixes language_2/foo/bar_test in dartk-*-x64, dartkp-*

and reset flakiness status and all previous approvals for the mentioned test and mentioned configurations before interpreting the results. This would work both for CQ and CI cases and would make sure that if CQ jobs are green then the test was actually fixed (and its failure is not hidden by either flakiness or approvals).

[athomas]

I like @alexmarkov's suggestion. Seems straightforward and clearly shows the intent to the reviewer.

[whesse]

Work on this issue is started by adding an "inactive" field to records in flaky.json that have been marked as not flaky, and changing the current removal of flaky records that have been stable for 100 runs to marking the records as inactive.

When an inactive flaky record becomes flaky again, mark the field as no longer inactive, and increment a "reactivated_count" field.

Make tools that read flaky.json recognize the "inactive" field.