search

pdehaan / zeit-serverless-test

Testing serverless functions on Vercel (formerly ZEIT).
0 stars 0 forks source link

Test and test infra alerts #7

Open rpappalax opened 4 years ago

rpappalax commented 4 years ago

Being able to leverage a service like this would help my team for monitoring issues (github) / bugs (bugzilla) filed by code sheriffs.

This would be an ideal workflow

  1. Sheriff logs an bug in bugzilla with: Whiteboard term: "mobile-infra-impact" or a github issue with label "intermittent-test" in org: mozilla-mobile
  2. Either of those would be forwarded to a slack channel like: mobile-test-alerts within a reasonable amount of time (1 hour max?)

This way, we could aggregate any issue pertaining to test or test infrastructure into a channel where most team communication is visible throughout the day

pdehaan commented 4 years ago

OK, cool. So not sure what the deal w/ Bugzilla would be yet. Options ahoy!

  1. We have to poll bugzilla on an interval and track which issues we've reported on (since bugzilla doesn't have webhooks, as far as I know).

    PROS: ? CONS: Polling bugzilla API and need to track issues we've already reported on. (same as below).

  2. We poll bugzilla and the migrate the ticket/bug/issue into GitHub, at which point our GitHub webhook takes over and sends the message via Slack).

    PROS: One notification channel and location. CONS: Polling bugzilla API and need to track which issues we've seen (unless we can update the Bugzilla ticket and add a whiteboard term ourselves for "migrated-to-github-issues").

pdehaan commented 4 years ago
  1. Sheriff logs an bug in bugzilla with: Whiteboard term: "mobile-infra-impact" or a github issue with label "intermittent-test" in org: mozilla-mobile

Not sure if GitHub can give us results for all of mozilla-mobile org, or if we have to only watch single repos.

rpappalax commented 4 years ago

Some potentially interesting data to report / alert on:

  1. Crash bugs filed (github/bugzilla)
  2. Intermittent test fails (github/bugzilla)
  3. Test infra fails / optimizations (bugzilla)
pdehaan commented 4 years ago

Great ideas, keep em all coming! I have been doing a bit of work on the dashboard and pushed a slightly newer version to production: https://repo-health-report.now.sh/github/mozilla-mobile/android-components/

Notable additions are the mozilla-mobile repos you requested and I went through each repo and did my best to identify some GitHub labels we might want to scrape. Not sure what to do with the data yet or how to display it, but pretty sure I know how to fetch all issues that match a single label (so we'd need to possibly fetch 3-4 labels separately and merge results or whatever).

I'll also copy your comment into the https://github.com/pdehaan/repo-health-report repo for the dashboard side of it.