search

fleetdm / fleet

Open-source platform for IT, security, and infrastructure teams. (Linux, macOS, Chrome, Windows, cloud, data center)
https://fleetdm.com
Other
2.92k stars 406 forks source link

Collect metrics from the Vulnerability repo #18275

Open sharon-fdm opened 4 months ago

sharon-fdm commented 4 months ago

Goal

User story
As an engineer in FleetDM,
I want to know about any malfunction in the vuln repo as soon as possible
so that I can fix it and avoid customers not getting vuln info from it.

Context

We currently collect metrics and send it to DataDog. Use the same mechanism to send info from the GitHub action on the vuln repo directly to DataDog. Collect:

  1. How much time passed from the previous release to the current one.
  2. How many CVEs we published.
  3. Avg number of CPEs to CVE
  4. How many times the release was downloaded in the last 24 hours

Changes

Product

Engineering

ℹ️  Please read this issue carefully and understand it. Pay special attention to UI wireframes, especially "dev notes".

QA

Risk assessment

Manual testing steps

  1. Step 1
  2. Step 2
  3. Step 3

Testing notes

Confirmation

  1. [ ] Engineer (@____): Added comment to user story confirming successful completion of QA.
  2. [ ] QA (@____): Added comment to user story confirming successful completion of QA.
sharon-fdm commented 4 months ago

cc: @lukeheath @noahtalerman @mostlikelee Moved to specified per this:

image.png
lukeheath commented 4 months ago

@sharon-fdm This seems valuable because it will give us visibility into what's happening inside the vulnerabilities GitHub workflow. Could we also report how many times the release has been downloaded?

Our ability to work on this will be dependent on the estimate.

lukeheath commented 4 months ago

@sharon-fdm I'm assigning back to you to take to estimation.

sharon-fdm commented 4 months ago

@lukeheath

how many times the release has been downloaded

Good metric. Added.

sharon-fdm commented 4 months ago

Vuln Repo: 5 points Heroku + Datadog : 1 point

lukeheath commented 4 months ago

@sharon-fdm It won't be easy to prioritize this soon at a 5-point estimate. Can we reduce scope and not include datadog at all? What if we just fire a Slack notification to #help-engineering if the job fails? Seems like we could do that in 1-2 hours.

sharon-fdm commented 4 months ago

@lukeheath makes sense to shoot critical alerts only. @mostlikelee, two questions:

  1. Can we reduce the scope to reduce the effort to shoot info to Heroku?
  2. If not, we can create a slack app to easily send events. (I just created such app a month ago and can help with that)

TMWYH

mostlikelee commented 4 months ago

@sharon-fdm we already have failures posting to the P1-Help channel. We could timebox the metric effort to 2-3 points.