vectordotdev / vector

A high-performance observability data pipeline.
https://vector.dev
Mozilla Public License 2.0
17.61k stars 1.55k forks source link

New `github_metrics` source #993

Open binarylogic opened 5 years ago

binarylogic commented 5 years ago

I'm opening this to start a broader discussion around plugins and integrations with external services. This idea came about after looking into https://github.com/infinityworks/github-exporter. This is an interesting way to pull data from an API/service on an interval. For this source, specifically, we would pull gauges of star counts, release download counts, repo page view counts, and more.

Open questions:

  1. Is this better provided as a plugin? And do we want to explore a plugin system?
  2. 992 is an interesting way to solve this. You could provide a simple github_stats command that would expose the metrics data when executed.

  3. 991 is also an interesting way to solve this with the aforementioned Github exporter.

  4. We could explore a generic http source that polls an endpoint and parses the response.

There are a lot of big questions we need to answer before we can begin work on this.

ghost commented 4 years ago

I'm not sure that this still would be called github_metric, but I'd like such source to also generate raw events, such as commits, pull requests, comments, and other event types using the event API.