search

abcxyz / github-metrics-aggregator

Apache License 2.0
12 stars 3 forks source link

feat: leech pipeline implementation #56

Closed bradegler closed 1 year ago

bradegler commented 1 year ago

Initial Dataflow pipeline implementation to ingest logs from GitHub into GCS.

There are a few open TODOs and outside of building the container image there is no way to deploy this yet - will get those in a follow up.

bradegler commented 1 year ago

What do you think about making this a subcommand instead of a new binary?

I'm not sure how this would work with the beam runner. I'll see if I can make it work if you think it is the correct approach. I started there and pivoted away when I switch to Dataflow.

bradegler commented 1 year ago

The code lgtm but I wonder if we are able to add some unit test of the pipeline part?

Yes. I'm working on tests today.