search

kostmo / circleci-failure-tracker

A log analyzer for CircleCI. Note that this project is now hosted at pytorch/dr-ci
https://github.com/pytorch/dr-ci
5 stars 2 forks source link

Use a worker queue to asynchronously process GitHub status events #58

Closed kostmo closed 4 years ago

kostmo commented 4 years ago

The Dr. CI GitHub app uses a webhook to subscribe to GitHub "status" events posted to Pull Requests.

Currently, these events are processed synchronously by the gh-notification-ingest-env Elastic Beanstalk backend, including scanning build logs and posting a summary back to the user's Pull Request.

This can be overwhelmed, however, by peaky loads. A better approach is to use a message queue to buffer the incoming GitHub statuses and process them asynchronously.

kostmo commented 4 years ago

There is now a small webservice (named gh-notification-ingest-env in Elastic Beanstalk, and hosted at domain github-notifications-ingest.pytorch.org) that receives GitHub webhook notifications and stores them (synchronously) in a database. A periodic (3-minute interval) AWS Lambda task EnqueSQSBuildScansFunction queries for unprocessed notifications in the database, and enqueues an SQS message for each of them. Finally, an Elastic Beanstalk Worker-tier server named log-scanning-worker process the SQS messages as capacity allows.