search

cpan-testers / cpantesters-backend

Backend data processing for CPAN Testers
Other
0 stars 4 forks source link

Make incoming test reports trigger backend processing jobs #5

Closed preaction closed 7 years ago

preaction commented 7 years ago

Once we have a test reports feed (https://github.com/cpan-testers/cpantesters-api/issues/6) we should listen to the feed to trigger new processing jobs.

The existing processing job (#1) should be modified to use Beam::Minion to queue a job. Each job should process one report.

We will need one daemon to watch the feed and trigger the jobs. This daemon should listen for each feed and then queue a job with the data. This daemon should be monitored by a runit file.

There will need to be an API in Beam::Minion to queue new jobs with arbitrary data (not just the command-line arguments of the current API) (see: https://github.com/preaction/Beam-Minion/issues/1).

preaction commented 7 years ago

This needs to be done as soon as possible: The scanning of the entire database for unprocessed entries takes much too long. There might be a way to fix that problem, but it will simply be easier to make incoming reports trigger individual processing jobs.

preaction commented 7 years ago

This has been completed by https://github.com/cpan-testers/cpantesters-api/commit/2842283ae189a9f80dab7673c19a614265cefca2 and https://github.com/cpan-testers/cpantesters-backend/commit/ecbfbf47cbb981f0e003735dd53590fa73468532 so this can be closed. There's another way to do this which involves message-passing via Mercury (https://github.com/cpan-testers/cpantesters-api/issues/6) and a daemon that knows how to queue up processing jobs based on messages, but that's another ticket.