We have 80,000,000 reports in the old metabase cache that need to be upgraded to the new test report format. We have CPAN::Testers::Backend::Migrate::MetabaseCache to do this, but there are too many reports to migrate all in one go.
We need a way to use Minion to process these reports, at a low priority (so that new incoming test reports are processed first). This new process should:
Check how many Metabase cache upgrade jobs are in the queue
Look for unprocessed cache entries that do not already have a job in the queue
Add new jobs to the queue until there are a set number of jobs (starting with 1000)
Every time the process is run, it should try to refill the queue with new jobs. This process will be run periodically to keep refilling the queue until every piece of data is processed.
We have 80,000,000 reports in the old metabase cache that need to be upgraded to the new test report format. We have CPAN::Testers::Backend::Migrate::MetabaseCache to do this, but there are too many reports to migrate all in one go.
We need a way to use Minion to process these reports, at a low priority (so that new incoming test reports are processed first). This new process should:
Every time the process is run, it should try to refill the queue with new jobs. This process will be run periodically to keep refilling the queue until every piece of data is processed.