I'm looking for feedback from @esbenp before I dig into a PR for this.
Goal:
I'd like to adapt pdf-bot to be a scaleable pdf rendering microservice which can have resources added/removed on demand to handle workload fluctuations.
Because PG is a shared database, it' possible to scale the work load horizontally across many machines in parallel. To accomplish this, we would need to change the queue locking mechanism to be on a per-job basis, and adapt the generation commands (shift:all comes to mind) to support this.
There are a few concerns here:
1) This would require a database migration of some sort to support
2) Process crashes, unhandled errors, etc could result in jobs never being processed if implemented poorly
3) ?
Purposed implementation:
Add a processing_started_at date column to the jobs table
Adapt getAllUnfinished to select jobs where they aren't completed and processing_started_at is greater than a given a configurable amount (30 sec default maybe)
Make isBusy calls return false always (maybe?)
Adapt cli scripts (shift, shift:all, etc) to handle the possibility of getting an empty array of jobs instead of relying on an isBusy call (maybe?)
Remove setIsBusy calls (maybe?)
Add changes to LowDb as well (maybe?)
Remove worker table
id
processing_started_at
completed_at
1
2018-01-08 17:31:17.825153
2018-01-08 17:31:48.925153
2
2018-01-08 17:31:17.825153
null
3
2018-01-08 17:31:47.925153
null
4
2018-01-08 17:31:48.925153
null
5
null
null
6
null
null
Given this sample data, jobs 2, 5, and 6 would be eligible for the next generation worker to start processing, while jobs 3 and 4 are assumed to be currently processing.
If this all sounds like too big of an overhaul, I'd be open to other suggestions. I'd also be willing to add the support to a new Redis database adapter instead as well.
I'm looking for feedback from @esbenp before I dig into a PR for this.
Goal:
I'd like to adapt
pdf-botto be a scaleable pdf rendering microservice which can have resources added/removed on demand to handle workload fluctuations.Problem:
Because of pdf-bot's PostgreSQL database wide queue locking, only one machine can render pdf's for the given API endpoint at a time.
Because PG is a shared database, it' possible to scale the work load horizontally across many machines in parallel. To accomplish this, we would need to change the queue locking mechanism to be on a per-job basis, and adapt the generation commands (
shift:allcomes to mind) to support this.There are a few concerns here:
1) This would require a database migration of some sort to support 2) Process crashes, unhandled errors, etc could result in jobs never being processed if implemented poorly 3) ?
Purposed implementation:
processing_started_atdate column to the jobs tablegetAllUnfinishedto select jobs where they aren't completed andprocessing_started_atis greater than a given a configurable amount (30 sec default maybe)isBusycalls return false always (maybe?)shift,shift:all, etc) to handle the possibility of getting an empty array of jobs instead of relying on anisBusycall (maybe?)setIsBusycalls (maybe?)LowDbas well (maybe?)workertableGiven this sample data, jobs 2, 5, and 6 would be eligible for the next generation worker to start processing, while jobs 3 and 4 are assumed to be currently processing.
If this all sounds like too big of an overhaul, I'd be open to other suggestions. I'd also be willing to add the support to a new Redis database adapter instead as well.