How to process and report on large batch jobs that are specific per user

[sieppl]

Thank you for the great package, after reading the documentation I am trying to understand how pg-boss can be used in my context. My requirements are:

• Synchronizing data between two systems selectively and reportable
• Users select with a filter what elements should be synchronized between system A and system B (between 1 and 10.000 elements depending on the filter)
• each sync run should show progress on how many elements are pending/successful/failed
• It should be possible to click on a run and see the status of each element that was synchronized

The scope of pg-boss for my scenario

• each sync run will be an own queue with a name that I can identify later for this user
• I have to publish() each element that needs to be synchronized to this queue (so between 1 and 10.000 pushes)
• then I subscribe() to this queue and process all elements
• onComplete() will fire when all jobs inside this queue are processed
• I can use getQueueSize() in order to show progress of the users sync run
• the internal job table of pg-boss will hold the status of each job unless I delete them on purpose

Out of scope of pg-boss for my scenario

• Persist which queue was created for which user in order to show all queues a user has created
• Persisting the state of a queue, because pg-boss does not explicitly have states for queues (just for jobs)
• Showing a full report of sync run means I need to pull all jobs of a specific queue. According to the documentation I could use fetchCompleted(), but it will be not efficient to render tables for a specific queue and support pagination. I will have to access the job table directly as mentioned in the documentation.

Can you confirm that I understood the scope and features of pg-boss correct?

[timgit]

the internal job table of pg-boss will hold the status of each job unless I delete them on purpose

This is mostly true based on your configuration. There are 2 jobs tables, job and archive. Once a job is completed, it will be moved to the archive after maintenance is run. It will be deleted from the archive eventually.