Keep finished jobs in the database for history, but skip it while processing the queue

vgarvardt / gue

Golang queue on top of PostgreSQL

MIT License

278 stars 23 forks source link

Keep finished jobs in the database for history, but skip it while processing the queue #255

Open sadpenguinn opened 10 months ago

sadpenguinn commented 10 months ago

It's a useful feature to keep finished jobs in the database. Especially for production environment where this feature can help to discover bug cases in service logic by keeping failed and successful jobs. Clients can use it as well as logs. Gue will ensure to keep finished jobs in the database using the job's 'SkipDelete' property. But client should handle and monitor the memory size that these jobs take up and reduce the size when it's necessary.

vgarvardt commented 10 months ago

Hooks were designed specifically for this purpose - you can set one with WithWorkerHooksJobDone or WithPoolHooksJobDone and store finished jobs to the log with all the additional required information. This will fulfill your requirement and keep the table and code untouched.

kydemy-fran commented 3 months ago

I'd like to comment on this issue.

Deleting the jobs is the right thing to do for performance and tidiness. But it would be great if the library had the support out of the box to store the completed jobs:

Add a Option when configuring the pool: WithStoreDoneJobs
Add another migration to create a table: gue_jobs_completed
If the Option is set, automatically move the records to the other table.

I can help with a PR if that helps. Thanks!

vgarvardt commented 3 months ago

I would not want to add this functionality to the core library, but I'm thinking about adding some sort of plugins system, so that some pieces of functionality could be easily plugged into the core library w/out changing the core itself. This could be the first candidate as core already have everything to implement this functionality.