timgit / pg-boss

Queueing jobs in Node.js using PostgreSQL like a boss
MIT License
1.79k stars 149 forks source link

fetchCompleted always returning null #345

Closed NGPixel closed 1 year ago

NGPixel commented 1 year ago

I'm trying to retrieve the list of recently completed jobs but calling .fetchCompleted('*', 25, { includeMeta: true }) always return null, even though there're many completed jobs when looking at the jobs table in the DB:

image

I tried querying a specific queue name as well (.fetchCompleted('job-quickTest', 25, { includeMeta: true })) but I still get null.

Am I missing something obvious?

Thanks!

timgit commented 1 year ago

Yes, I think the completed jobs don't exist. Completed jobs are opt-in via the option onComplete: true from send(). When they're in the table, they will have the prefix __state__completed__

NGPixel commented 1 year ago

@timgit Ok, I think the documentation (or rather the terminology) used is rather confusing. Even after multiple reads, it's not clear what many of the methods actually do or how they work.

You would expect fetchCompleted to simply return a list of jobs that got executed and completed. But that's not the case, it's an extra step you can add to your workflow. Now you have 2 different "completed" concepts, a job that is in completed state and another that rename jobs with a prefix so you can fetch them again and execute something else.

I want to display a list of recently executed jobs, but there doesn't seem to be any built-in methods for that (a "peek" method)? I can query the DB directly, but I'm just surprised this isn't possible natively. Again, the terminology used doesn't help, as one would expect "fetch" to just "read / peek", not actually retrieve jobs and mark them for work.

Another point I find perplexing is the use of UUID v1. I get that it's configurable to v4, but is there any reason why v1 is the default? Especially for something that generates a large number of UUIDs, v1 has a very high risk of collision in such scenario. PostgreSQL can also natively generate v4 UUID so not using this capability is a bit strange...

Thanks for the great library, it works great otherwise! Just some points that could be improved.

timgit commented 1 year ago

Thanks for the feedback.

In regards to "fetch" semantics, I agree naming things is one of the hardest problems in computer science. Do you think "receive" like SQS has is better? I'm fine with renaming things in a future semver major.

Also, I've had others agree with you regarding v1 vs. v4 for uuid, so I just switched the default to v4 in 8.1.0. Over the years, I've relied less and less on the uuid package, migrating most operations to pgcrypto. In a future release, I'd prefer to kick it out, because who doesn't like 1 less dependency, amirite?