search

timgit / pg-boss

Queueing jobs in Postgres from Node.js like a boss
MIT License
2.04k stars 157 forks source link

Using startAfter to schedule far-future jobs? #145

Closed kevboh closed 4 years ago

kevboh commented 4 years ago

This library is great, my team loves it. We have jobs that become known to our system potentially months before they should run. startAfter seems like a natural fit in this case, but expireIn implies that jobs should not be set to startAfter dates in the far future, as they'd expire way before they run. Is this the case? Would you consider using startAfter to schedule jobs far in advance an antipattern in terms of pg-boss?

kevboh commented 4 years ago

Ah, if I'm reading the code right it looks like jobs that have not started don't expire. If so, only my second question stands: is this an antipattern?

timgit commented 4 years ago

Thanks for the feedback! Intriguing timing here on your question, as I'm in the middle of implementing retention policies as a configuration option ( see #144 ). My current thinking is that startAfter is subject to any configured retention policy, so if you were to add a job that had startAfter set to a date even a year from now, as long as you set a config option such as retentionDays to accommodate it, that would work and I wouldn't consider it an anti-pattern as long as you're happy with how the job table (and postgres) handles the performance in turns of the volume. For example, if you had 10 million jobs that are delayed until next year, you might be able to claim an anti-pattern, since the assumption of the queue is that there is very little overhead for the btree index to locate the next job, and once you have this many jobs in the table, the fetch time jumps up an order of magnitude and things like auto-vacuum overhead start impacting operations.

kevboh commented 4 years ago

Of course! I love the idea of retention, and think these two settings working in conjunction make sense. It might be worth warning a user if they set startAfter after a job's retention period.

Our system will not be scheduling millions of future jobs, but it's a good note.

For what it's worth, I also found the definition of expireIn confusing, and I'm willing to PR a docs change to clarify it if that would be useful to you.

timgit commented 4 years ago

PRs are always welcome, and I’d like another outside opinion on expireIn, too. Thanks!