when master does not have workers for queues it does not process, configured, the job rescuer loses jobs

[elee1766]

the insert only clients fail to rescue jobs, because https://github.com/riverqueue/river/blob/master/internal/maintenance/job_rescuer.go#L272-L277

it seems it looks to see if there are jobs registered, but since insert only clients have no workers configured, it will fail.

hearkening back to https://github.com/riverqueue/river/issues/336 disabling the ability for insert only clients to become masters would resolve this. (us not needing sessions would still be blocked behind #345 though, sadly)

this does lead into a better question of how job affinity should be done. its been previously recommended i believe to use this approach, along with different queues - was the correct "hack" to get job affinity (a much needed a missing feature from river), but i don't think given this restriction that this hack even works, as it makes the job queue a lot less reliable

[bgentry]

I'm not sure I understand the situation you're encountering. Insert-only clients are those which do not have Queues configured, and they cannot be started. They do not run leadership election, or run maintenance processes like job rescuing.

Can you provide more details about what you're encountering and how your clients are configured/initialized?

[elee1766]

oh, sorry, that's my bad, this is internal terminology that is bleeding over. i will update the issue.

so on processes that we do not want them to be able to process jobs, we simply do not declare the queues as the other workers. internally we call this "insert-only", as it is "only inserting jobs for these queues, and does not process them"

but this worker as a result also does not have the workers registered to it (it can't actually process the jobs).

i understand i could add the code for these workers and simply not process the queue - but this is very scary as it breaks very clear separation of needs, we would be initializing a large amount of worker functions (which are all stateful) that we would have no need for. even worse would be an accidental processing of the job, which would break everything.

i could also simply not have any affinity and use only actual insert-only clients for now. this would be almost okay, as i could refactor the tasks that can be processed on both machines to not be, but this is suboptimal, and also throws a wrench into future plans regarding affinity

[elee1766]

https://github.com/riverqueue/river/issues/336#issuecomment-2093914821

basically, it's me playing around and finding this out.

[bgentry]

@elee1766 Just getting back to this, I still do not understand the issue you're describing here. What is your setup? What do you mean that "the job rescuer loses jobs"?