elee1766
commented
3 months ago that all the different applications that create a River client need to use the same River config (with the same set of periodic jobs including those that are not in scope for the specific service)?
not the exact same river config, but yes, sharing the same scheduled_job across both workers and insert only client is what we are doing.
i ran into a similar problem here, you can see a response: https://github.com/riverqueue/river/issues/336#issuecomment-2093895895
I've observed (expected) behavior while developing with River (v0.4.1) that periodic jobs are only enqueued by an elected leader which I now understand to be part of the maintenance service responsibilities of an elected leader.
The "problem" that I'm experiencing is that I'm attempting to use river (and initialize River Clients) across a few different services that all have different workers and different river periodic job configurations and are mostly isolated by different queues. So if I have service A with a periodic job that executes every hour and service B with a different periodic job, if service B is the elected leader for more than 1 hour, service A's periodic job will not be enqueued per its configured schedule.
This is less of an issue and more of a question around how River was designed and if this use case was considered. Presumably, several services sharing a background job queue is a common pattern and we'd like for each service to contain the logic of processing jobs specific to that service. Does this necessitate that scheduled jobs should be declared and managed by a single service or that all the different applications that create a River client need to use the same River config (with the same set of periodic jobs including those that are not in scope for the specific service)?
The other workaround currently under discussion at the moment is creating separate DB schemas per service and instantiating the River tables in each schema so that each service has it's own elected leader. On the surface this seems like something River should be able to handle and I'm worried about the increased overhead with maintaining separate DB schemas for each application.
I was hoping you would be able to share insights/context into River's design and if this was something that was considered at any point in time and how you thought about it and if this is on your radar/roadmap. Curious what your recommendation is for a solution: consolidating job definition vs. separate DB schemas (or something else entirely) ?
Another last idea that I had was to create a new DB table in River
river_periodic_jobwhere the definition/configuration for periodic jobs could live and be shared across all services in a way that the river client configuration state can be pushed into the DB layer. In this way, the elected leader would have visibility into all periodic jobs and be able to enqueue jobs based on the periodic jobs defined in that DB table. Curious your thoughts on this idea as well - happy to help put forth a proposal and contribute time/work if you think this feature would be doable.Thanks for the time for reading that through! I've been using River for a few months now and have loved it. If anything, this "problem" has arisen out of my exuberance to extend River's capabilities to many of our backend services.