search

riverqueue / river

Fast and reliable background jobs in Go
https://riverqueue.com
Mozilla Public License 2.0
3.59k stars 94 forks source link

Add internal "pilot" concept #627

Closed bgentry closed 1 month ago

bgentry commented 1 month ago

This builds out a new layer to sit between the Client and Driver, focused more on higher level operations than individual database queries. This allows those operations to be implemented with a combination of multiple database queries in a transaction, rather than just a single one.

Naming here was tough, I really didn't like any of the options I came up with and ultimately landed on "pilot", but I'm pretty indifferent to it if a truly good one can be found. I spent enough time trying to find a good one that I truly don't care at this point 😆

For now, the only operations implemented in the pilot are InsertMany and JobSetStateIfRunningMany (from the completer). I also made some changes within River to use these at the correct places and to propagate the pilot as needed. JobCompleteTx was updated to pull a Client out of the context in order to access this and use the same underlying query (via the pilot) as the BatchCompleter.

Based on #614 (including #624).

bgentry commented 1 month ago

I'd still call this a bit of a work in progress and I think it will evolve or even change significantly, but for now I think it's good enough to move forward and iterate on.

nexovec commented 1 month ago

Would you mind telling me what this is even for? The implementation is basically empty now, what's the pilot supposed to be doing?

bgentry commented 1 month ago

@nexovec essentially we need a way to be able to override or extend certain functions with more complex implementations to power upcoming River Pro features. Our "driver" concept mainly exists to power low level queries across different database engines with query or type differences, so abusing/reusing that concept for something like a higher level grouping of functionality didn't feel appropriate, hence the added layer. As of now the "pilot" in the OSS version is just a passthrough to the driver, I'm not sure whether that will change or not in the future, though it's unlikely to be anything users need to care about.

nexovec commented 1 month ago

So it's something for workflows?

bgentry commented 1 month ago

Not currently being used for workflows, no. But yes for the next Pro feature and some of the ones after that 🚀