search

rails / solid_queue

Database-backed Active Job backend
MIT License
1.95k stars 130 forks source link

Jobs and transactional integrity docs don't seem to be what actually happens in Solid Queue when using a single DB #421

Open leondmello opened 1 day ago

leondmello commented 1 day ago

Having your jobs in the same ACID-compliant database as your application data enables a powerful yet sharp tool: taking advantage of transactional integrity to ensure some action in your app is not committed unless your job is also committed and viceversa, and ensuring that your job won't be enqueued until the transaction within which you're enqueuing it is committed.

The above text seems to indicate that if you use the same DB for ActiveRecord application models and the Solid Queue models, job enqueuing will only happen if the application model changes are successfully committed to the DB & vice versa. We changed our implementation to move from a separate DB to a single DB to avail of this. But in practice, we are seeing that this is not the case.

After we installed some debuggers, we found that eventually, this line of code is hit. That rails method creates a new nested transaction.

Such nested transactions will not roll back all transactions correctly when using something like postgres. We confirmed this behaviour by explicitly rolling back the application transaction after enqueuing the job. The Job record was saved, the application model record was not.

leondmello commented 1 day ago

An extension to this issue: https://github.com/rails/solid_queue/issues/409

leondmello commented 17 hours ago

Maybe a simple fix is to split that up into a find & create! That does not seem to create a new transaction

rosa commented 8 hours ago

Hey @leondmello, sorry for the delay, I'm on-call this week.

But in practice, we are seeing that this is not the case.

Do you have a minimal reproduction for this where you're seeing this is not the case due to the nested transactions?

Maybe a simple fix is to split that up into a find & create!

Unfortunately, that's not the same as create_or_find_by!, as the find might return a stale read and then the create! fails with a duplicate key exception.

rosa commented 8 hours ago

An extension to this issue: https://github.com/rails/solid_queue/issues/409

Hmm... I'm not sure how it's an extension 🤔 That other issue is about waiting until all transactions have been committed to actually enqueue the jobs, even if you called Job.perform_later inside a transaction. It's done by Active Job, and the default behaviour with respect to that, and how to configure it changed from Rails 7.2 to Rails 8, and the README was outdated.