SQLite queue database corruption

I'm seeing a strange error. I dropped @fractaledmind a note and he suggested I post here.

W, [2024-09-05T18:49:32.074281 #809854]  WARN -- : SolidQueue-0.3.4 Error deregistering Supervisor (1.1ms)  process_id: 68, pid: 564289, hostname: "HOST", last_heartbeat_at: Sat, 31 Aug 2024 15:16:55.474066000 UTC +00:00, claimed_size: 0, pruned: {:process=>#<SolidQueue::Process id: 68>, :pruned=>true, :claimed_size=>0, :error=>#<ActiveRecord::StatementInvalid: SQLite3::CorruptException: database disk image is malformed>, :exception=>["ActiveRecord::StatementInvalid", "SQLite3::CorruptException: database disk image is malformed"], :exception_object=>#<ActiveRecord::StatementInvalid: SQLite3::CorruptException: database disk image is malformed>}, error: "ActiveRecord::StatementInvalid SQLite3::CorruptException: database disk image is malformed"

In sqlite3 client.

sqlite> .open production-queue.sqlite3
sqlite> pragma integrity_check;
*** in database main ***
On tree page 523 cell 23: Rowid 146 out of order
NULL value in solid_queue_jobs.created_at
NULL value in solid_queue_jobs.updated_at
row 146 missing from index index_solid_queue_jobs_for_alerting
row 146 missing from index index_solid_queue_jobs_for_filtering
row 146 missing from index index_solid_queue_jobs_on_finished_at
row 146 missing from index index_solid_queue_jobs_on_active_job_id
row 146 missing from index index_solid_queue_jobs_on_class_name
NULL value in solid_queue_jobs.created_at
NULL value in solid_queue_jobs.updated_at
row 147 missing from index index_solid_queue_jobs_for_alerting
row 147 missing from index index_solid_queue_jobs_for_filtering
row 147 missing from index index_solid_queue_jobs_on_finished_at
row 147 missing from index index_solid_queue_jobs_on_active_job_id
row 147 missing from index index_solid_queue_jobs_on_class_name
NULL value in solid_queue_jobs.created_at
NULL value in solid_queue_jobs.updated_at
row 148 missing from index index_solid_queue_jobs_for_alerting
row 148 missing from index index_solid_queue_jobs_for_filtering
row 148 missing from index index_solid_queue_jobs_on_finished_at
row 148 missing from index index_solid_queue_jobs_on_active_job_id
row 148 missing from index index_solid_queue_jobs_on_class_name
NULL value in solid_queue_jobs.created_at
NULL value in solid_queue_jobs.updated_at
row 149 missing from index index_solid_queue_jobs_for_alerting
row 149 missing from index index_solid_queue_jobs_for_filtering
row 149 missing from index index_solid_queue_jobs_on_finished_at
row 149 missing from index index_solid_queue_jobs_on_active_job_id
row 149 missing from index index_solid_queue_jobs_on_class_name
Error: stepping, database disk image is malformed (11)
sqlite> pragma integrity_check;
*** in database main ***
On tree page 523 cell 23: Rowid 146 out of order
NULL value in solid_queue_jobs.created_at
NULL value in solid_queue_jobs.updated_at
row 146 missing from index index_solid_queue_jobs_for_alerting
row 146 missing from index index_solid_queue_jobs_for_filtering
row 146 missing from index index_solid_queue_jobs_on_finished_at
row 146 missing from index index_solid_queue_jobs_on_active_job_id
row 146 missing from index index_solid_queue_jobs_on_class_name
Error: stepping, database disk image is malformed (11)

Doing a .clone doesn't seem to fix.

I'm not in "real" production and this is a job queue and thus somewhat destroyable, so I'm going to take that approach.

I've also seen similar corruption on my local development environment too akin to that mentioned by @marcoroth on X.

https://x.com/marcoroth_/status/1831045885423497223

In that case I just destroyed and recreated the file and moved on.

Yes, this is excactly what I was seeing as well. Thanks for opening the issue and tagging me @andycroll as I wanted to open an issue for this too 🙌🏼

This happened to me in this open source repo: http://github.com/adrienpoly/rubyvideo

You can easily reproduce it by running rails s as a separate process and doing a double Ctrl-C in quick succession.

I think it only happens with the Puma Plugin from what I could tell.

In production, I'm running as a separate worker, not the puma plugin.

Ohhh! See also https://github.com/rails/solid_queue/issues/294, where @wdiechmann provided some advice in the last comment, which I'm not sure if it applies to your case.

The Puma plugin detail is quite interesting! Did you manage to reproduce only with that?

Ah, sorry, just reread and I see @andycroll's error happened when running just the supervisor separately 🤔 Do you know what happened right before that?

In my case a deploy (on a Hatchbox managed server) so killing the worker process was certainly happening when the error started.

It's also very surprising to me that you can corrupt a database with only Active Record and quite regular queries 🤔

*** in database main ***
On tree page 523 cell 23: Rowid 146 out of order
NULL value in solid_queue_jobs.created_at
NULL value in solid_queue_jobs.updated_at
row 146 missing from index index_solid_queue_jobs_for_alerting
row 146 missing from index index_solid_queue_jobs_for_filtering
row 146 missing from index index_solid_queue_jobs_on_finished_at
row 146 missing from index index_solid_queue_jobs_on_active_job_id
row 146 missing from index index_solid_queue_jobs_on_class_name
NULL value in solid_queue_jobs.created_at
NULL value in solid_queue_jobs.updated_at

I wouldn't know how to cause this on purpose 😅 A row missing from an index? 🤔

I'll use @marcoroth's instructions to reproduce and see if I can find out what's going on!

I haven't been able to dig in myself yet, but after reading this: https://www.sqlite.org/howtocorrupt.html and knowing the bits that I know about the supervisor architecture, I was first drawn to:

2.6. Carrying an open database connection across a fork()

If it helps, here's how my pragma integrity_check looks like:

❯ sqlite3
SQLite version 3.43.2 2023-10-10 13:08:14
Enter ".help" for usage hints.
Connected to a transient in-memory database.
Use ".open FILENAME" to reopen on a persistent database.
sqlite> .open ./storage/development-queue.sqlite3
sqlite> pragma integrity_check;
row 1 missing from index index_solid_queue_processes_on_supervisor_id
row 2 missing from index index_solid_queue_processes_on_supervisor_id
row 3 missing from index index_solid_queue_processes_on_supervisor_id
wrong # of entries in index index_solid_queue_processes_on_supervisor_id
sqlite>

Ohhhh, thanks a lot @fractaledmind. That'd make perfect sense as the culprit! I thought Rails would handle this properly and wouldn't reuse the same connection but I need to dig more.

Yeah, this sounds like it might be what is occurring:

Do not even call sqlite3_close() on a database connection from a child process if the connection was opened in the parent. It is safe to close the underlying file descriptor, but the sqlite3_close() interface might invoke cleanup activities that will delete content out from under the parent, leading to errors and perhaps even database corruption.

I thought Rails would handle this properly and wouldn't reuse the same connection but I need to dig more.

The reason I thought that were these PRs that I had looked at in the past, because of another error (not as serious as this one) that someone reported with PostgreSQL 🤔

There must be something else going on here.

No idea yet if this is related, but https://github.com/rails/rails/pull/31173 has this in the description:

I think sqlite3 is immune to this problem just because there is no server to confuse.

And no discard! method is defined (in that PR or currently) for the SQLite3 AR adapter

Thanks @fractaledmind! I see this code has changed quite a lot since these PRs, (for example https://github.com/rails/rails/pull/37296), so I was trying to understand how everything worked... That comment caught my attention too but I honestly wasn't sure why "there is no server to confuse" makes SQLite immune to this problem 😳

@marcoroth, I've been trying to reproduce the problem following your steps but I can't. I have to say I have upgraded Solid Queue in rubyvideo to version 0.8.2 as it was using a quite old version so maybe there's something else at play there.

I have to say I have upgraded Solid Queue in rubyvideo to version 0.8.2 as it was using a quite old version so maybe there's something else at play there.

Tried without upgrading and same thing... I can't manage to reproduce it by starting the app with rails s and then Ctrl + C, double, in quick succession 😕

I've also tried to artificially delay the jobs that are being run (in particular, MeiliSearch::Rails::MSJob, that were failing because I don't have Meilisearch running, so they were failing very fast) by adding some sleeps there, and still same behaviour. I don't manage to corrupt the file.

Oh, @andycroll, I just realised you're running version 0.3.4 in your logs above. The issue I referred to previously with PostgreSQL (https://github.com/rails/solid_queue/issues/264) got fixed in version 0.4.0. However, in the rubyvideo repo, solid_queue has been on version >= 0.4.0 for a few weeks. @marcoroth, can you reliably reproduce it with that version still? Just to make sure.

@marcoroth: I created this repo to hopefully help us find a reproduction: https://github.com/fractaledmind/solid_queue_bughunt

You will see that I have a LongRunningQueryJob that I enqueue automatically once Solid Queue is loaded. I am trying to ctrl + c twice once the job starts up such that the Rails process is killed and the job dies. But, the job always resolves. I'm curious what your experience is. Can you get everything to stop mid-operation? Or does the job eventually resolve for you to?

You can see if the job eventually resolves by just waiting 5 seconds or so after you kill the Rails process and waiting for logs to print. All you have to do is bin/rails s and everything goes automatically.

You will see I have commented out automatically killing the Rails server process. I was trying that as well. Still, the job resolves. My hunch is that it is/was somehow possible to force Solid Queue to not stop gracefully and if a job was running and connected to the database, that could corrupt. But I am not sure. Maybe if we all work on this reproduction repo, we can hunt it down.

Thanks a lot @fractaledmind! 🙏 I've been playing with it a bit, also tried downgrading Solid Queue there to 0.3.4 just in case. I've tried deleting the USR1 signal and just killing the Solid Queue supervisor in different ways, even via SIGKILL. So far I've only managed to get these

SolidQueue-0.3.4 Error deregistering Worker (21.1ms)  process_id: 15, pid: 96758, hostname: "Rosas-Air-M2.localdomain", last_heartbeat_at: Fri, 06 Sep 2024 14:46:55.769528000 UTC +00:00, claimed_size: 2, pruned: {:process=>#<SolidQueue::Process id: 15, kind: "Worker", last_heartbeat_at: "2024-09-06 14:46:55.769528000 +0000", supervisor_id: 13, pid: 96758, hostname: "Rosas-Air-M2.localdomain", metadata: {"polling_interval"=>0.1, "queues"=>"*", "thread_pool_size"=>3}, created_at: "2024-09-06 14:46:55.778267000 +0000">, :pruned=>false, :claimed_size=>3, :error=>#<ActiveRecord::StatementInvalid: SQLite3::BusyException: database is locked>, :exception=>["ActiveRecord::StatementInvalid", "SQLite3::BusyException: database is locked"], :exception_object=>#<ActiveRecord::StatementInvalid: SQLite3::BusyException: database is locked>}, error: "ActiveRecord::StatementInvalid SQLite3::BusyException: database is locked"
SolidQueue-0.3.4 Error in thread (0.0ms)  error: "ActiveRecord::StatementInvalid SQLite3::BusyException: database is locked"
SolidQueue-0.3.4 Shut down Worker (32876.4ms)  pid: 96758, hostname: "Rosas-Air-M2.localdomain", polling_interval: 0.1, queues: "*", thread_pool_size: 3

but I think those were expected as I tried to start the server again when the workers themselves hadn't realised their parent had been killed and were still around.

Also, here is a one-liner to check the queue database integrity: sqlite3 storage/development-queue.sqlite3 'pragma integrity_check;'

Nothing so far. Tried also configuring recurring long running jobs (that don't send any signals) every 10 seconds to add more stuff to the mix (and because that other report of a corrupted file was using recurring jobs heavily), but still haven't managed to reproduce this 😖

I'm trying to hang to any possible thread here as I'm quite stumped about why I can't manage to reproduce this 😅

In my case a deploy (on a Hatchbox managed server) so killing the worker process was certainly happening when the error started.

@andycroll, do you know how the deploy is happening there? Does it happen by sending a specific signal to the Solid Queue process so it stops? With any grace period to terminate orderly? Or just terminate and then "pull the plug" right away...?

Is there any chance that this is dependent on the sqlite3 gem variant you get?

I'm running the x86_64-darwin one (and I assume @andycroll is probably also running a x86_64-* variant in production) whereas most of you are probably running a arm64-* variant on your local dev machines.

I have had this happen twice on my local dev machine this week while running in the puma plugin and im on arm64. Unfortunately have no clue how it happened. I thought it might have something to do with switching between the puma plugin and running bin/jobs start and killing the process mid job but i cant recreate again. /shrug

Im also using a separate db for queue stuff if that matters at all.

@npezza93: Try pulling down my repro repo and see if you can set something up that causes corruption consistently: https://github.com/fractaledmind/solid_queue_bughunt

Until we find a way to reproduce, we will never solve this.

I upgraded SolidQueue to 0.6.0 so I could later upgrade to 0.8.x to see if the issue is still present on the latest version.

But I couldn't reproduce it on 0.6.0. And now, after downgrading again, I can't reproduce it on 0.5.0 anymore weirdly enough 🤔

Prior to upgrading to 0.6.0 I was able to reliably reproduce it.

I think im on to something. Just had it happen to me. Cant quite reproduce reliably yet but i think it may have something to do with me keeping the server running with the puma plugin running and closing my laptop for some period of time. Cant confirm yet but just a hunch. Will report back if i mind reliable instructions. Heres the log. It's weird that there are rollbacks happening trying to update the Process model and then it corrupts.

Turbo::StreamsChannel is transmitting the subscription confirmation
Turbo::StreamsChannel is streaming from Z2lkOi8vcWlyYS9TZWN1cml0eS80Nw:chart:1D
Unsubscribing from channel: {"channel":"Turbo::StreamsChannel","signed_stream_name":"IloybGtPaTh2Y1dseVlTOVRaV04xY21sMGVTODBOdzpjaGFydDoxRCI=--6b7a8177c089474a55c40174ae74943ecba61bc5effaf1d922cc00e95ded31a4"}
Turbo::StreamsChannel stopped streaming from Z2lkOi8vcWlyYS9TZWN1cml0eS80Nw:chart:1D
Turbo::StreamsChannel is transmitting the subscription confirmation
Turbo::StreamsChannel is streaming from Z2lkOi8vcWlyYS9TZWN1cml0eS80Nw:chart:1D
  [1m[35mSQL (0.2ms)[0m  [1m[36mBEGIN immediate TRANSACTION[0m
  [1m[36mSolidQueue::Process Update (3.9ms)[0m  [1m[33mUPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?[0m  [["last_heartbeat_at", "2024-09-06 14:20:18.968432"], ["id", 93]]
  [1m[36mTRANSACTION (0.1ms)[0m  [1m[36mCOMMIT TRANSACTION[0m
  [1m[35mSQL (0.1ms)[0m  [1m[36mBEGIN immediate TRANSACTION[0m
  [1m[36mSolidQueue::Process Update (1.1ms)[0m  [1m[33mUPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?[0m  [["last_heartbeat_at", "2024-09-06 14:20:19.144763"], ["id", 95]]
  [1m[36mTRANSACTION (0.0ms)[0m  [1m[31mROLLBACK TRANSACTION[0m
  [1m[35mSQL (0.0ms)[0m  [1m[36mBEGIN immediate TRANSACTION[0m
  [1m[36mSolidQueue::Process Update (0.6ms)[0m  [1m[33mUPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?[0m  [["last_heartbeat_at", "2024-09-06 14:20:19.164246"], ["id", 96]]
  [1m[36mTRANSACTION (0.0ms)[0m  [1m[31mROLLBACK TRANSACTION[0m
  [1m[35mSQL (0.0ms)[0m  [1m[36mBEGIN immediate TRANSACTION[0m
  [1m[36mSolidQueue::Process Update (0.7ms)[0m  [1m[33mUPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?[0m  [["last_heartbeat_at", "2024-09-06 14:20:19.169932"], ["id", 98]]
  [1m[36mTRANSACTION (0.0ms)[0m  [1m[31mROLLBACK TRANSACTION[0m
  [1m[35mSQL (8.7ms)[0m  [1m[36mBEGIN immediate TRANSACTION[0m
  [1m[36mSolidQueue::Process Update (9.5ms)[0m  [1m[33mUPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?[0m  [["last_heartbeat_at", "2024-09-06 14:20:19.164249"], ["id", 97]]
  [1m[36mTRANSACTION (0.0ms)[0m  [1m[31mROLLBACK TRANSACTION[0m
  [1m[35mSQL (13.8ms)[0m  [1m[36mBEGIN immediate TRANSACTION[0m
  [1m[36mSolidQueue::Process Update (15.1ms)[0m  [1m[33mUPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?[0m  [["last_heartbeat_at", "2024-09-06 14:20:19.164685"], ["id", 94]]
  [1m[36mTRANSACTION (0.1ms)[0m  [1m[36mCOMMIT TRANSACTION[0m
  [1m[35mSQL (0.1ms)[0m  [1m[36mBEGIN immediate TRANSACTION[0m
  [1m[36mSolidQueue::Process Update (1.7ms)[0m  [1m[33mUPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?[0m  [["last_heartbeat_at", "2024-09-06 14:20:19.196651"], ["id", 99]]
  [1m[36mTRANSACTION (0.0ms)[0m  [1m[31mROLLBACK TRANSACTION[0m
SolidQueue-0.8.2 Error in thread (0.0ms)  error: "ActiveRecord::StatementInvalid SQLite3::CorruptException: database disk image is malformed"
  [1m[35mSQL (0.0ms)[0m  [1m[36mBEGIN immediate TRANSACTION[0m
  [1m[36mSolidQueue::Process Update (0.5ms)[0m  [1m[33mUPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?[0m  [["last_heartbeat_at", "2024-09-06 14:21:19.012875"], ["id", 93]]
  [1m[36mTRANSACTION (0.0ms)[0m  [1m[36mCOMMIT TRANSACTION[0m
  [1m[35mSQL (0.0ms)[0m  [1m[36mBEGIN immediate TRANSACTION[0m
  [1m[36mSolidQueue::Process Update (0.7ms)[0m  [1m[33mUPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?[0m  [["last_heartbeat_at", "2024-09-06 14:21:19.160688"], ["id", 95]]
  [1m[36mTRANSACTION (0.0ms)[0m  [1m[31mROLLBACK TRANSACTION[0m
SolidQueue-0.8.2 Error in thread (0.0ms)  error: "ActiveRecord::StatementInvalid SQLite3::CorruptException: database disk image is malformed"
  [1m[35mSQL (0.0ms)[0m  [1m[36mBEGIN immediate TRANSACTION[0m
  [1m[36mSolidQueue::Process Update (0.6ms)[0m  [1m[33mUPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?[0m  [["last_heartbeat_at", "2024-09-06 14:21:19.176302"], ["id", 96]]
  [1m[36mTRANSACTION (0.0ms)[0m  [1m[31mROLLBACK TRANSACTION[0m
  [1m[35mSQL (0.0ms)[0m  [1m[36mBEGIN immediate TRANSACTION[0m
  [1m[36mSolidQueue::Process Update (1.1ms)[0m  [1m[33mUPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?[0m  [["last_heartbeat_at", "2024-09-06 14:21:19.177137"], ["id", 98]]
SolidQueue-0.8.2 Error in thread (0.0ms)  error: "ActiveRecord::StatementInvalid SQLite3::CorruptException: database disk image is malformed"
  [1m[36mTRANSACTION (0.0ms)[0m  [1m[31mROLLBACK TRANSACTION[0m
SolidQueue-0.8.2 Error in thread (0.0ms)  error: "ActiveRecord::StatementInvalid SQLite3::CorruptException: database disk image is malformed"
  [1m[35mSQL (0.0ms)[0m  [1m[36mBEGIN immediate TRANSACTION[0m
  [1m[36mSolidQueue::Process Update (0.8ms)[0m  [1m[33mUPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?[0m  [["last_heartbeat_at", "2024-09-06 14:21:19.184487"], ["id", 94]]
  [1m[36mTRANSACTION (0.0ms)[0m  [1m[36mCOMMIT TRANSACTION[0m
  [1m[35mSQL (3.6ms)[0m  [1m[36mBEGIN immediate TRANSACTION[0m
  [1m[36mSolidQueue::Process Update (4.3ms)[0m  [1m[33mUPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?[0m  [["last_heartbeat_at", "2024-09-06 14:21:19.184518"], ["id", 97]]
  [1m[36mTRANSACTION (0.0ms)[0m  [1m[31mROLLBACK TRANSACTION[0m
  [1m[35mSQL (0.1ms)[0m  [1m[36mBEGIN immediate TRANSACTION[0m
  [1m[36mSolidQueue::Process Update (1.0ms)[0m  [1m[33mUPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?[0m  [["last_heartbeat_at", "2024-09-06 14:21:19.201697"], ["id", 99]]
  [1m[36mTRANSACTION (0.0ms)[0m  [1m[31mROLLBACK TRANSACTION[0m
  [1m[36mSolidQueue::Process Load (0.1ms)[0m  [1m[34mSELECT "solid_queue_processes".* FROM "solid_queue_processes" WHERE "solid_queue_processes"."last_heartbeat_at" <= ? ORDER BY "solid_queue_processes"."id" ASC LIMIT ? [0m  [["last_heartbeat_at", "2024-09-06 14:17:18.947007"], ["LIMIT", 50]]
  [1m[35mSQL (0.1ms)[0m  [1m[36mBEGIN immediate TRANSACTION[0m
  [1m[36mSolidQueue::Process Destroy (1.1ms)[0m  [1m[31mDELETE FROM "solid_queue_processes" WHERE "solid_queue_processes"."id" = ?[0m  [["id", 8]]
  [1m[36mTRANSACTION (0.0ms)[0m  [1m[31mROLLBACK TRANSACTION[0m
SolidQueue-0.8.2 Error deregistering Supervisor (4.8ms)  process_id: 8, pid: 87668, hostname: "Nicks-MacBook-Air.local", name: "supervisor-fb2dc5791db13f37a778", last_heartbeat_at: "2024-09-05T23:20:38-04:00", claimed_size: nil, pruned: true, error: "ActiveRecord::StatementInvalid SQLite3::CorruptException: database disk image is malformed"
SolidQueue-0.8.2 Prune dead processes (30.5ms)  size: 13
SolidQueue-0.8.2 Error in thread (0.0ms)  error: "ActiveRecord::StatementInvalid SQLite3::CorruptException: database disk image is malformed"
  [1m[35mSQL (0.0ms)[0m  [1m[36mBEGIN immediate TRANSACTION[0m
  [1m[36mSolidQueue::Process Update (2.0ms)[0m  [1m[33mUPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?[0m  [["last_heartbeat_at", "2024-09-06 14:22:19.021586"], ["id", 93]]
  [1m[36mTRANSACTION (0.0ms)[0m  [1m[36mCOMMIT TRANSACTION[0m
  [1m[35mSQL (0.0ms)[0m  [1m[36mBEGIN immediate TRANSACTION[0m
  [1m[36mSolidQueue::Process Update (1.4ms)[0m  [1m[33mUPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?[0m  [["last_heartbeat_at", "2024-09-06 14:22:19.165340"], ["id", 95]]
  [1m[36mTRANSACTION (0.0ms)[0m  [1m[31mROLLBACK TRANSACTION[0m
  [1m[35mSQL (0.0ms)[0m  [1m[36mBEGIN immediate TRANSACTION[0m
  [1m[36mSolidQueue::Process Update (0.5ms)[0m  [1m[33mUPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?[0m  [["last_heartbeat_at", "2024-09-06 14:22:19.181037"], ["id", 96]]
  [1m[36mTRANSACTION (0.0ms)[0m  [1m[31mROLLBACK TRANSACTION[0m
  [1m[35mSQL (0.0ms)[0m  [1m[36mBEGIN immediate TRANSACTION[0m
  [1m[36mSolidQueue::Process Update (0.9ms)[0m  [1m[33mUPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?[0m  [["last_heartbeat_at", "2024-09-06 14:22:19.182902"], ["id", 98]]
  [1m[36mTRANSACTION (0.0ms)[0m  [1m[31mROLLBACK TRANSACTION[0m
  [1m[35mSQL (0.0ms)[0m  [1m[36mBEGIN immediate TRANSACTION[0m
  [1m[36mSolidQueue::Process Update (0.5ms)[0m  [1m[33mUPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?[0m  [["last_heartbeat_at", "2024-09-06 14:22:19.192213"], ["id", 94]]
  [1m[36mTRANSACTION (0.1ms)[0m  [1m[36mCOMMIT TRANSACTION[0m
  [1m[35mSQL (0.0ms)[0m  [1m[36mBEGIN immediate TRANSACTION[0m
  [1m[36mSolidQueue::Process Update (1.4ms)[0m  [1m[33mUPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?[0m  [["last_heartbeat_at", "2024-09-06 14:22:19.194242"], ["id", 97]]
  [1m[36mTRANSACTION (0.0ms)[0m  [1m[31mROLLBACK TRANSACTION[0m
SolidQueue-0.8.2 Error in thread (0.0ms)  error: "ActiveRecord::StatementInvalid SQLite3::CorruptException: database disk image is malformed"
  [1m[35mSQL (0.0ms)[0m  [1m[36mBEGIN immediate TRANSACTION[0m
  [1m[36mSolidQueue::Process Update (1.1ms)[0m  [1m[33mUPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?[0m  [["last_heartbeat_at", "2024-09-06 14:22:19.210680"], ["id", 99]]
  [1m[36mTRANSACTION (0.0ms)[0m  [1m[31mROLLBACK TRANSACTION[0m
SolidQueue-0.8.2 Error in thread (0.0ms)  error: "ActiveRecord::StatementInvalid SQLite3::CorruptException: database disk image is malformed"
  [1m[35mSQL (0.0ms)[0m  [1m[36mBEGIN immediate TRANSACTION[0m
  [1m[36mSolidQueue::Process Update (1.2ms)[0m  [1m[33mUPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?[0m  [["last_heartbeat_at", "2024-09-06 14:23:19.027648"], ["id", 93]]
  [1m[36mTRANSACTION (0.0ms)[0m  [1m[36mCOMMIT TRANSACTION[0m
  [1m[35mSQL (0.0ms)[0m  [1m[36mBEGIN immediate TRANSACTION[0m
  [1m[36mSolidQueue::Process Update (1.3ms)[0m  [1m[33mUPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?[0m  [["last_heartbeat_at", "2024-09-06 14:23:19.173473"], ["id", 95]]
  [1m[36mTRANSACTION (0.0ms)[0m  [1m[31mROLLBACK TRANSACTION[0m
SolidQueue-0.8.2 Error in thread (0.0ms)  error: "ActiveRecord::StatementInvalid SQLite3::CorruptException: database disk image is malformed"
  [1m[35mSQL (0.0ms)[0m  [1m[36mBEGIN immediate TRANSACTION[0m
  [1m[36mSolidQueue::Process Update (0.4ms)[0m  [1m[33mUPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?[0m  [["last_heartbeat_at", "2024-09-06 14:23:19.184874"], ["id", 96]]
  [1m[36mTRANSACTION (0.0ms)[0m  [1m[31mROLLBACK TRANSACTION[0m
SolidQueue-0.8.2 Error in thread (0.0ms)  error: "ActiveRecord::StatementInvalid SQLite3::CorruptException: database disk image is malformed"
  [1m[35mSQL (0.0ms)[0m  [1m[36mBEGIN immediate TRANSACTION[0m
  [1m[36mSolidQueue::Process Update (0.5ms)[0m  [1m[33mUPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?[0m  [["last_heartbeat_at", "2024-09-06 14:23:19.187570"], ["id", 98]]
  [1m[36mTRANSACTION (0.0ms)[0m  [1m[31mROLLBACK TRANSACTION[0m
SolidQueue-0.8.2 Error in thread (0.0ms)  error: "ActiveRecord::StatementInvalid SQLite3::CorruptException: database disk image is malformed"
  [1m[35mSQL (0.0ms)[0m  [1m[36mBEGIN immediate TRANSACTION[0m
  [1m[36mSolidQueue::Process Update (0.4ms)[0m  [1m[33mUPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?[0m  [["last_heartbeat_at", "2024-09-06 14:23:19.199510"], ["id", 94]]
  [1m[36mTRANSACTION (0.1ms)[0m  [1m[36mCOMMIT TRANSACTION[0m
  [1m[35mSQL (0.0ms)[0m  [1m[36mBEGIN immediate TRANSACTION[0m
  [1m[36mSolidQueue::Process Update (0.4ms)[0m  [1m[33mUPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?[0m  [["last_heartbeat_at", "2024-09-06 14:23:19.202226"], ["id", 97]]
  [1m[36mTRANSACTION (0.0ms)[0m  [1m[31mROLLBACK TRANSACTION[0m
SolidQueue-0.8.2 Error in thread (0.0ms)  error: "ActiveRecord::StatementInvalid SQLite3::CorruptException: database disk image is malformed"
  [1m[35mSQL (0.0ms)[0m  [1m[36mBEGIN immediate TRANSACTION[0m
  [1m[36mSolidQueue::Process Update (0.4ms)[0m  [1m[33mUPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?[0m  [["last_heartbeat_at", "2024-09-06 14:23:19.214210"], ["id", 99]]
  [1m[36mTRANSACTION (0.0ms)[0m  [1m[31mROLLBACK TRANSACTION[0m
SolidQueue-0.8.2 Error in thread (0.0ms)  error: "ActiveRecord::StatementInvalid SQLite3::CorruptException: database disk image is malformed"
  [1m[35mSQL (0.1ms)[0m  [1m[36mBEGIN immediate TRANSACTION[0m
  [1m[36mSolidQueue::Process Update (1.2ms)[0m  [1m[33mUPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?[0m  [["last_heartbeat_at", "2024-09-06 14:24:19.033318"], ["id", 93]]
  [1m[36mTRANSACTION (0.0ms)[0m  [1m[36mCOMMIT TRANSACTION[0m
  [1m[35mSQL (0.1ms)[0m  [1m[36mBEGIN immediate TRANSACTION[0m
  [1m[36mSolidQueue::Process Update (2.1ms)[0m  [1m[33mUPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?[0m  [["last_heartbeat_at", "2024-09-06 14:24:19.181729"], ["id", 95]]
  [1m[36mTRANSACTION (0.1ms)[0m  [1m[31mROLLBACK TRANSACTION[0m
  [1m[35mSQL (0.1ms)[0m  [1m[36mBEGIN immediate TRANSACTION[0m
SolidQueue-0.8.2 Error in thread (0.0ms)  error: "ActiveRecord::StatementInvalid SQLite3::CorruptException: database disk image is malformed"
  [1m[36mSolidQueue::Process Update (1.8ms)[0m  [1m[33mUPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?[0m  [["last_heartbeat_at", "2024-09-06 14:24:19.187835"], ["id", 96]]
  [1m[36mTRANSACTION (0.1ms)[0m  [1m[31mROLLBACK TRANSACTION[0m
  [1m[35mSQL (0.0ms)[0m  [1m[36mBEGIN immediate TRANSACTION[0m
SolidQueue-0.8.2 Error in thread (0.0ms)  error: "ActiveRecord::StatementInvalid SQLite3::CorruptException: database disk image is malformed"
  [1m[36mSolidQueue::Process Update (0.7ms)[0m  [1m[33mUPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?[0m  [["last_heartbeat_at", "2024-09-06 14:24:19.194634"], ["id", 98]]
  [1m[36mTRANSACTION (0.0ms)[0m  [1m[31mROLLBACK TRANSACTION[0m
SolidQueue-0.8.2 Error in thread (0.0ms)  error: "ActiveRecord::StatementInvalid SQLite3::CorruptException: database disk image is malformed"
  [1m[35mSQL (0.0ms)[0m  [1m[36mBEGIN immediate TRANSACTION[0m
  [1m[36mSolidQueue::Process Update (0.9ms)[0m  [1m[33mUPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?[0m  [["last_heartbeat_at", "2024-09-06 14:24:19.205474"], ["id", 94]]
  [1m[36mTRANSACTION (0.1ms)[0m  [1m[36mCOMMIT TRANSACTION[0m
  [1m[35mSQL (1.1ms)[0m  [1m[36mBEGIN immediate TRANSACTION[0m
  [1m[36mSolidQueue::Process Update (2.5ms)[0m  [1m[33mUPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?[0m  [["last_heartbeat_at", "2024-09-06 14:24:19.206692"], ["id", 97]]
  [1m[36mTRANSACTION (0.0ms)[0m  [1m[31mROLLBACK TRANSACTION[0m
  [1m[35mSQL (0.0ms)[0m  [1m[36mBEGIN immediate TRANSACTION[0m
  [1m[36mSolidQueue::Process Update (1.0ms)[0m  [1m[33mUPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?[0m  [["last_heartbeat_at", "2024-09-06 14:24:19.219473"], ["id", 99]]
  [1m[36mTRANSACTION (0.0ms)[0m  [1m[31mROLLBACK TRANSACTION[0m
  [1m[35mSQL (0.2ms)[0m  [1m[36mBEGIN immediate TRANSACTION[0m
  [1m[36mSolidQueue::Process Update (3.2ms)[0m  [1m[33mUPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?[0m  [["last_heartbeat_at", "2024-09-06 14:25:19.053217"], ["id", 93]]
  [1m[36mTRANSACTION (0.1ms)[0m  [1m[36mCOMMIT TRANSACTION[0m
  [1m[35mSQL (0.1ms)[0m  [1m[36mBEGIN immediate TRANSACTION[0m
  [1m[36mSolidQueue::Process Update (3.1ms)[0m  [1m[33mUPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?[0m  [["last_heartbeat_at", "2024-09-06 14:25:19.195502"], ["id", 95]]
  [1m[36mTRANSACTION (0.0ms)[0m  [1m[31mROLLBACK TRANSACTION[0m
  [1m[35mSQL (1.3ms)[0m  [1m[36mBEGIN immediate TRANSACTION[0m
  [1m[36mSolidQueue::Process Update (2.4ms)[0m  [1m[33mUPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?[0m  [["last_heartbeat_at", "2024-09-06 14:25:19.212315"], ["id", 94]]
  [1m[36mTRANSACTION (0.2ms)[0m  [1m[36mCOMMIT TRANSACTION[0m
  [1m[35mSQL (2.6ms)[0m  [1m[36mBEGIN immediate TRANSACTION[0m
  [1m[36mSolidQueue::Process Update (15.1ms)[0m  [1m[33mUPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?[0m  [["last_heartbeat_at", "2024-09-06 14:25:19.201020"], ["id", 98]]
  [1m[36mTRANSACTION (0.0ms)[0m  [1m[31mROLLBACK TRANSACTION[0m
  [1m[35mSQL (16.3ms)[0m  [1m[36mBEGIN immediate TRANSACTION[0m
SolidQueue-0.8.2 Error in thread (0.0ms)  error: "ActiveRecord::StatementInvalid SQLite3::CorruptException: database disk image is malformed"
  [1m[36mSolidQueue::Process Update (17.1ms)[0m  [1m[33mUPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?[0m  [["last_heartbeat_at", "2024-09-06 14:25:19.201014"], ["id", 96]]
  [1m[36mTRANSACTION (0.0ms)[0m  [1m[31mROLLBACK TRANSACTION[0m
SolidQueue-0.8.2 Error in thread (0.0ms)  error: "ActiveRecord::StatementInvalid SQLite3::CorruptException: database disk image is malformed"
  [1m[35mSQL (7.6ms)[0m  [1m[36mBEGIN immediate TRANSACTION[0m
  [1m[36mSolidQueue::Process Update (13.9ms)[0m  [1m[33mUPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?[0m  [["last_heartbeat_at", "2024-09-06 14:25:19.218068"], ["id", 97]]
  [1m[36mTRANSACTION (0.0ms)[0m  [1m[31mROLLBACK TRANSACTION[0m
  [1m[35mSQL (3.9ms)[0m  [1m[36mBEGIN immediate TRANSACTION[0m
SolidQueue-0.8.2 Error in thread (0.0ms)  error: "ActiveRecord::StatementInvalid SQLite3::CorruptException: database disk image is malformed"
  [1m[36mSolidQueue::Process Update (10.3ms)[0m  [1m[33mUPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?[0m  [["last_heartbeat_at", "2024-09-06 14:25:19.228835"], ["id", 99]]
  [1m[36mTRANSACTION (0.0ms)[0m  [1m[31mROLLBACK TRANSACTION[0m
SolidQueue-0.8.2 Error in thread (0.0ms)  error: "ActiveRecord::StatementInvalid SQLite3::CorruptException: database disk image is malformed"
  [1m[35mSQL (0.0ms)[0m  [1m[36mBEGIN immediate TRANSACTION[0m
  [1m[36mSolidQueue::Process Update (0.6ms)[0m  [1m[33mUPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?[0m  [["last_heartbeat_at", "2024-09-06 14:26:19.065500"], ["id", 93]]
  [1m[36mTRANSACTION (0.0ms)[0m  [1m[36mCOMMIT TRANSACTION[0m
  [1m[35mSQL (0.0ms)[0m  [1m[36mBEGIN immediate TRANSACTION[0m
  [1m[36mSolidQueue::Process Update (0.5ms)[0m  [1m[33mUPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?[0m  [["last_heartbeat_at", "2024-09-06 14:26:19.217067"], ["id", 95]]
  [1m[36mTRANSACTION (0.0ms)[0m  [1m[31mROLLBACK TRANSACTION[0m
SolidQueue-0.8.2 Error in thread (0.0ms)  error: "ActiveRecord::StatementInvalid SQLite3::CorruptException: database disk image is malformed"
  [1m[35mSQL (0.0ms)[0m  [1m[36mBEGIN immediate TRANSACTION[0m
  [1m[36mSolidQueue::Process Update (0.4ms)[0m  [1m[33mUPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?[0m  [["last_heartbeat_at", "2024-09-06 14:26:19.218612"], ["id", 94]]
  [1m[36mTRANSACTION (0.0ms)[0m  [1m[36mCOMMIT TRANSACTION[0m
  [1m[35mSQL (0.0ms)[0m  [1m[36mBEGIN immediate TRANSACTION[0m
  [1m[36mSolidQueue::Process Update (1.1ms)[0m  [1m[33mUPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?[0m  [["last_heartbeat_at", "2024-09-06 14:26:19.223827"], ["id", 98]]
  [1m[36mTRANSACTION (0.0ms)[0m  [1m[31mROLLBACK TRANSACTION[0m
  [1m[35mSQL (0.0ms)[0m  [1m[36mBEGIN immediate TRANSACTION[0m
  [1m[36mSolidQueue::Process Update (0.6ms)[0m  [1m[33mUPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?[0m  [["last_heartbeat_at", "2024-09-06 14:26:19.226618"], ["id", 96]]
  [1m[36mTRANSACTION (0.0ms)[0m  [1m[31mROLLBACK TRANSACTION[0m
  [1m[35mSQL (0.0ms)[0m  [1m[36mBEGIN immediate TRANSACTION[0m
  [1m[36mSolidQueue::Process Update (0.4ms)[0m  [1m[33mUPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?[0m  [["last_heartbeat_at", "2024-09-06 14:26:19.234613"], ["id", 97]]
  [1m[36mTRANSACTION (0.0ms)[0m  [1m[31mROLLBACK TRANSACTION[0m
  [1m[35mSQL (0.0ms)[0m  [1m[36mBEGIN immediate TRANSACTION[0m
  [1m[36mSolidQueue::Process Update (0.5ms)[0m  [1m[33mUPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?[0m  [["last_heartbeat_at", "2024-09-06 14:26:19.242735"], ["id", 99]]
  [1m[36mTRANSACTION (0.0ms)[0m  [1m[31mROLLBACK TRANSACTION[0m
  [1m[36mSolidQueue::Process Load (0.3ms)[0m  [1m[34mSELECT "solid_queue_processes".* FROM "solid_queue_processes" WHERE "solid_queue_processes"."last_heartbeat_at" <= ? ORDER BY "solid_queue_processes"."id" ASC LIMIT ? [0m  [["last_heartbeat_at", "2024-09-06 14:22:18.983585"], ["LIMIT", 50]]
  [1m[35mSQL (0.1ms)[0m  [1m[36mBEGIN immediate TRANSACTION[0m
  [1m[36mSolidQueue::Process Destroy (1.5ms)[0m  [1m[31mDELETE FROM "solid_queue_processes" WHERE "solid_queue_processes"."id" = ?[0m  [["id", 8]]
  [1m[36mTRANSACTION (0.0ms)[0m  [1m[31mROLLBACK TRANSACTION[0m
SolidQueue-0.8.2 Error deregistering Supervisor (5.5ms)  process_id: 8, pid: 87668, hostname: "Nicks-MacBook-Air.local", name: "supervisor-fb2dc5791db13f37a778", last_heartbeat_at: "2024-09-05T23:20:38-04:00", claimed_size: nil, pruned: true, error: "ActiveRecord::StatementInvalid SQLite3::CorruptException: database disk image is malformed"
SolidQueue-0.8.2 Prune dead processes (13.7ms)  size: 13
SolidQueue-0.8.2 Error in thread (0.0ms)  error: "ActiveRecord::StatementInvalid SQLite3::CorruptException: database disk image is malformed"
  [1m[35mSQL (0.1ms)[0m  [1m[36mBEGIN immediate TRANSACTION[0m
  [1m[36mSolidQueue::Process Update (1.8ms)[0m  [1m[33mUPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?[0m  [["last_heartbeat_at", "2024-09-06 14:27:19.069270"], ["id", 93]]
  [1m[36mTRANSACTION (0.1ms)[0m  [1m[36mCOMMIT TRANSACTION[0m
  [1m[36mSolidQueue::Semaphore Pluck (0.2ms)[0m  [1m[34mSELECT "solid_queue_semaphores"."id" FROM "solid_queue_semaphores" WHERE "solid_queue_semaphores"."expires_at" < ? ORDER BY "solid_queue_semaphores"."id" ASC LIMIT ?[0m  [["expires_at", "2024-09-06 14:27:19.140496"], ["LIMIT", 500]]
  [1m[36mSolidQueue::BlockedExecution Pluck (0.2ms)[0m  [1m[34mSELECT DISTINCT "solid_queue_blocked_executions"."concurrency_key" FROM "solid_queue_blocked_executions" WHERE "solid_queue_blocked_executions"."expires_at" < ? LIMIT ?[0m  [["expires_at", "2024-09-06 14:27:19.143734"], ["LIMIT", 500]]
SolidQueue-0.8.2 Unblock jobs (1.9ms)  limit: 500, size: 0
  [1m[35mSQL (0.1ms)[0m  [1m[36mBEGIN immediate TRANSACTION[0m
  [1m[36mSolidQueue::Process Update (1.5ms)[0m  [1m[33mUPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?[0m  [["last_heartbeat_at", "2024-09-06 14:27:19.221337"], ["id", 95]]
  [1m[36mTRANSACTION (0.1ms)[0m  [1m[31mROLLBACK TRANSACTION[0m
  [1m[35mSQL (2.7ms)[0m  [1m[36mBEGIN immediate TRANSACTION[0m
  [1m[36mSolidQueue::Process Update (3.7ms)[0m  [1m[33mUPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?[0m  [["last_heartbeat_at", "2024-09-06 14:27:19.224777"], ["id", 94]]
  [1m[36mTRANSACTION (0.1ms)[0m  [1m[36mCOMMIT TRANSACTION[0m
  [1m[35mSQL (1.4ms)[0m  [1m[36mBEGIN immediate TRANSACTION[0m
  [1m[36mSolidQueue::Process Update (2.8ms)[0m  [1m[33mUPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?[0m  [["last_heartbeat_at", "2024-09-06 14:27:19.229198"], ["id", 98]]
  [1m[36mTRANSACTION (0.0ms)[0m  [1m[31mROLLBACK TRANSACTION[0m
SolidQueue-0.8.2 Error in thread (0.0ms)  error: "ActiveRecord::StatementInvalid SQLite3::CorruptException: database disk image is malformed"
  [1m[35mSQL (0.2ms)[0m  [1m[36mBEGIN immediate TRANSACTION[0m
  [1m[36mSolidQueue::Process Update (1.3ms)[0m  [1m[33mUPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?[0m  [["last_heartbeat_at", "2024-09-06 14:27:19.233770"], ["id", 96]]
  [1m[36mTRANSACTION (0.0ms)[0m  [1m[31mROLLBACK TRANSACTION[0m
SolidQueue-0.8.2 Error in thread (0.0ms)  error: "ActiveRecord::StatementInvalid SQLite3::CorruptException: database disk image is malformed"
  [1m[35mSQL (0.1ms)[0m  [1m[36mBEGIN immediate TRANSACTION[0m
  [1m[36mSolidQueue::Process Update (1.0ms)[0m  [1m[33mUPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?[0m  [["last_heartbeat_at", "2024-09-06 14:27:19.241697"], ["id", 97]]
  [1m[36mTRANSACTION (0.0ms)[0m  [1m[31mROLLBACK TRANSACTION[0m
SolidQueue-0.8.2 Error in thread (0.0ms)  error: "ActiveRecord::StatementInvalid SQLite3::CorruptException: database disk image is malformed"
  [1m[35mSQL (0.1ms)[0m  [1m[36mBEGIN immediate TRANSACTION[0m
  [1m[36mSolidQueue::Process Update (1.2ms)[0m  [1m[33mUPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?[0m  [["last_heartbeat_at", "2024-09-06 14:27:19.250172"], ["id", 99]]
  [1m[36mTRANSACTION (0.0ms)[0m  [1m[31mROLLBACK TRANSACTION[0m
SolidQueue-0.8.2 Error in thread (0.0ms)  error: "ActiveRecord::StatementInvalid SQLite3::CorruptException: database disk image is malformed"
  [1m[35mSQL (1.3ms)[0m  [1m[36mBEGIN immediate TRANSACTION[0m
  [1m[36mSolidQueue::Process Update (3.2ms)[0m  [1m[33mUPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?[0m  [["last_heartbeat_at", "2024-09-06 14:28:19.080873"], ["id", 93]]
  [1m[36mTRANSACTION (0.2ms)[0m  [1m[36mCOMMIT TRANSACTION[0m
  [1m[35mSQL (0.1ms)[0m  [1m[36mBEGIN immediate TRANSACTION[0m
  [1m[36mSolidQueue::Process Update (1.7ms)[0m  [1m[33mUPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?[0m  [["last_heartbeat_at", "2024-09-06 14:28:19.234057"], ["id", 94]]
  [1m[35mSQL (2.9ms)[0m  [1m[36mBEGIN immediate TRANSACTION[0m
  [1m[36mTRANSACTION (0.1ms)[0m  [1m[36mCOMMIT TRANSACTION[0m
  [1m[36mSolidQueue::Process Update (4.7ms)[0m  [1m[33mUPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?[0m  [["last_heartbeat_at", "2024-09-06 14:28:19.234057"], ["id", 95]]
  [1m[36mTRANSACTION (0.0ms)[0m  [1m[31mROLLBACK TRANSACTION[0m
  [1m[35mSQL (0.1ms)[0m  [1m[36mBEGIN immediate TRANSACTION[0m
SolidQueue-0.8.2 Error in thread (0.0ms)  error: "ActiveRecord::StatementInvalid SQLite3::CorruptException: database disk image is malformed"
  [1m[36mSolidQueue::Process Update (1.0ms)[0m  [1m[33mUPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?[0m  [["last_heartbeat_at", "2024-09-06 14:28:19.240062"], ["id", 98]]
  [1m[36mTRANSACTION (0.0ms)[0m  [1m[31mROLLBACK TRANSACTION[0m
  [1m[35mSQL (1.3ms)[0m  [1m[36mBEGIN immediate TRANSACTION[0m
SolidQueue-0.8.2 Error in thread (0.0ms)  error: "ActiveRecord::StatementInvalid SQLite3::CorruptException: database disk image is malformed"
  [1m[36mSolidQueue::Process Update (2.7ms)[0m  [1m[33mUPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?[0m  [["last_heartbeat_at", "2024-09-06 14:28:19.240870"], ["id", 96]]
  [1m[36mTRANSACTION (0.0ms)[0m  [1m[31mROLLBACK TRANSACTION[0m
SolidQueue-0.8.2 Error in thread (0.0ms)  error: "ActiveRecord::StatementInvalid SQLite3::CorruptException: database disk image is malformed"
  [1m[35mSQL (0.0ms)[0m  [1m[36mBEGIN immediate TRANSACTION[0m
  [1m[36mSolidQueue::Process Update (0.9ms)[0m  [1m[33mUPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?[0m  [["last_heartbeat_at", "2024-09-06 14:28:19.246019"], ["id", 97]]
  [1m[36mTRANSACTION (0.1ms)[0m  [1m[31mROLLBACK TRANSACTION[0m
SolidQueue-0.8.2 Error in thread (0.0ms)  error: "ActiveRecord::StatementInvalid SQLite3::CorruptException: database disk image is malformed"
  [1m[35mSQL (0.1ms)[0m  [1m[36mBEGIN immediate TRANSACTION[0m
  [1m[36mSolidQueue::Process Update (1.1ms)[0m  [1m[33mUPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?[0m  [["last_heartbeat_at", "2024-09-06 14:28:19.257539"], ["id", 99]]
  [1m[36mTRANSACTION (0.0ms)[0m  [1m[31mROLLBACK TRANSACTION[0m
SolidQueue-0.8.2 Error in thread (0.0ms)  error: "ActiveRecord::StatementInvalid SQLite3::CorruptException: database disk image is malformed"
  [1m[35mSQL (1.1ms)[0m  [1m[36mBEGIN immediate TRANSACTION[0m
  [1m[36mSolidQueue::Process Update (2.9ms)[0m  [1m[33mUPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?[0m  [["last_heartbeat_at", "2024-09-06 14:29:19.091749"], ["id", 93]]
  [1m[36mTRANSACTION (0.1ms)[0m  [1m[36mCOMMIT TRANSACTION[0m
  [1m[35mSQL (0.1ms)[0m  [1m[36mBEGIN immediate TRANSACTION[0m
  [1m[36mSolidQueue::Process Update (1.4ms)[0m  [1m[33mUPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?[0m  [["last_heartbeat_at", "2024-09-06 14:29:19.245256"], ["id", 94]]
  [1m[36mTRANSACTION (0.1ms)[0m  [1m[36mCOMMIT TRANSACTION[0m
  [1m[35mSQL (2.7ms)[0m  [1m[36mBEGIN immediate TRANSACTION[0m
  [1m[36mSolidQueue::Process Update (3.8ms)[0m  [1m[33mUPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?[0m  [["last_heartbeat_at", "2024-09-06 14:29:19.245256"], ["id", 98]]
  [1m[36mTRANSACTION (0.1ms)[0m  [1m[31mROLLBACK TRANSACTION[0m
  [1m[35mSQL (4.1ms)[0m  [1m[36mBEGIN immediate TRANSACTION[0m
  [1m[36mSolidQueue::Process Update (5.3ms)[0m  [1m[33mUPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?[0m  [["last_heartbeat_at", "2024-09-06 14:29:19.250051"], ["id", 97]]
  [1m[36mTRANSACTION (0.1ms)[0m  [1m[31mROLLBACK TRANSACTION[0m
  [1m[35mSQL (12.0ms)[0m  [1m[36mBEGIN immediate TRANSACTION[0m
  [1m[36mSolidQueue::Process Update (12.9ms)[0m  [1m[33mUPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?[0m  [["last_heartbeat_at", "2024-09-06 14:29:19.247287"], ["id", 95]]
  [1m[36mTRANSACTION (0.0ms)[0m  [1m[31mROLLBACK TRANSACTION[0m
  [1m[35mSQL (9.4ms)[0m  [1m[36mBEGIN immediate TRANSACTION[0m
  [1m[36mSolidQueue::Process Update (10.4ms)[0m  [1m[33mUPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?[0m  [["last_heartbeat_at", "2024-09-06 14:29:19.252059"], ["id", 96]]
  [1m[36mTRANSACTION (0.0ms)[0m  [1m[31mROLLBACK TRANSACTION[0m
  [1m[35mSQL (0.0ms)[0m  [1m[36mBEGIN immediate TRANSACTION[0m
  [1m[36mSolidQueue::Process Update (0.8ms)[0m  [1m[33mUPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?[0m  [["last_heartbeat_at", "2024-09-06 14:29:19.265814"], ["id", 99]]
  [1m[36mTRANSACTION (0.0ms)[0m  [1m[31mROLLBACK TRANSACTION[0m
  [1m[35mSQL (0.2ms)[0m  [1m[36mBEGIN immediate TRANSACTION[0m
  [1m[36mSolidQueue::Process Update (2.3ms)[0m  [1m[33mUPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?[0m  [["last_heartbeat_at", "2024-09-06 14:30:19.101218"], ["id", 93]]
  [1m[36mTRANSACTION (0.2ms)[0m  [1m[36mCOMMIT TRANSACTION[0m
  [1m[35mSQL (0.1ms)[0m  [1m[36mBEGIN immediate TRANSACTION[0m
  [1m[36mSolidQueue::Process Update (1.3ms)[0m  [1m[33mUPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?[0m  [["last_heartbeat_at", "2024-09-06 14:30:19.255702"], ["id", 94]]
  [1m[36mTRANSACTION (0.2ms)[0m  [1m[36mCOMMIT TRANSACTION[0m
  [1m[35mSQL (0.3ms)[0m  [1m[36mBEGIN immediate TRANSACTION[0m
  [1m[36mSolidQueue::Process Update (2.6ms)[0m  [1m[33mUPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?[0m  [["last_heartbeat_at", "2024-09-06 14:30:19.257937"], ["id", 98]]
  [1m[36mTRANSACTION (0.1ms)[0m  [1m[31mROLLBACK TRANSACTION[0m
SolidQueue-0.8.2 Error in thread (0.0ms)  error: "ActiveRecord::StatementInvalid SQLite3::CorruptException: database disk image is malformed"
  [1m[35mSQL (0.2ms)[0m  [1m[36mBEGIN immediate TRANSACTION[0m
  [1m[36mSolidQueue::Process Update (2.3ms)[0m  [1m[33mUPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?[0m  [["last_heartbeat_at", "2024-09-06 14:30:19.267211"], ["id", 97]]
  [1m[36mTRANSACTION (0.1ms)[0m  [1m[31mROLLBACK TRANSACTION[0m
  [1m[35mSQL (2.7ms)[0m  [1m[36mBEGIN immediate TRANSACTION[0m
  [1m[36mSolidQueue::Process Update (4.0ms)[0m  [1m[33mUPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?[0m  [["last_heartbeat_at", "2024-09-06 14:30:19.268574"], ["id", 96]]
SolidQueue-0.8.2 Error in thread (0.0ms)  error: "ActiveRecord::StatementInvalid SQLite3::CorruptException: database disk image is malformed"
  [1m[36mTRANSACTION (0.1ms)[0m  [1m[31mROLLBACK TRANSACTION[0m
  [1m[35mSQL (3.4ms)[0m  [1m[36mBEGIN immediate TRANSACTION[0m
  [1m[36mSolidQueue::Process Update (4.2ms)[0m  [1m[33mUPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?[0m  [["last_heartbeat_at", "2024-09-06 14:30:19.270769"], ["id", 95]]
SolidQueue-0.8.2 Error in thread (0.0ms)  error: "ActiveRecord::StatementInvalid SQLite3::CorruptException: database disk image is malformed"
  [1m[36mTRANSACTION (0.0ms)[0m  [1m[31mROLLBACK TRANSACTION[0m
  [1m[35mSQL (2.7ms)[0m  [1m[36mBEGIN immediate TRANSACTION[0m
SolidQueue-0.8.2 Error in thread (0.0ms)  error: "ActiveRecord::StatementInvalid SQLite3::CorruptException: database disk image is malformed"
  [1m[36mSolidQueue::Process Update (3.9ms)[0m  [1m[33mUPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?[0m  [["last_heartbeat_at", "2024-09-06 14:30:19.273699"], ["id", 99]]
  [1m[36mTRANSACTION (0.0ms)[0m  [1m[31mROLLBACK TRANSACTION[0m
SolidQueue-0.8.2 Error in thread (0.0ms)  error: "ActiveRecord::StatementInvalid SQLite3::CorruptException: database disk image is malformed"
  [1m[35mSQL (0.1ms)[0m  [1m[36mBEGIN immediate TRANSACTION[0m
  [1m[36mSolidQueue::Process Update (0.8ms)[0m  [1m[33mUPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?[0m  [["last_heartbeat_at", "2024-09-06 14:31:19.112942"], ["id", 93]]
  [1m[36mTRANSACTION (0.0ms)[0m  [1m[36mCOMMIT TRANSACTION[0m
  [1m[35mSQL (0.0ms)[0m  [1m[36mBEGIN immediate TRANSACTION[0m
  [1m[36mSolidQueue::Process Update (0.5ms)[0m  [1m[33mUPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?[0m  [["last_heartbeat_at", "2024-09-06 14:31:19.264649"], ["id", 94]]
  [1m[36mTRANSACTION (0.0ms)[0m  [1m[36mCOMMIT TRANSACTION[0m
  [1m[35mSQL (0.1ms)[0m  [1m[36mBEGIN immediate TRANSACTION[0m
  [1m[36mSolidQueue::Process Update (1.3ms)[0m  [1m[33mUPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?[0m  [["last_heartbeat_at", "2024-09-06 14:31:19.272308"], ["id", 98]]
  [1m[36mTRANSACTION (0.0ms)[0m  [1m[31mROLLBACK TRANSACTION[0m
  [1m[35mSQL (0.1ms)[0m  [1m[36mBEGIN immediate TRANSACTION[0m
  [1m[36mSolidQueue::Process Update (1.8ms)[0m  [1m[33mUPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?[0m  [["last_heartbeat_at", "2024-09-06 14:31:19.278502"], ["id", 97]]
  [1m[36mTRANSACTION (0.0ms)[0m  [1m[31mROLLBACK TRANSACTION[0m
  [1m[35mSQL (0.0ms)[0m  [1m[36mBEGIN immediate TRANSACTION[0m
  [1m[36mSolidQueue::Process Update (0.4ms)[0m  [1m[33mUPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?[0m  [["last_heartbeat_at", "2024-09-06 14:31:19.287002"], ["id", 96]]
  [1m[36mTRANSACTION (0.0ms)[0m  [1m[31mROLLBACK TRANSACTION[0m
  [1m[35mSQL (3.5ms)[0m  [1m[36mBEGIN immediate TRANSACTION[0m
  [1m[36mSolidQueue::Process Update (4.3ms)[0m  [1m[33mUPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?[0m  [["last_heartbeat_at", "2024-09-06 14:31:19.281150"], ["id", 95]]
  [1m[36mTRANSACTION (0.1ms)[0m  [1m[31mROLLBACK TRANSACTION[0m
  [1m[35mSQL (0.0ms)[0m  [1m[36mBEGIN immediate TRANSACTION[0m
  [1m[36mSolidQueue::Process Update (0.4ms)[0m  [1m[33mUPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?[0m  [["last_heartbeat_at", "2024-09-06 14:31:19.287669"], ["id", 99]]
  [1m[36mTRANSACTION (0.0ms)[0m  [1m[31mROLLBACK TRANSACTION[0m
  [1m[36mSolidQueue::Process Load (0.2ms)[0m  [1m[34mSELECT "solid_queue_processes".* FROM "solid_queue_processes" WHERE "solid_queue_processes"."last_heartbeat_at" <= ? ORDER BY "solid_queue_processes"."id" ASC LIMIT ? [0m  [["last_heartbeat_at", "2024-09-06 14:27:19.003458"], ["LIMIT", 50]]
  [1m[35mSQL (0.1ms)[0m  [1m[36mBEGIN immediate TRANSACTION[0m
  [1m[36mSolidQueue::Process Destroy (1.3ms)[0m  [1m[31mDELETE FROM "solid_queue_processes" WHERE "solid_queue_processes"."id" = ?[0m  [["id", 8]]
  [1m[36mTRANSACTION (0.0ms)[0m  [1m[31mROLLBACK TRANSACTION[0m

That reminds me that was my hunch when I saw this a while back. I upgrade aggressively and I've been conference prepping, so I haven't seen this personally in a while. But I recall wondering if it was related to closing the laptop.

@npezza93 thanks a lot for that 🙏 It's telling that the error happens when updating the heartbeat, because the thread that runs the heartbeat task is started on the supervisor before forking 🤔

@rosa interesting! Dumb question: What would cause the touch to rollback quietly like its doing? There are no explicit rollbacks and if it was a database thing i would expect to see something raise like it is later on. Also not seeing any after_* hooks to that may trigger the rollback 🤔

Oh, not a dumb question at all, at least I hope 😅 , because I don't know! You're right that there's nothing else that could cause a rollback here, not even having an invalid record as there aren't any validations for Process. With a DB thing you should see indeed the error logged as it happens later, or like here:

  TRANSACTION (0.1ms)  begin transaction
  TRANSACTION (0.0ms)  begin transaction
  TRANSACTION (0.1ms)  begin transaction
  SolidQueue::Process Update (5180.6ms)  UPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?  [["last_heartbeat_at", "2024-09-06 19:32:45.712124"], ["id", 51]]
  TRANSACTION (0.1ms)  rollback transaction
SolidQueue-0.8.2 Error in thread (0.0ms)  error: "ActiveRecord::StatementInvalid SQLite3::BusyException: database is locked"
  SolidQueue::Job Update (7631.3ms)  UPDATE "solid_queue_jobs" SET "updated_at" = ?, "finished_at" = ? WHERE "solid_queue_jobs"."id" = ?  [["updated_at", "2024-09-06 19:32:33.311406"], ["finished_at", "2024-09-06 19:32:33.311406"], ["id", 31]]
  SolidQueue::ClaimedExecution Destroy (0.1ms)  DELETE FROM "solid_queue_claimed_executions" WHERE "solid_queue_claimed_executions"."id" = ?  [["id", 66]]
  SolidQueue::Job Update (15556.6ms)  UPDATE "solid_queue_jobs" SET "updated_at" = ?, "finished_at" = ? WHERE "solid_queue_jobs"."id" = ?  [["updated_at", "2024-09-06 19:32:35.742560"], ["finished_at", "2024-09-06 19:32:35.742560"], ["id", 35]]
  TRANSACTION (0.1ms)  begin transaction
  TRANSACTION (0.1ms)  rollback transaction
  TRANSACTION (0.3ms)  commit transaction
  TRANSACTION (0.0ms)  begin transaction
  TRANSACTION (0.2ms)  begin transaction
  TRANSACTION (0.3ms)  begin transaction
  SolidQueue::Job Update (5196.6ms)  UPDATE "solid_queue_jobs" SET "updated_at" = ?, "finished_at" = ? WHERE "solid_queue_jobs"."id" = ?  [["updated_at", "2024-09-06 19:32:32.378991"], ["finished_at", "2024-09-06 19:32:32.378991"], ["id", 26]]
  TRANSACTION (0.0ms)  begin transaction
  SolidQueue::Job Update (10390.5ms)  UPDATE "solid_queue_jobs" SET "updated_at" = ?, "finished_at" = ? WHERE "solid_queue_jobs"."id" = ?  [["updated_at", "2024-09-06 19:32:35.729882"], ["finished_at", "2024-09-06 19:32:35.729882"], ["id", 30]]
  TRANSACTION (0.0ms)  rollback transaction
  SolidQueue::Job Update (15552.8ms)  UPDATE "solid_queue_jobs" SET "updated_at" = ?, "finished_at" = ? WHERE "solid_queue_jobs"."id" = ?  [["updated_at", "2024-09-06 19:32:40.950934"], ["finished_at", "2024-09-06 19:32:40.950934"], ["id", 34]]
SolidQueue-0.8.2 Error in thread (0.0ms)  error: "ActiveRecord::StatementInvalid SQLite3::BusyException: database is locked"
  SolidQueue::Process Update (13.3ms)  UPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?  [["last_heartbeat_at", "2024-09-06 19:32:51.300728"], ["id", 49]]
  TRANSACTION (0.0ms)  rollback transaction
SolidQueue-0.8.2 Error in thread (0.0ms)  error: "ActiveRecord::StatementInvalid SQLite3::BusyException: database is locked"
SolidQueue-0.8.2 Error in thread (0.0ms)  error: "ActiveRecord::StatementInvalid SQLite3::BusyException: database is locked"
  TRANSACTION (0.1ms)  commit transaction
  TRANSACTION (0.0ms)  rollback transaction

And another strange thing is that you still have transactions that commit successfully after you start getting the corrupt DB errors. I'd expect that nothing would work after that. Maybe the logs are very interleaved and disordered? But the values for last_heartbeat_at don't suggest that, they seem to be in order.

A question: were you running this via Puma or bin/jobs? And are your logs copied from log/development.log? And would you mind sharing your solid queue configuration? Just to know the number of processes and threads and configure @fractaledmind's app with the same.

were you running this via Puma or bin/jobs?

puma

And are your logs copied from log/development.log?

yup!

config

development:
  dispatchers:
    - polling_interval: 5
      batch_size: 500
  workers:
    - queues: "*"
      threads: 5
      processes: 5
      polling_interval: 2

production:
  dispatchers:
    - polling_interval: 1
      batch_size: 500
  workers:
    - queues: "*"
      threads: 6
      processes: 1
      polling_interval: 0.1

AY OH. Got it to reliably show the rollback. Doesnt seem to be corrupting yet but progress :)

Changed the config to:

development:
  dispatchers:
    - polling_interval: 0.5
      batch_size: 500
  workers:
    - queues: "*"
      threads: 5
      processes: 5
      polling_interval: 0.5

production:
  dispatchers:
    - polling_interval: 1
      batch_size: 500
  workers:
    - queues: "*"
      threads: 6
      processes: 1
      polling_interval: 0.1

and set config.solid_queue.process_heartbeat_interval = 1 cause im impatient.

Then start up the server
Go into the console and run: SolidQueue::Process.delete_all

Go into the console and run: SolidQueue::Process.delete_all

Ah! This would indeed cause the rollback because the processes would have a Process object that's been deleted under them. However, I can't imagine SQLite getting corrupted for that reason. There must be something else...

I've also cleaned up the logs from before a bit, and noticed something interesting too:

Turbo::StreamsChannel is transmitting the subscription confirmation
Turbo::StreamsChannel is streaming from Z2lkOi8vcWlyYS9TZWN1cml0eS80Nw:chart:1D
Unsubscribing from channel: {"channel":"Turbo::StreamsChannel","signed_stream_name":"IloybGtPaTh2Y1dseVlTOVRaV04xY21sMGVTODBOdzpjaGFydDoxRCI=--6b7a8177c089474a55c40174ae74943ecba61bc5effaf1d922cc00e95ded31a4"}
Turbo::StreamsChannel stopped streaming from Z2lkOi8vcWlyYS9TZWN1cml0eS80Nw:chart:1D
Turbo::StreamsChannel is transmitting the subscription confirmation
Turbo::StreamsChannel is streaming from Z2lkOi8vcWlyYS9TZWN1cml0eS80Nw:chart:1D
  SQL (0.2ms)  BEGIN immediate TRANSACTION
  SolidQueue::Process Update (3.9ms)  UPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?  [["last_heartbeat_at", "2024-09-06 14:20:18.968432"], ["id", 93]]
  TRANSACTION (0.1ms)  COMMIT TRANSACTION

  SQL (0.1ms)  BEGIN immediate TRANSACTION
  SolidQueue::Process Update (1.1ms)  UPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?  [["last_heartbeat_at", "2024-09-06 14:20:19.144763"], ["id", 95]]
  TRANSACTION (0.0ms)  ROLLBACK TRANSACTION
  SQL (0.0ms)  BEGIN immediate TRANSACTION
  SolidQueue::Process Update (0.6ms)  UPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?  [["last_heartbeat_at", "2024-09-06 14:20:19.164246"], ["id", 96]]
  TRANSACTION (0.0ms)  ROLLBACK TRANSACTION
  SQL (0.0ms)  BEGIN immediate TRANSACTION
  SolidQueue::Process Update (0.7ms)  UPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?  [["last_heartbeat_at", "2024-09-06 14:20:19.169932"], ["id", 98]]
  TRANSACTION (0.0ms)  ROLLBACK TRANSACTION
  SQL (8.7ms)  BEGIN immediate TRANSACTION
  SolidQueue::Process Update (9.5ms)  UPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?  [["last_heartbeat_at", "2024-09-06 14:20:19.164249"], ["id", 97]]
  TRANSACTION (0.0ms)  ROLLBACK TRANSACTION
  SQL (13.8ms)  BEGIN immediate TRANSACTION
  SolidQueue::Process Update (15.1ms)  UPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?  [["last_heartbeat_at", "2024-09-06 14:20:19.164685"], ["id", 94]]
  TRANSACTION (0.1ms)  COMMIT TRANSACTION

  SQL (0.1ms)  BEGIN immediate TRANSACTION
  SolidQueue::Process Update (1.7ms)  UPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?  [["last_heartbeat_at", "2024-09-06 14:20:19.196651"], ["id", 99]]
  TRANSACTION (0.0ms)  ROLLBACK TRANSACTION
SolidQueue-0.8.2 Error in thread (0.0ms)  error: "ActiveRecord::StatementInvalid SQLite3::CorruptException: database disk image is malformed"
  SQL (0.0ms)  BEGIN immediate TRANSACTION
  SolidQueue::Process Update (0.5ms)  UPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?  [["last_heartbeat_at", "2024-09-06 14:21:19.012875"], ["id", 93]]
  TRANSACTION (0.0ms)  COMMIT TRANSACTION

  SQL (0.0ms)  BEGIN immediate TRANSACTION
  SolidQueue::Process Update (0.7ms)  UPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?  [["last_heartbeat_at", "2024-09-06 14:21:19.160688"], ["id", 95]]
  TRANSACTION (0.0ms)  ROLLBACK TRANSACTION
SolidQueue-0.8.2 Error in thread (0.0ms)  error: "ActiveRecord::StatementInvalid SQLite3::CorruptException: database disk image is malformed"
  SQL (0.0ms)  BEGIN immediate TRANSACTION
  SolidQueue::Process Update (0.6ms)  UPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?  [["last_heartbeat_at", "2024-09-06 14:21:19.176302"], ["id", 96]]
  TRANSACTION (0.0ms)  ROLLBACK TRANSACTION
  SQL (0.0ms)  BEGIN immediate TRANSACTION
  SolidQueue::Process Update (1.1ms)  UPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?  [["last_heartbeat_at", "2024-09-06 14:21:19.177137"], ["id", 98]]
SolidQueue-0.8.2 Error in thread (0.0ms)  error: "ActiveRecord::StatementInvalid SQLite3::CorruptException: database disk image is malformed"
  TRANSACTION (0.0ms)  ROLLBACK TRANSACTION
SolidQueue-0.8.2 Error in thread (0.0ms)  error: "ActiveRecord::StatementInvalid SQLite3::CorruptException: database disk image is malformed"
  SQL (0.0ms)  BEGIN immediate TRANSACTION
  SolidQueue::Process Update (0.8ms)  UPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?  [["last_heartbeat_at", "2024-09-06 14:21:19.184487"], ["id", 94]]
  TRANSACTION (0.0ms)  COMMIT TRANSACTION

  SQL (3.6ms)  BEGIN immediate TRANSACTION
  SolidQueue::Process Update (4.3ms)  UPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?  [["last_heartbeat_at", "2024-09-06 14:21:19.184518"], ["id", 97]]
  TRANSACTION (0.0ms)  ROLLBACK TRANSACTION
  SQL (0.1ms)  BEGIN immediate TRANSACTION
  SolidQueue::Process Update (1.0ms)  UPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?  [["last_heartbeat_at", "2024-09-06 14:21:19.201697"], ["id", 99]]
  TRANSACTION (0.0ms)  ROLLBACK TRANSACTION
  SolidQueue::Process Load (0.1ms)  [34mSELECT "solid_queue_processes".* FROM "solid_queue_processes" WHERE "solid_queue_processes"."last_heartbeat_at" <= ? ORDER BY "solid_queue_processes"."id" ASC LIMIT ?   [["last_heartbeat_at", "2024-09-06 14:17:18.947007"], ["LIMIT", 50]]
  SQL (0.1ms)  BEGIN immediate TRANSACTION
  SolidQueue::Process Destroy (1.1ms)  DELETE FROM "solid_queue_processes" WHERE "solid_queue_processes"."id" = ?  [["id", 8]]
  TRANSACTION (0.0ms)  ROLLBACK TRANSACTION
SolidQueue-0.8.2 Error deregistering Supervisor (4.8ms)  process_id: 8, pid: 87668, hostname: "Nicks-MacBook-Air.local", name: "supervisor-fb2dc5791db13f37a778", last_heartbeat_at: "2024-09-05T23:20:38-04:00", claimed_size: nil, pruned: true, error: "ActiveRecord::StatementInvalid SQLite3::CorruptException: database disk image is malformed"
SolidQueue-0.8.2 Prune dead processes (30.5ms)  size: 13
SolidQueue-0.8.2 Error in thread (0.0ms)  error: "ActiveRecord::StatementInvalid SQLite3::CorruptException: database disk image is malformed"
  SQL (0.0ms)  BEGIN immediate TRANSACTION
  SolidQueue::Process Update (2.0ms)  UPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?  [["last_heartbeat_at", "2024-09-06 14:22:19.021586"], ["id", 93]]
  TRANSACTION (0.0ms)  COMMIT TRANSACTION

  SQL (0.0ms)  BEGIN immediate TRANSACTION
  SolidQueue::Process Update (1.4ms)  UPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?  [["last_heartbeat_at", "2024-09-06 14:22:19.165340"], ["id", 95]]
  TRANSACTION (0.0ms)  ROLLBACK TRANSACTION
  SQL (0.0ms)  BEGIN immediate TRANSACTION
  SolidQueue::Process Update (0.5ms)  UPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?  [["last_heartbeat_at", "2024-09-06 14:22:19.181037"], ["id", 96]]
  TRANSACTION (0.0ms)  ROLLBACK TRANSACTION
  SQL (0.0ms)  BEGIN immediate TRANSACTION
  SolidQueue::Process Update (0.9ms)  UPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?  [["last_heartbeat_at", "2024-09-06 14:22:19.182902"], ["id", 98]]
  TRANSACTION (0.0ms)  ROLLBACK TRANSACTION
  SQL (0.0ms)  BEGIN immediate TRANSACTION
  SolidQueue::Process Update (0.5ms)  UPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?  [["last_heartbeat_at", "2024-09-06 14:22:19.192213"], ["id", 94]]
  TRANSACTION (0.1ms)  COMMIT TRANSACTION

  SQL (0.0ms)  BEGIN immediate TRANSACTION
  SolidQueue::Process Update (1.4ms)  UPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?  [["last_heartbeat_at", "2024-09-06 14:22:19.194242"], ["id", 97]]
  TRANSACTION (0.0ms)  ROLLBACK TRANSACTION
SolidQueue-0.8.2 Error in thread (0.0ms)  error: "ActiveRecord::StatementInvalid SQLite3::CorruptException: database disk image is malformed"
  SQL (0.0ms)  BEGIN immediate TRANSACTION
  SolidQueue::Process Update (1.1ms)  UPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?  [["last_heartbeat_at", "2024-09-06 14:22:19.210680"], ["id", 99]]
  TRANSACTION (0.0ms)  ROLLBACK TRANSACTION
SolidQueue-0.8.2 Error in thread (0.0ms)  error: "ActiveRecord::StatementInvalid SQLite3::CorruptException: database disk image is malformed"
  SQL (0.0ms)  BEGIN immediate TRANSACTION
  SolidQueue::Process Update (1.2ms)  UPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?  [["last_heartbeat_at", "2024-09-06 14:23:19.027648"], ["id", 93]]
  TRANSACTION (0.0ms)  COMMIT TRANSACTION

  SQL (0.0ms)  BEGIN immediate TRANSACTION
  SolidQueue::Process Update (1.3ms)  UPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?  [["last_heartbeat_at", "2024-09-06 14:23:19.173473"], ["id", 95]]
  TRANSACTION (0.0ms)  ROLLBACK TRANSACTION
SolidQueue-0.8.2 Error in thread (0.0ms)  error: "ActiveRecord::StatementInvalid SQLite3::CorruptException: database disk image is malformed"
  SQL (0.0ms)  BEGIN immediate TRANSACTION
  SolidQueue::Process Update (0.4ms)  UPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?  [["last_heartbeat_at", "2024-09-06 14:23:19.184874"], ["id", 96]]
  TRANSACTION (0.0ms)  ROLLBACK TRANSACTION
SolidQueue-0.8.2 Error in thread (0.0ms)  error: "ActiveRecord::StatementInvalid SQLite3::CorruptException: database disk image is malformed"
  SQL (0.0ms)  BEGIN immediate TRANSACTION
  SolidQueue::Process Update (0.5ms)  UPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?  [["last_heartbeat_at", "2024-09-06 14:23:19.187570"], ["id", 98]]
  TRANSACTION (0.0ms)  ROLLBACK TRANSACTION
SolidQueue-0.8.2 Error in thread (0.0ms)  error: "ActiveRecord::StatementInvalid SQLite3::CorruptException: database disk image is malformed"
  SQL (0.0ms)  BEGIN immediate TRANSACTION
  SolidQueue::Process Update (0.4ms)  UPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?  [["last_heartbeat_at", "2024-09-06 14:23:19.199510"], ["id", 94]]
  TRANSACTION (0.1ms)  COMMIT TRANSACTION

  SQL (0.0ms)  BEGIN immediate TRANSACTION
  SolidQueue::Process Update (0.4ms)  UPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?  [["last_heartbeat_at", "2024-09-06 14:23:19.202226"], ["id", 97]]
  TRANSACTION (0.0ms)  ROLLBACK TRANSACTION
SolidQueue-0.8.2 Error in thread (0.0ms)  error: "ActiveRecord::StatementInvalid SQLite3::CorruptException: database disk image is malformed"
  SQL (0.0ms)  BEGIN immediate TRANSACTION
  SolidQueue::Process Update (0.4ms)  UPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?  [["last_heartbeat_at", "2024-09-06 14:23:19.214210"], ["id", 99]]
  TRANSACTION (0.0ms)  ROLLBACK TRANSACTION
SolidQueue-0.8.2 Error in thread (0.0ms)  error: "ActiveRecord::StatementInvalid SQLite3::CorruptException: database disk image is malformed"
  SQL (0.1ms)  BEGIN immediate TRANSACTION
  SolidQueue::Process Update (1.2ms)  UPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?  [["last_heartbeat_at", "2024-09-06 14:24:19.033318"], ["id", 93]]
  TRANSACTION (0.0ms)  COMMIT TRANSACTION

  SQL (0.1ms)  BEGIN immediate TRANSACTION
  SolidQueue::Process Update (2.1ms)  UPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?  [["last_heartbeat_at", "2024-09-06 14:24:19.181729"], ["id", 95]]
  TRANSACTION (0.1ms)  ROLLBACK TRANSACTION
  SQL (0.1ms)  BEGIN immediate TRANSACTION
SolidQueue-0.8.2 Error in thread (0.0ms)  error: "ActiveRecord::StatementInvalid SQLite3::CorruptException: database disk image is malformed"
  SolidQueue::Process Update (1.8ms)  UPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?  [["last_heartbeat_at", "2024-09-06 14:24:19.187835"], ["id", 96]]
  TRANSACTION (0.1ms)  ROLLBACK TRANSACTION
  SQL (0.0ms)  BEGIN immediate TRANSACTION
SolidQueue-0.8.2 Error in thread (0.0ms)  error: "ActiveRecord::StatementInvalid SQLite3::CorruptException: database disk image is malformed"
  SolidQueue::Process Update (0.7ms)  UPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?  [["last_heartbeat_at", "2024-09-06 14:24:19.194634"], ["id", 98]]
  TRANSACTION (0.0ms)  ROLLBACK TRANSACTION
SolidQueue-0.8.2 Error in thread (0.0ms)  error: "ActiveRecord::StatementInvalid SQLite3::CorruptException: database disk image is malformed"
  SQL (0.0ms)  BEGIN immediate TRANSACTION
  SolidQueue::Process Update (0.9ms)  UPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?  [["last_heartbeat_at", "2024-09-06 14:24:19.205474"], ["id", 94]]
  TRANSACTION (0.1ms)  COMMIT TRANSACTION

  SQL (1.1ms)  BEGIN immediate TRANSACTION
  SolidQueue::Process Update (2.5ms)  UPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?  [["last_heartbeat_at", "2024-09-06 14:24:19.206692"], ["id", 97]]
  TRANSACTION (0.0ms)  ROLLBACK TRANSACTION
  SQL (0.0ms)  BEGIN immediate TRANSACTION
  SolidQueue::Process Update (1.0ms)  UPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?  [["last_heartbeat_at", "2024-09-06 14:24:19.219473"], ["id", 99]]
  TRANSACTION (0.0ms)  ROLLBACK TRANSACTION
  SQL (0.2ms)  BEGIN immediate TRANSACTION
  SolidQueue::Process Update (3.2ms)  UPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?  [["last_heartbeat_at", "2024-09-06 14:25:19.053217"], ["id", 93]]
  TRANSACTION (0.1ms)  COMMIT TRANSACTION

  SQL (0.1ms)  BEGIN immediate TRANSACTION
  SolidQueue::Process Update (3.1ms)  UPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?  [["last_heartbeat_at", "2024-09-06 14:25:19.195502"], ["id", 95]]
  TRANSACTION (0.0ms)  ROLLBACK TRANSACTION
  SQL (1.3ms)  BEGIN immediate TRANSACTION
  SolidQueue::Process Update (2.4ms)  UPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?  [["last_heartbeat_at", "2024-09-06 14:25:19.212315"], ["id", 94]]
  TRANSACTION (0.2ms)  COMMIT TRANSACTION

  SQL (2.6ms)  BEGIN immediate TRANSACTION
  SolidQueue::Process Update (15.1ms)  UPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?  [["last_heartbeat_at", "2024-09-06 14:25:19.201020"], ["id", 98]]
  TRANSACTION (0.0ms)  ROLLBACK TRANSACTION
  SQL (16.3ms)  BEGIN immediate TRANSACTION
SolidQueue-0.8.2 Error in thread (0.0ms)  error: "ActiveRecord::StatementInvalid SQLite3::CorruptException: database disk image is malformed"
  SolidQueue::Process Update (17.1ms)  UPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?  [["last_heartbeat_at", "2024-09-06 14:25:19.201014"], ["id", 96]]
  TRANSACTION (0.0ms)  ROLLBACK TRANSACTION
SolidQueue-0.8.2 Error in thread (0.0ms)  error: "ActiveRecord::StatementInvalid SQLite3::CorruptException: database disk image is malformed"
  SQL (7.6ms)  BEGIN immediate TRANSACTION
  SolidQueue::Process Update (13.9ms)  UPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?  [["last_heartbeat_at", "2024-09-06 14:25:19.218068"], ["id", 97]]
  TRANSACTION (0.0ms)  ROLLBACK TRANSACTION
  SQL (3.9ms)  BEGIN immediate TRANSACTION
SolidQueue-0.8.2 Error in thread (0.0ms)  error: "ActiveRecord::StatementInvalid SQLite3::CorruptException: database disk image is malformed"
  SolidQueue::Process Update (10.3ms)  UPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?  [["last_heartbeat_at", "2024-09-06 14:25:19.228835"], ["id", 99]]
  TRANSACTION (0.0ms)  ROLLBACK TRANSACTION
SolidQueue-0.8.2 Error in thread (0.0ms)  error: "ActiveRecord::StatementInvalid SQLite3::CorruptException: database disk image is malformed"
  SQL (0.0ms)  BEGIN immediate TRANSACTION
  SolidQueue::Process Update (0.6ms)  UPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?  [["last_heartbeat_at", "2024-09-06 14:26:19.065500"], ["id", 93]]
  TRANSACTION (0.0ms)  COMMIT TRANSACTION

  SQL (0.0ms)  BEGIN immediate TRANSACTION
  SolidQueue::Process Update (0.5ms)  UPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?  [["last_heartbeat_at", "2024-09-06 14:26:19.217067"], ["id", 95]]
  TRANSACTION (0.0ms)  ROLLBACK TRANSACTION
SolidQueue-0.8.2 Error in thread (0.0ms)  error: "ActiveRecord::StatementInvalid SQLite3::CorruptException: database disk image is malformed"
  SQL (0.0ms)  BEGIN immediate TRANSACTION
  SolidQueue::Process Update (0.4ms)  UPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?  [["last_heartbeat_at", "2024-09-06 14:26:19.218612"], ["id", 94]]
  TRANSACTION (0.0ms)  COMMIT TRANSACTION

  SQL (0.0ms)  BEGIN immediate TRANSACTION
  SolidQueue::Process Update (1.1ms)  UPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?  [["last_heartbeat_at", "2024-09-06 14:26:19.223827"], ["id", 98]]
  TRANSACTION (0.0ms)  ROLLBACK TRANSACTION
  SQL (0.0ms)  BEGIN immediate TRANSACTION
  SolidQueue::Process Update (0.6ms)  UPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?  [["last_heartbeat_at", "2024-09-06 14:26:19.226618"], ["id", 96]]
  TRANSACTION (0.0ms)  ROLLBACK TRANSACTION
  SQL (0.0ms)  BEGIN immediate TRANSACTION
  SolidQueue::Process Update (0.4ms)  UPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?  [["last_heartbeat_at", "2024-09-06 14:26:19.234613"], ["id", 97]]
  TRANSACTION (0.0ms)  ROLLBACK TRANSACTION
  SQL (0.0ms)  BEGIN immediate TRANSACTION
  SolidQueue::Process Update (0.5ms)  UPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?  [["last_heartbeat_at", "2024-09-06 14:26:19.242735"], ["id", 99]]
  TRANSACTION (0.0ms)  ROLLBACK TRANSACTION
  SolidQueue::Process Load (0.3ms)  [34mSELECT "solid_queue_processes".* FROM "solid_queue_processes" WHERE "solid_queue_processes"."last_heartbeat_at" <= ? ORDER BY "solid_queue_processes"."id" ASC LIMIT ?   [["last_heartbeat_at", "2024-09-06 14:22:18.983585"], ["LIMIT", 50]]
  SQL (0.1ms)  BEGIN immediate TRANSACTION
  SolidQueue::Process Destroy (1.5ms)  DELETE FROM "solid_queue_processes" WHERE "solid_queue_processes"."id" = ?  [["id", 8]]
  TRANSACTION (0.0ms)  ROLLBACK TRANSACTION
SolidQueue-0.8.2 Error deregistering Supervisor (5.5ms)  process_id: 8, pid: 87668, hostname: "Nicks-MacBook-Air.local", name: "supervisor-fb2dc5791db13f37a778", last_heartbeat_at: "2024-09-05T23:20:38-04:00", claimed_size: nil, pruned: true, error: "ActiveRecord::StatementInvalid SQLite3::CorruptException: database disk image is malformed"
SolidQueue-0.8.2 Prune dead processes (13.7ms)  size: 13
SolidQueue-0.8.2 Error in thread (0.0ms)  error: "ActiveRecord::StatementInvalid SQLite3::CorruptException: database disk image is malformed"
  SQL (0.1ms)  BEGIN immediate TRANSACTION
  SolidQueue::Process Update (1.8ms)  UPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?  [["last_heartbeat_at", "2024-09-06 14:27:19.069270"], ["id", 93]]
  TRANSACTION (0.1ms)  COMMIT TRANSACTION

  SolidQueue::Semaphore Pluck (0.2ms)  [34mSELECT "solid_queue_semaphores"."id" FROM "solid_queue_semaphores" WHERE "solid_queue_semaphores"."expires_at" < ? ORDER BY "solid_queue_semaphores"."id" ASC LIMIT ?  [["expires_at", "2024-09-06 14:27:19.140496"], ["LIMIT", 500]]
  SolidQueue::BlockedExecution Pluck (0.2ms)  [34mSELECT DISTINCT "solid_queue_blocked_executions"."concurrency_key" FROM "solid_queue_blocked_executions" WHERE "solid_queue_blocked_executions"."expires_at" < ? LIMIT ?  [["expires_at", "2024-09-06 14:27:19.143734"], ["LIMIT", 500]]
SolidQueue-0.8.2 Unblock jobs (1.9ms)  limit: 500, size: 0
  SQL (0.1ms)  BEGIN immediate TRANSACTION
  SolidQueue::Process Update (1.5ms)  UPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?  [["last_heartbeat_at", "2024-09-06 14:27:19.221337"], ["id", 95]]
  TRANSACTION (0.1ms)  ROLLBACK TRANSACTION
  SQL (2.7ms)  BEGIN immediate TRANSACTION
  SolidQueue::Process Update (3.7ms)  UPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?  [["last_heartbeat_at", "2024-09-06 14:27:19.224777"], ["id", 94]]
  TRANSACTION (0.1ms)  COMMIT TRANSACTION

  SQL (1.4ms)  BEGIN immediate TRANSACTION
  SolidQueue::Process Update (2.8ms)  UPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?  [["last_heartbeat_at", "2024-09-06 14:27:19.229198"], ["id", 98]]
  TRANSACTION (0.0ms)  ROLLBACK TRANSACTION
SolidQueue-0.8.2 Error in thread (0.0ms)  error: "ActiveRecord::StatementInvalid SQLite3::CorruptException: database disk image is malformed"
  SQL (0.2ms)  BEGIN immediate TRANSACTION
  SolidQueue::Process Update (1.3ms)  UPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?  [["last_heartbeat_at", "2024-09-06 14:27:19.233770"], ["id", 96]]
  TRANSACTION (0.0ms)  ROLLBACK TRANSACTION
SolidQueue-0.8.2 Error in thread (0.0ms)  error: "ActiveRecord::StatementInvalid SQLite3::CorruptException: database disk image is malformed"
  SQL (0.1ms)  BEGIN immediate TRANSACTION
  SolidQueue::Process Update (1.0ms)  UPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?  [["last_heartbeat_at", "2024-09-06 14:27:19.241697"], ["id", 97]]
  TRANSACTION (0.0ms)  ROLLBACK TRANSACTION
SolidQueue-0.8.2 Error in thread (0.0ms)  error: "ActiveRecord::StatementInvalid SQLite3::CorruptException: database disk image is malformed"
  SQL (0.1ms)  BEGIN immediate TRANSACTION
  SolidQueue::Process Update (1.2ms)  UPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?  [["last_heartbeat_at", "2024-09-06 14:27:19.250172"], ["id", 99]]
  TRANSACTION (0.0ms)  ROLLBACK TRANSACTION
SolidQueue-0.8.2 Error in thread (0.0ms)  error: "ActiveRecord::StatementInvalid SQLite3::CorruptException: database disk image is malformed"
  SQL (1.3ms)  BEGIN immediate TRANSACTION
  SolidQueue::Process Update (3.2ms)  UPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?  [["last_heartbeat_at", "2024-09-06 14:28:19.080873"], ["id", 93]]
  TRANSACTION (0.2ms)  COMMIT TRANSACTION

  SQL (0.1ms)  BEGIN immediate TRANSACTION
  SolidQueue::Process Update (1.7ms)  UPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?  [["last_heartbeat_at", "2024-09-06 14:28:19.234057"], ["id", 94]]
  SQL (2.9ms)  BEGIN immediate TRANSACTION
  TRANSACTION (0.1ms)  COMMIT TRANSACTION

  SolidQueue::Process Update (4.7ms)  UPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?  [["last_heartbeat_at", "2024-09-06 14:28:19.234057"], ["id", 95]]
  TRANSACTION (0.0ms)  ROLLBACK TRANSACTION
  SQL (0.1ms)  BEGIN immediate TRANSACTION
SolidQueue-0.8.2 Error in thread (0.0ms)  error: "ActiveRecord::StatementInvalid SQLite3::CorruptException: database disk image is malformed"
  SolidQueue::Process Update (1.0ms)  UPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?  [["last_heartbeat_at", "2024-09-06 14:28:19.240062"], ["id", 98]]
  TRANSACTION (0.0ms)  ROLLBACK TRANSACTION
  SQL (1.3ms)  BEGIN immediate TRANSACTION
SolidQueue-0.8.2 Error in thread (0.0ms)  error: "ActiveRecord::StatementInvalid SQLite3::CorruptException: database disk image is malformed"
  SolidQueue::Process Update (2.7ms)  UPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?  [["last_heartbeat_at", "2024-09-06 14:28:19.240870"], ["id", 96]]
  TRANSACTION (0.0ms)  ROLLBACK TRANSACTION
SolidQueue-0.8.2 Error in thread (0.0ms)  error: "ActiveRecord::StatementInvalid SQLite3::CorruptException: database disk image is malformed"
  SQL (0.0ms)  BEGIN immediate TRANSACTION
  SolidQueue::Process Update (0.9ms)  UPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?  [["last_heartbeat_at", "2024-09-06 14:28:19.246019"], ["id", 97]]
  TRANSACTION (0.1ms)  ROLLBACK TRANSACTION
SolidQueue-0.8.2 Error in thread (0.0ms)  error: "ActiveRecord::StatementInvalid SQLite3::CorruptException: database disk image is malformed"
  SQL (0.1ms)  BEGIN immediate TRANSACTION
  SolidQueue::Process Update (1.1ms)  UPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?  [["last_heartbeat_at", "2024-09-06 14:28:19.257539"], ["id", 99]]
  TRANSACTION (0.0ms)  ROLLBACK TRANSACTION
SolidQueue-0.8.2 Error in thread (0.0ms)  error: "ActiveRecord::StatementInvalid SQLite3::CorruptException: database disk image is malformed"
  SQL (1.1ms)  BEGIN immediate TRANSACTION
  SolidQueue::Process Update (2.9ms)  UPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?  [["last_heartbeat_at", "2024-09-06 14:29:19.091749"], ["id", 93]]
  TRANSACTION (0.1ms)  COMMIT TRANSACTION

  SQL (0.1ms)  BEGIN immediate TRANSACTION
  SolidQueue::Process Update (1.4ms)  UPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?  [["last_heartbeat_at", "2024-09-06 14:29:19.245256"], ["id", 94]]
  TRANSACTION (0.1ms)  COMMIT TRANSACTION

  SQL (2.7ms)  BEGIN immediate TRANSACTION
  SolidQueue::Process Update (3.8ms)  UPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?  [["last_heartbeat_at", "2024-09-06 14:29:19.245256"], ["id", 98]]
  TRANSACTION (0.1ms)  ROLLBACK TRANSACTION
  SQL (4.1ms)  BEGIN immediate TRANSACTION
  SolidQueue::Process Update (5.3ms)  UPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?  [["last_heartbeat_at", "2024-09-06 14:29:19.250051"], ["id", 97]]
  TRANSACTION (0.1ms)  ROLLBACK TRANSACTION
  SQL (12.0ms)  BEGIN immediate TRANSACTION
  SolidQueue::Process Update (12.9ms)  UPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?  [["last_heartbeat_at", "2024-09-06 14:29:19.247287"], ["id", 95]]
  TRANSACTION (0.0ms)  ROLLBACK TRANSACTION
  SQL (9.4ms)  BEGIN immediate TRANSACTION
  SolidQueue::Process Update (10.4ms)  UPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?  [["last_heartbeat_at", "2024-09-06 14:29:19.252059"], ["id", 96]]
  TRANSACTION (0.0ms)  ROLLBACK TRANSACTION
  SQL (0.0ms)  BEGIN immediate TRANSACTION
  SolidQueue::Process Update (0.8ms)  UPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?  [["last_heartbeat_at", "2024-09-06 14:29:19.265814"], ["id", 99]]
  TRANSACTION (0.0ms)  ROLLBACK TRANSACTION
  SQL (0.2ms)  BEGIN immediate TRANSACTION
  SolidQueue::Process Update (2.3ms)  UPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?  [["last_heartbeat_at", "2024-09-06 14:30:19.101218"], ["id", 93]]
  TRANSACTION (0.2ms)  COMMIT TRANSACTION

  SQL (0.1ms)  BEGIN immediate TRANSACTION
  SolidQueue::Process Update (1.3ms)  UPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?  [["last_heartbeat_at", "2024-09-06 14:30:19.255702"], ["id", 94]]
  TRANSACTION (0.2ms)  COMMIT TRANSACTION

  SQL (0.3ms)  BEGIN immediate TRANSACTION
  SolidQueue::Process Update (2.6ms)  UPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?  [["last_heartbeat_at", "2024-09-06 14:30:19.257937"], ["id", 98]]
  TRANSACTION (0.1ms)  ROLLBACK TRANSACTION
SolidQueue-0.8.2 Error in thread (0.0ms)  error: "ActiveRecord::StatementInvalid SQLite3::CorruptException: database disk image is malformed"
  SQL (0.2ms)  BEGIN immediate TRANSACTION
  SolidQueue::Process Update (2.3ms)  UPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?  [["last_heartbeat_at", "2024-09-06 14:30:19.267211"], ["id", 97]]
  TRANSACTION (0.1ms)  ROLLBACK TRANSACTION
  SQL (2.7ms)  BEGIN immediate TRANSACTION
  SolidQueue::Process Update (4.0ms)  UPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?  [["last_heartbeat_at", "2024-09-06 14:30:19.268574"], ["id", 96]]
SolidQueue-0.8.2 Error in thread (0.0ms)  error: "ActiveRecord::StatementInvalid SQLite3::CorruptException: database disk image is malformed"
  TRANSACTION (0.1ms)  ROLLBACK TRANSACTION
  SQL (3.4ms)  BEGIN immediate TRANSACTION
  SolidQueue::Process Update (4.2ms)  UPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?  [["last_heartbeat_at", "2024-09-06 14:30:19.270769"], ["id", 95]]
SolidQueue-0.8.2 Error in thread (0.0ms)  error: "ActiveRecord::StatementInvalid SQLite3::CorruptException: database disk image is malformed"
  TRANSACTION (0.0ms)  ROLLBACK TRANSACTION
  SQL (2.7ms)  BEGIN immediate TRANSACTION
SolidQueue-0.8.2 Error in thread (0.0ms)  error: "ActiveRecord::StatementInvalid SQLite3::CorruptException: database disk image is malformed"
  SolidQueue::Process Update (3.9ms)  UPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?  [["last_heartbeat_at", "2024-09-06 14:30:19.273699"], ["id", 99]]
  TRANSACTION (0.0ms)  ROLLBACK TRANSACTION
SolidQueue-0.8.2 Error in thread (0.0ms)  error: "ActiveRecord::StatementInvalid SQLite3::CorruptException: database disk image is malformed"
  SQL (0.1ms)  BEGIN immediate TRANSACTION
  SolidQueue::Process Update (0.8ms)  UPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?  [["last_heartbeat_at", "2024-09-06 14:31:19.112942"], ["id", 93]]
  TRANSACTION (0.0ms)  COMMIT TRANSACTION

  SQL (0.0ms)  BEGIN immediate TRANSACTION
  SolidQueue::Process Update (0.5ms)  UPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?  [["last_heartbeat_at", "2024-09-06 14:31:19.264649"], ["id", 94]]
  TRANSACTION (0.0ms)  COMMIT TRANSACTION

  SQL (0.1ms)  BEGIN immediate TRANSACTION
  SolidQueue::Process Update (1.3ms)  UPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?  [["last_heartbeat_at", "2024-09-06 14:31:19.272308"], ["id", 98]]
  TRANSACTION (0.0ms)  ROLLBACK TRANSACTION
  SQL (0.1ms)  BEGIN immediate TRANSACTION
  SolidQueue::Process Update (1.8ms)  UPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?  [["last_heartbeat_at", "2024-09-06 14:31:19.278502"], ["id", 97]]
  TRANSACTION (0.0ms)  ROLLBACK TRANSACTION
  SQL (0.0ms)  BEGIN immediate TRANSACTION
  SolidQueue::Process Update (0.4ms)  UPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?  [["last_heartbeat_at", "2024-09-06 14:31:19.287002"], ["id", 96]]
  TRANSACTION (0.0ms)  ROLLBACK TRANSACTION
  SQL (3.5ms)  BEGIN immediate TRANSACTION
  SolidQueue::Process Update (4.3ms)  UPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?  [["last_heartbeat_at", "2024-09-06 14:31:19.281150"], ["id", 95]]
  TRANSACTION (0.1ms)  ROLLBACK TRANSACTION
  SQL (0.0ms)  BEGIN immediate TRANSACTION
  SolidQueue::Process Update (0.4ms)  UPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?  [["last_heartbeat_at", "2024-09-06 14:31:19.287669"], ["id", 99]]
  TRANSACTION (0.0ms)  ROLLBACK TRANSACTION
  SolidQueue::Process Load (0.2ms)  [34mSELECT "solid_queue_processes".* FROM "solid_queue_processes" WHERE "solid_queue_processes"."last_heartbeat_at" <= ? ORDER BY "solid_queue_processes"."id" ASC LIMIT ?   [["last_heartbeat_at", "2024-09-06 14:27:19.003458"], ["LIMIT", 50]]
  SQL (0.1ms)  BEGIN immediate TRANSACTION
  SolidQueue::Process Destroy (1.3ms)  DELETE FROM "solid_queue_processes" WHERE "solid_queue_processes"."id" = ?  [["id", 8]]

The following processes IDs are heart-beating:

"id", 93]]
"id", 94]]
"id", 95]]
"id", 96]]
"id", 97]]
"id", 98]]
"id", 99]]

Because of the order of registration, 93 must be the supervisor, then, they're started in the order returned by the configuration, which for version 0.8.2 is dispachers first, then workers. This is not guaranteed because these are being forked so it's possible one process forked in second place could register before the one forked in first place, but since registering is the first thing that happens, I think it's unlikely. This means that 94 would be your single dispatcher, and then 95 to 99 are your 5 workers. Then, we can see that the ones that are getting their heartbeat updates rolled-back are the workers. Supervisor (93) and dispatcher (94) are committed just fine across the whole log:

  SQL (0.2ms)  BEGIN immediate TRANSACTION
  SolidQueue::Process Update (3.9ms)  UPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?  [["last_heartbeat_at", "2024-09-06 14:20:18.968432"], ["id", 93]]
  TRANSACTION (0.1ms)  COMMIT TRANSACTION

  SQL (2.7ms)  BEGIN immediate TRANSACTION
  SolidQueue::Process Update (3.7ms)  UPDATE "solid_queue_processes" SET "last_heartbeat_at" = ? WHERE "solid_queue_processes"."id" = ?  [["last_heartbeat_at", "2024-09-06 14:27:19.224777"], ["id", 94]]
  TRANSACTION (0.1ms)  COMMIT TRANSACTION

whereas all updates for processes 95 to 99 are rolled back, so all workers.

I'm not sure what this means yet 😅

Ok think i know why the rollback is happening. If you touch an object and it returns false rails calls rollback automatically: https://github.com/rails/rails/blob/f52525e53e4ed0cb9f286352f6c87b7acd982b00/activerecord/lib/active_record/transactions.rb#L419.

it returns false if affected rows isn't one: https://github.com/rails/rails/blob/f52525e53e4ed0cb9f286352f6c87b7acd982b00/activerecord/lib/active_record/persistence.rb#L807

In my example that explains where the rollback is coming from but doesnt explain why touch is returning false in the first place and only for the workers.

Something else to add to the whiteboard i noticed:

This is what ls storage shows where all the db files are stored. Notice how the queue_development has 2 extra files that are the -whl and -shm. None of the other databases have that. Anyone know what they are for? @fractaledmind . I have also tried removing just those two files to see if they were just random but as soon as i booted the server back up they reappeared.

Those are the files SQLite produces when in WAL mode and writes are occurring/have occurred. If you opened a console and created a record in your primary db, you'd see those files added for that sb as well.

Just a small reminder for years of debugging in the dark: save your brain cycles and energy trying to find the problem. Definitionally we have missing info and/or faulty assumptions somewhere. The exclusive focus for us all should be creating a reproduction. I promise we won't solve this without that. So containing to explore the space of scenarios in an attempt to deterministically reproduce is going to be the highest leverage activity.

@fractaledmind indeed! I'm trying to apply what @npezza93 is seeing and the info from the other folks to your solid_queue_bughunt app with different termination dances for all the processes 😅 But so far, no luck and no corruption... I wondered if @marcoroth was able to reproduce the problem in a fresh clone + setup of the rubyvideo app...

Yeah, I want to see more reproductions, even if anecdotal. Because the temptation is to presume that it "accidentally" got fixed in some version > 0.5.x.

Luckily (😅), once it becomes a Rails default, if it is still lurking in there, a hundred people will find it and that should help find a repro. 😬

Because the temptation is to presume that it "accidentally" got fixed in some version > 0.5.x.

Ah! No, I don't think that's the case because @npezza93 is running the last version, 0.8.2. So I've been trying with that one now for a while.

Another thing I've noticed is that I run all the time into database is locked (SQLite3::BusyException) when having 5 workers (like @npezza93), which makes sense with the default config and not https://github.com/rails/rails/pull/51958? Are you running any patches or a different value for retries in the database config... or something that prevents these? I'm surprised you don't run into them all the time as well, @npezza93 😕

The busy exceptions are much more often caused by not running IMMEDIATE transactions than the busy timeout. So running Solid Queue backed by SQLite on Rails <8.0 without any tuning will certain raise busy exceptions with more than 1 worker and some concurrent job queuing.

Ahhh! That's it, and Nick's transactions were all immediate. I've set default_transaction_mode: immediate in the repro app as well.

@npezza93: If/when you have a corrupted database, can you share the result of sqlite3 storage/development-queue.sqlite3 'pragma integrity_check;'

@rosa Yea im living on the edge and following main.

gem "rails", github: "npezza93/rails", branch: "union-fix"

with a database config of:

default: &default
  adapter: sqlite3
  pool: <%= ENV.fetch("RAILS_MAX_THREADS") { 5 } %>
  timeout: 5000

development:
  primary:
    <<: *default
    database: storage/dev.sqlite3
  queue:
    <<: *default
    migrations_paths: db/queue_migrate
    database: storage/queue_development.sqlite3

If/when you have a corrupted database, can you share the result of sqlite3 storage/development-queue.sqlite3 'pragma integrity_check;'

For sure! Never been so excited to get a corrupted database before 😅

@rosa Alright it is definitely related to closing the laptop and reopening it. It just happened again to me. Still working on getting reliable repro steps(I think it might be a timing thing though as ill explain later so getting reliable repro steps might be a challenge). I've attached the three database files and the log. The beginning of the log is the start of the server.

Walking through the log:

Heartbeat at "2024-09-06 20:07:33.745511" which i believe happened after i closed the laptop. And i think while this batch of heartbeats was happening macos did something to go to sleep and suspend stuff because when it woke back up there was a singular heartbeat at "2024-09-06 20:23:53.595583". 15ish minutes later when the laptop woke back up.
One thing ive also noticed while trying to reproduce doing the opening and closing the laptop dance is that when it wakes back up it destroys all the process records in the db. which youll see right after that last heartbeat at "2024-09-06 20:23:53.595583". It never creates new Process records so when it goes to run the heartbeat method on the process records it had previously cached you see the rollback because they were destroyed and cant be updated. It also looks like 2 lingering heartbeats happened after the destruction stuff which is weird because those two heartbeats went through even though the processes had been destroyed just before. Only thing i can think is the logging was done out of order.
Moving on to the rollbacks. One heartbeat happened at "2024-09-06 20:23:53.763842" and one happened at "2024-09-06 20:23:53.763843" and then we got the malformation error. These happened virtually at the same time so something could be happening with these two things happening at the same time combined with executing the rollback.
When running the integrity check mentioned by Stephen sqlite says everything is ok. boo

If i had to hazard a fix it would be that once solid queue deregisters all the processes once waking back up, it should ensure those processes are for sure killed.

When i wake it back up and it deregisters everything but doesnt corrupt i can feed it jobs and it runs fine so those processes are either still up and running or it has spun up new ones and not logged them.
Once it is in this corrupted state its not executing jobs

Once everything is deregistered and killed it should spin up new processes and record them in the db that way the rollbacks never happen and we are heartbeating live processes. Or alternately, dont have it deregister the process when waking back up.

I can reliably reproduce the deregistering process stff where it destroys all the Process records from the database. Using all the configs mentioned above and that rails branch which is essentially main, run solid queue in puma. Let it go through one or two heartbeat cycles and then close your laptop. Wait like 10 -15 minutes and then open your laptop up and you should see it deregister the processes in the logs. Go into the console and if you run SolidQueue::Process.count it should show zero and you should see the rollbacks happen in the log.

Let me know if theres anything i can run before wiping the slate clean to try and reproduce the corruption error.

other3.log corrupt.zip

Thank you so much @npezza93, this is super helpful. I definitely didn't consider the case of processes just being suspended in this way and then coming back to life with their heartbeats expired but very much alive and then having their process records deleted under them! 😅 Fantastic catch 🙇‍♀️ . I'll work on a fix for this. It's still quite surprising to me that this leads to corruption in the DB, but we'll get there!

The strange thing is that when I try to run pragma integrity_check; in your DB, I get ok 😕

sqlite3 ~/Desktop/corrupt/queue_development.sqlite3  'pragma integrity_check;'
ok

Could this be due to the SQLite version? I'm running this:

sqlite3 -version
3.39.5 2022-10-14 20:58:05 554764a6e721fab307c63a4f98cd958c8428a5d9d8edfde951858d6fd02daapl

~~What do you get when you run pragma integrity_check; in that DB?~~

Ah, I'm dumb, missed that you said:

When running the integrity check mentioned by Stephen sqlite says everything is ok. boo

This is so strange! 😕

I can reproduce the issue with the processes being deleted as well, just by tweaking the heartbeat interval and the alive process threshold, so that processes are considered dead even though they aren't. However, I haven't managed to get the SQLite3::CorruptException in any way.

I've also upgraded to Rails edge as I still got the SQLite3::BusyException relatively often, which led to processes crashing for real and being replaced by new records. It doesn't seem to happen after upgrading to Rails 8.

Another thing I've tried is to run different jobs while the processes are running but they already lost their process records so all their heartbeats fail, and also killing them in different ways when they're in that state. Also nothing.

What an elusive bug!

I can reproduce this issue after Mac hibernation with this config:

Mac Sonoma 14.6.1 / Intel
Solid_queue : 0.8.2
Rails : main
Dedicated Sqlite DB
Using puma plugin
SQlite version 3.43.2 2023-10-10 13:08:14 1b37c146ee9ebb7acd0160c0ab1fd11017a419fa8a3187386ed8cb32b709aapl (64-bit)
sqlite3 storage/queue.sqlite3 "PRAGMA integrity_check" returns ok
log solidqueue.log

note:

rails and solid_queue don't share the same time zone (don't know why)
my configuration is mostly default settings from rails 'main'

config:

default: &default
   dispatchers:
     - polling_interval: 1
       batch_size: 500
       recurring_tasks:
         my_periodic_job:
           class: CheckJob
           args:
           schedule: every 10 min
   workers:
     - queues: "*"
       threads: 3
       processes: 1
       polling_interval: 0.1

 development:
  <<: *default

 test:
  <<: *default

Edit: sqlite version

Thanks a lot for all the info, @alvernier, and for the log file. Quite a bunch of interesting errors there that I hadn't seen before for this error:

  SQL (0.0ms)  BEGIN immediate TRANSACTION /*application='Ding'*/
  SolidQueue::Process Destroy (6.3ms)  DELETE FROM "solid_queue_processes" WHERE "solid_queue_processes"."id" = 11 /*application='Ding'*/
  TRANSACTION (0.6ms)  ROLLBACK TRANSACTION /*application='Ding'*/
SolidQueue-0.8.2 Error deregistering Dispatcher (36.7ms)  process_id: 11, pid: 83006, hostname: "macav.home", name: "dispatcher-756b1bcf2b335f37f255", last_heartbeat_at: "2024-09-07T14:36:25Z", claimed_size: nil, pruned: false, error: "ActiveRecord::StatementInvalid SQLite3::SQLException: cannot rollback - no transaction is active"
SolidQueue-0.8.2 Shutdown Dispatcher (49.8ms)  pid: 83006, hostname: "macav.home", process_id: 11, name: "dispatcher-756b1bcf2b335f37f255", polling_interval: 1, batch_size: 500, concurrency_maintenance_interval: 600, recurring_schedule: ["my_periodic_job"]
/Users/avernier/.rvm/gems/ruby-3.3.4/bundler/gems/rails-90ab51b91008/activerecord/lib/active_record/connection_adapters/sqlite3/database_statements.rb:100:in `step': 
SQLite3::SQLException: cannot rollback - no transaction is active (ActiveRecord::StatementInvalid)

which then continues with

/Users/avernier/.rvm/gems/ruby-3.3.4/bundler/gems/rails-90ab51b91008/activerecord/lib/active_record/connection_adapters/sqlite3/database_statements.rb:100:in `step': 
SQLite3::IOException: disk I/O error (ActiveRecord::StatementInvalid)

Then

  SQL (5003.7ms)  BEGIN immediate TRANSACTION /*application='Ding'*/
  SolidQueue::Process Create (5008.8ms)  INSERT INTO "solid_queue_processes" ("kind", "last_heartbeat_at", "supervisor_id", "pid", "hostname", "metadata", "created_at", "name") VALUES ('Worker', '2024-09-07 14:36:26.577564', 10, 9713, 'macav.home', '{"polling_interval":0.1,"queues":"*","thread_pool_size":3}', '2024-09-07 14:36:26.591125', 'worker-d4340070c21482275252') RETURNING "id" /*application='Ding'*/
SolidQueue-0.8.2 Error registering Worker (5062.7ms)  pid: 9713, hostname: "macav.home", name: "worker-d4340070c21482275252", error: "ActiveRecord::StatementTimeout SQLite3::BusyException: database is locked"
SolidQueue-0.8.2 Started Worker (5073.1ms)  pid: 9713, hostname: "macav.home", process_id: nil, name: "worker-d4340070c21482275252", polling_interval: 0.1, queues: "*", thread_pool_size: 3
/Users/avernier/.rvm/gems/ruby-3.3.4/bundler/gems/rails-90ab51b91008/activerecord/lib/active_record/connection_adapters/sqlite3/database_statements.rb:100:in `step': 
SQLite3::BusyException: database is locked (ActiveRecord::StatementTimeout)

And then everything finishes correctly:

  SolidQueue::Process Load (0.2ms)  SELECT "solid_queue_processes".* FROM "solid_queue_processes" WHERE "solid_queue_processes"."supervisor_id" = 10 AND "solid_queue_processes"."name" = 'worker-d4340070c21482275252' LIMIT 1 /*application='Ding'*/
SolidQueue-0.8.2 Supervisor terminated gracefully (3043.9ms)  process_id: 10, supervisor_pid: 82999, supervised_processes: [83006, 9713]
SolidQueue-0.8.2 Supervisor terminated gracefully (0.0ms)  process_id: 10, supervisor_pid: 82999, supervised_processes: []
  SQL (0.0ms)  BEGIN immediate TRANSACTION /*application='Ding'*/
  SolidQueue::Process Destroy (5.8ms)  DELETE FROM "solid_queue_processes" WHERE "solid_queue_processes"."id" = 10 /*application='Ding'*/
  TRANSACTION (0.0ms)  COMMIT TRANSACTION /*application='Ding'*/
  SolidQueue::Process Load (0.8ms)  SELECT "solid_queue_processes".* FROM "solid_queue_processes" WHERE "solid_queue_processes"."supervisor_id" = 10 /*application='Ding'*/
SolidQueue-0.8.2 Deregister Supervisor (23.9ms)  process_id: 10, pid: 82999, hostname: "macav.home", name: "supervisor-2a34bccf0ad7ab139560", last_heartbeat_at: "2024-09-07T14:36:26Z", claimed_size: nil, pruned: false
SolidQueue-0.8.2 Shutdown Supervisor (27.8ms)  pid: 82999, hostname: "macav.home", process_id: 10, name: "supervisor-2a34bccf0ad7ab139560"
- Gracefully stopping, waiting for requests to finish
=== puma shutdown: 2024-09-07 16:36:31 +0200 ===
- Goodbye!

And the DB is not corrupted after all, just like @npezza93's . This seems definitely like a different problem from the one reported originally by @andycroll and @marcoroth because the DB ended up being corrupted for them. I wonder why a SQLite3::CorruptException: database disk image is malformed (ActiveRecord::StatementInvalid) error would be raised if the DB is not really corrupted 😕

Complete guess, but I wonder if the WAL is corrupted and hasn't been checkpointed back to main db file yet. Maybe we should try manually check pointing to get everything in the main db file, then run the integrity check 🤷‍♂️

rails / solid_queue

SQLite queue database corruption #324