Error database is locked after running for sometime

[huksley]

[2] 7:10PM ERR sqlite.go:499 > Deleted unexpected number of messages message_id=1827377270830927872 queue=tasks rowsAffected=0 tenant_id=1
[2] 7:10PM INF zerolog.go:82 > Success ip=127.0.0.1 latency="192.083µs" method=POST status=200 url=/
[2] 7:10PM ERR zerolog.go:86 > Server error error="database is locked" ip=127.0.0.1 latency=5.184703667s method=POST status=500 url=/
[2] 7:10PM ERR zerolog.go:86 > Server error error="database is locked" ip=127.0.0.1 latency=5.187398083s method=POST status=500 url=/
[4]  GET /api/auth/session 200 in 7ms
[4]  GET /api/host/cm08c2haz000jxixqc8rno1rq 200 in 11ms
[2] 7:10PM ERR zerolog.go:86 > Server error error="database is locked" ip=127.0.0.1 latency=5.187968584s method=POST status=500 url=/
[4] InternalFailure: database is locked
[4]     at throwDefaultError (/Users/ruslan/src/dollar/node_modules/@smithy/smithy-client/dist-cjs/index.js:839:20)
[4]     at /Users/ruslan/src/dollar/node_modules/@smithy/smithy-client/dist-cjs/index.js:848:5
[4]     at de_CommandError (/Users/ruslan/src/dollar/node_modules/@aws-sdk/client-sqs/dist-cjs/index.js:1282:14)
[4]     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
[4]     at async /Users/ruslan/src/dollar/node_modules/@smithy/middleware-serde/dist-cjs/index.js:35:20
[4]     at async /Users/ruslan/src/dollar/node_modules/@smithy/core/dist-cjs/index.js:165:18
[4]     at async /Users/ruslan/src/dollar/node_modules/@smithy/middleware-retry/dist-cjs/index.js:320:38
[4]     at async /Users/ruslan/src/dollar/node_modules/@aws-sdk/middleware-sdk-sqs/dist-cjs/index.js:92:18
[4]     at async /Users/ruslan/src/dollar/node_modules/@aws-sdk/middleware-logger/dist-cjs/index.js:34:22
[4]     at async processQueue (webpack-internal:///(api)/./lib/processQueue.ts:443:35) {

[poundifdef]

Thank you. Can you share more details on how I can reproduce? How long had the server been running, and approximately what was the rate of messages being inserted and consumed? How many producers and how many consumers?

[huksley]

I think it was working for 4 hours and there was not so many messages to be frank. Maybe around 100. The error was persistent until I restarted and deleted the sqlite file.

Great development productivity boost btw when you developing locally and do not need to the AWS to fiddle with real queues and permissions.

P.S. smoothmq does not require setting MessageAttributeNames when checking for new messages and returns all the attributes anyway. This is only difference in implementation I found comparing to production AWS SQS.

const receiveMessageCommand = new ReceiveMessageCommand({
    QueueUrl: queueUrl,
    MaxNumberOfMessages: 1,
    // Needed for AWS SQS but not for smoothmq
    MessageAttributeNames: ["All"]
  });

[sundbry]

I had a similar error happen this weekend. I had to restart the SmoothMQ process to unlock it. A simliar error triggered the lockup: Deleted unexpected number of messages

9:47PM ERR sqlite.go:519 > Deleted unexpected number of messages message_id=1829274328370253827 queue=forum-usage-queue-1 rowsAffected=0 tenant_id=1
9:47PM ERR zerolog.go:86 > Server error error="database is locked" ip=127.0.0.1 latency=5.016242719s method=POST status=500 url=/1/vendor-usage-Um85vzZ6yXg
9:47PM ERR zerolog.go:86 > Server error error="database is locked" ip=127.0.0.1 latency=5.02429745s method=POST status=500 url=/1/forum-usage-queue-1
9:47PM ERR zerolog.go:86 > Server error error="database is locked" ip=127.0.0.1 latency=5.018417555s method=POST status=500 url=/1/forum-usage-queue-1
....

[poundifdef]

I've made some major changes to the DB since the version you're running. In addition to updating the structure, it also uses an ORM to better manage transactions.

You may not be able to upgrade without re-creating the sqlite db (hopefully this won't be as painful in the future with gorm's auto-migration tool.) Would you mind giving it a try?

[huksley]

Yes, please! Could you make a release tag, so I can built it using a macos brew toolkit, thank you!

[poundifdef]

I've created a release tag. I'll close this issue for now but feel free to re-create if it comes up again.

[huksley]

thank you @poundifdef would be nice if releases would have at least some release information (i.e. changes) I think you can draft a release out of tag easily and just add some notes