search

chuot / rdio-scanner

Rdio Scanner is an open source software that ingest and distribute audio files generated by various software-defined radio recorders. Its interface tries to reproduce the user experience of a real police scanner, while adding its own touch.
GNU General Public License v3.0
435 stars 62 forks source link

Database locked/SQLITE_BUSY/constraint failed errors until container restart #156

Closed l0gik closed 2 years ago

l0gik commented 2 years ago

Twice today I have caught rdio-scanner v6.2.2 getting stuck in a "constraint failed" loop after starting a database prune. I've had to restart it both times to get it back online, but it gets stuck again, some time later. This rdio-scanner instance is fed from trunk-recorder (running on the same Ubuntu 20.04.4 machine, in an adjacent Docker container) via trunk-recorder's rdioscanner_uploader plugin. Rdio-scanner then feeds another downstream instance hosted on a remote server, but that downstream instance doesn't appear to be having this same issue.

Relevant output from rdio-scanner's container logs:

rdio-scanner | 2022-03-30T00:30:03.984202714Z 2022/03/29 20:30:03 database pruning rdio-scanner | 2022-03-30T00:30:14.781512139Z 2022/03/29 20:30:14 controller.ingestcall: talkgroups.read: database is locked (5) (SQLITE_BUSY) rdio-scanner | 2022-03-30T00:30:34.237382100Z 2022/03/29 20:30:34 scheduler.run: database is locked (5) (SQLITE_BUSY) rdio-scanner | 2022-03-30T00:30:44.302837426Z 2022/03/29 20:30:44 controller.ingestcall: systems.write: constraint failed: UNIQUE constraint failed: rdioScannerSystems.id (2067) rdio-scanner | 2022-03-30T00:30:54.334353995Z 2022/03/29 20:30:54 controller.ingestcall: systems.write: constraint failed: UNIQUE constraint failed: rdioScannerSystems.id (2067) rdio-scanner | 2022-03-30T00:31:04.371564821Z 2022/03/29 20:31:04 controller.ingestcall: systems.write: constraint failed: UNIQUE constraint failed: rdioScannerSystems.id (2067) rdio-scanner | 2022-03-30T00:31:14.402802804Z 2022/03/29 20:31:14 controller.ingestcall: systems.write: constraint failed: UNIQUE constraint failed: rdioScannerSystems.id (2067)

...and the systems.write line repeats until I restart rdio-scanner.

I don't remember seeing this happening until recently (my containers update themselves automatically, early on Sundays), so it might be an issue introduced in this version. Not all database pruning events triggers this behavior, and there is no obvious pattern of external events (trunk-recorder or anything else) that would seem to be triggering this behavior; but I'm happy investigate further, if needed.

EDIT: checked in on everything again this morning, and also found the downstream rdio-scanner failed in the same way. Also something curious that I hadn't noticed before: prior to the database locking up again, there were many duplicate calls being rejected from trunk-recorder, which caused trunk-recorder to go into its own loop of retrying and failing to send the rejected calls.

chuot commented 2 years ago

Please try with v6.2.3 which has just been released.

l0gik commented 2 years ago

Looks like this solved it. Thanks, and great work!