Stall PP/Update All when the other is running (DB locking)

[Lasithih]

Description of the bug

After downloading a volume, queue is stuck at importing. This happens randomly..

[2024-02-19 10:40:10][Task Handler][ERROR] An error occured while trying to run a task: 
Traceback (most recent call last):
File "/app/backend/tasks.py", line 265, in __run_task
  result = task.run()
File "/app/backend/tasks.py", line 199, in run
  refresh_and_scan()
File "/app/backend/volumes.py", line 529, in refresh_and_scan
  scan_files(Volume(volume[0]).get_info())
File "/app/backend/files.py", line 610, in scan_files
  file_id = _add_file(file)
File "/app/backend/files.py", line 472, in _add_file
  cursor.execute(
sqlite3.OperationalError: database is locked

Version info

v1.0.0-beta-4

[Casvt]

Just restart the container. I've already spent the utmost effort on reducing this error from coming up.

[Lasithih]

If it helps, this happens when it tries to import a download with 'Unknown' size

[Casvt]

Oh I just checked the error again and it arises when the queue is importing a download and at the same time an 'Update All' is run. It 'overwhelms' the database with those two happening at the same time. I already put considerable effort into avoiding this error from coming up.

Before I fixed this problem, I already noticed that most of the time this error arises, it's because the queue is importing something and a task is running at the same time (just like in this scenario). My fixes worked very well, because you're the first person reporting this error again after I implemented the fix 39 days ago. However, at this point, the most I can do (because I already did almost everything I could) is to make the importing of a download wait on the completion of the task if it's running.

[Casvt]

Ah it's coming together now I think. Was the file in the picture a very large file (say >1GB)? You can check the file size or the GC page where the download came from.

What I think is happening is that the import process got a lock on the database and then started to move the file. While this is happening, an 'Update All' starts and wants to write to the db. However, because the import process has a lock and is now spending tens of seconds moving the file (if it's large), the task errors out.

P.S. Sorry for rejecting your issue so harshly and quickly at the start. I used to get loads of people reporting this without taking the time to see that it was already reported. And almost always I was already working on it or it was even already fixed but they didn't update. You however, have actually found a bug, or better said a place which can be improved. I did almost everything to fix it, and you actually found a place that didn't already fall under that 'almost', so I can only thank you for that.

[Casvt]

Improved handling. Error shouldn't occur anymore. Will be included in the next release (v1.0.0-stable).

[Lasithih]

@Casvt Awesome! No worries. I'm looking foward to try the next version.