database is locked when launching multiple jobs

[kailaix]

Sometimes when I tried to launch multiple jobs, I got the following error

django.db.utils.OperationalError: database is locked

I used sleep 1 between two smt run ... & and used different labels for each run to avoid confliction. However, such procedures did not solve the problem.

What might be the reason? Thanks!

[slamer59]

Hello, Sumatra is run as "standalone" which is great when you don't want to setup a full server etc but problem like this can arise.

Django explain it here: https://docs.djangoproject.com/en/dev/ref/databases/#database-is-locked-errors

    SQLite is meant to be a **lightweight database**, and thus **can't support a high level of concurrency**. OperationalError: database is locked errors indicate that your application is experiencing more concurrency than sqlite can handle in default configuration. This error means that one thread or process has an exclusive lock on the database connection and another thread timed out waiting for the lock the be released.

    Python's SQLite wrapper **has a default timeout value** that determines how long the second thread is allowed to wait on the lock before it times out and raises the OperationalError: database is locked error.

    If you're getting this error, you can solve it by:
-  Switching to another database backend. At a certain point SQLite becomes too "lite" for real-world applications, and these sorts of concurrency errors indicate you've reached that point.
- Rewriting your code to reduce concurrency and ensure that database transactions are short-lived. 
- Increase the default timeout value by setting the timeout database option

The first proposition can be overkill for little experimentations but if you want other people to work with you or have much more experiments at the same time in parallel you should consider it.

For the second proposition: The command sleep 1 doesnot help obviously. For exemple if your second run is faster by one second there will be concurency. You might try in your script to test if database is locked before adding information.

How much time your experiment runs and how many runs ?

The third proposition is easy you can add a timeout in settings.py and OPTIONS

# settings.py
DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.sqlite3',
        'NAME': '/home/tomi/djangotest.sqlite',
        # for sqlite write lock timeout
        'OPTIONS': {
            'timeout': 5,
        }
    }
}

[apdavison]

As a follow up to the response from @slamer59, it is also possible to use a PostgreSQL database with Sumatra, either locally or on a remote server - see https://sumatra.readthedocs.io/en/latest/record_stores.html This avoids the locking problem.

It is simplest if you set this up when first creating a project, but it is also possible to migrate existing data to a new database, see https://sumatra.readthedocs.io/en/latest/record_stores.html#changing-record-stores

[kailaix]

Thanks. It's good to know that.