Label-Studio crash "Internal Server Error" when trying to filter by Annotation results

[Skier23]

Describe the bug Anytime I try to filter by annotation results I get an internal server error and label studio stops responding

To Reproduce Steps to reproduce the behavior:

1. Try filtering by annotation results
2. Label Studio stops responding

Expected behavior No crashes

Screenshots

Environment (please complete the following information):

• OS: Windows 10
• Label Studio Version 1.10.1dev

Logs:

[2024-01-22 09:03:52,010] [django.request::log_response::224] [ERROR] Internal Server Error: /api/dm/views/16/ Traceback (most recent call last): File "D:\test\NeuralDataSet\LabelStudio\env\Lib\site-packages\django\db\backends\utils.py", line 84, in _execute return self.cursor.execute(sql, params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\test\NeuralDataSet\LabelStudio\env\Lib\site-packages\django\db\backends\sqlite3\base.py", line 423, in execute return Database.Cursor.execute(self, query, params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sqlite3.OperationalError: database is locked

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "D:\test\NeuralDataSet\LabelStudio\env\Lib\site-packages\django\core\handlers\exception.py", line 47, in inner response = get_response(request) ^^^^^^^^^^^^^^^^^^^^^ File "D:\test\NeuralDataSet\LabelStudio\env\Lib\site-packages\django\core\handlers\base.py", line 171, in _get_response response = middleware_method(request, callback, callback_args, callback_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\test\NeuralDataSet\LabelStudio\env\Lib\site-packages\sentry_sdk\integrations\django\middleware.py", line 111, in sentry_wrapped_method return old_method(*args, *kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\test\NeuralDataSet\LabelStudio\env\Lib\site-packages\label_studio\core\middleware.py", line 166, in process_view request.user.update_last_activity() File "D:\test\NeuralDataSet\LabelStudio\env\Lib\site-packages\label_studio\users\models.py", line 69, in update_last_activity self.save(update_fields=['last_activity']) File "D:\test\NeuralDataSet\LabelStudio\env\Lib\site-packages\django\contrib\auth\base_user.py", line 67, in save super().save(args, **kwargs) File "D:\test\NeuralDataSet\LabelStudio\env\Lib\site-packages\django\db\models\base.py", line 739, in save self.save_base(using=using, force_insert=force_insert, File "D:\test\NeuralDataSet\LabelStudio\env\Lib\site-packages\django\db\models\base.py", line 776, in save_base updated = self._save_table( ^^^^^^^^^^^^^^^^^ File "D:\test\NeuralDataSet\LabelStudio\env\Lib\site-packages\django\db\models\base.py", line 858, in _save_table updated = self._do_update(base_qs, using, pk_val, values, update_fields, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\test\NeuralDataSet\LabelStudio\env\Lib\site-packages\django\db\models\base.py", line 912, in _do_update return filtered._update(values) > 0 ^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\test\NeuralDataSet\LabelStudio\env\Lib\site-packages\django\db\models\query.py", line 802, in _update return query.get_compiler(self.db).execute_sql(CURSOR) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\test\NeuralDataSet\LabelStudio\env\Lib\site-packages\django\db\models\sql\compiler.py", line 1559, in execute_sql cursor = super().execute_sql(result_type) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\test\NeuralDataSet\LabelStudio\env\Lib\site-packages\django\db\models\sql\compiler.py", line 1175, in execute_sql cursor.execute(sql, params) File "D:\test\NeuralDataSet\LabelStudio\env\Lib\site-packages\sentry_sdk\integrations\django__init__.py", line 641, in execute return real_execute(self, sql, params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\test\NeuralDataSet\LabelStudio\env\Lib\site-packages\django\db\backends\utils.py", line 66, in execute return self._execute_with_wrappers(sql, params, many=False, executor=self._execute) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\test\NeuralDataSet\LabelStudio\env\Lib\site-packages\django\db\backends\utils.py", line 75, in _execute_with_wrappers return executor(sql, params, many, context) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\test\NeuralDataSet\LabelStudio\env\Lib\site-packages\django\db\backends\utils.py", line 79, in _execute with self.db.wrap_database_errors: File "D:\test\NeuralDataSet\LabelStudio\env\Lib\site-packages\django\db\utils.py", line 90, in exit raise dj_exc_value.with_traceback(traceback) from exc_value File "D:\test\NeuralDataSet\LabelStudio\env\Lib\site-packages\django\db\backends\utils.py", line 84, in _execute return self.cursor.execute(sql, params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\test\NeuralDataSet\LabelStudio\env\Lib\site-packages\django\db\backends\sqlite3\base.py", line 423, in execute return Database.Cursor.execute(self, query, params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ django.db.utils.OperationalError: database is locked

[jombooth]

Hi @Skier23 - thanks for the report and stacktrace. Could you confirm that other filters work okay + it's just annotation results filtering that surfaces this issue?

[Skier23]

Yea all the other fields seem to work correctly (except for storage_filename but that seems to be a different bug): Cannot resolve keyword 'storage_filename' into field. Choices are: annotations, cancelled_annotations, comment_authors, comment_count, created_at, data, drafts, file_upload, file_upload_id, id, inner_id, io_storages_azureblobimportstoragelink, io_storages_gcsimportstoragelink, io_storages_localfilesimportstoragelink, io_storages_redisimportstoragelink, io_storages_s3importstoragelink, is_labeled, last_comment_updated_at, locks, meta, overlap, predictions, project, project_id, total_annotations, total_predictions, unresolved_comment_count, updated_at, updated_by, updated_by_id

Traceback (most recent call last): File "C:\Users\tyler\anaconda3\Lib\site-packages\django\db\models\sql\query.py", line 1955, in add_fields join_info = self.setup_joins(name.split(LOOKUP_SEP), opts, alias, allow_many=allow_m2m) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\tyler\anaconda3\Lib\site-packages\django\db\models\sql\query.py", line 1648, in setup_joins path, final_field, targets, rest = self.names_to_path( ^^^^^^^^^^^^^^^^^^^ File "C:\Users\tyler\anaconda3\Lib\site-packages\django\db\models\sql\query.py", line 1562, in names_to_path raise FieldError("Cannot resolve keyword '%s' into field. " django.core.exceptions.FieldError: Cannot resolve keyword 'storage_filename' into field. Choices are: annotations, cancelled_annotations, comment_authors, comment_count, created_at, data, drafts, file_upload, file_upload_id, id, inner_id, io_storages_azureblobimportstoragelink, io_storages_gcsimportstoragelink, io_storages_localfilesimportstoragelink, io_storages_redisimportstoragelink, io_storages_s3importstoragelink, is_labeled, last_comment_updated_at, locks, meta, overlap, predictions, project, project_id, total_annotations, total_predictions, unresolved_comment_count, updated_at, updated_by, updated_by_id

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "C:\Users\tyler\anaconda3\Lib\site-packages\rest_framework\views.py", line 506, in dispatch response = handler(request, *args, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\tyler\anaconda3\Lib\site-packages\django\utils\decorators.py", line 43, in _wrapper return bound_method(*args, *kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\tyler\anaconda3\Lib\site-packages\label_studio\data_manager\api.py", line 227, in get queryset = self.get_task_queryset(request, prepare_params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\tyler\anaconda3\Lib\site-packages\label_studio\data_manager\api.py", line 195, in get_task_queryset return Task.prepared.only_filtered(prepare_params=prepare_params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\tyler\anaconda3\Lib\site-packages\label_studio\data_manager\managers.py", line 661, in only_filtered return queryset.prepared(prepare_params=prepare_params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\tyler\anaconda3\Lib\site-packages\label_studio\data_manager\managers.py", line 472, in prepared queryset = apply_filters(queryset, prepare_params.filters, project, request) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\tyler\anaconda3\Lib\site-packages\label_studio\data_manager\managers.py", line 350, in apply_filters value_type = type(queryset.values_list(field_name, flat=True)[0]).name ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\tyler\anaconda3\Lib\site-packages\django\db\models\query.py", line 867, in values_list clone = self._values(_fields, expressions) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\tyler\anaconda3\Lib\site-packages\django\db\models\query.py", line 835, in _values clone.query.set_values(fields) File "C:\Users\tyler\anaconda3\Lib\site-packages\django\db\models\sql\query.py", line 2272, in set_values self.add_fields(field_names, True) File "C:\Users\tyler\anaconda3\Lib\site-packages\django\db\models\sql\query.py", line 1982, in add_fields raise FieldError("Cannot resolve keyword %r into field. " django.core.exceptions.FieldError: Cannot resolve keyword 'storage_filename' into field. Choices are: annotations, cancelled_annotations, comment_authors, comment_count, created_at, data, drafts, file_upload, file_upload_id, id, inner_id, io_storages_azureblobimportstoragelink, io_storages_gcsimportstoragelink, io_storages_localfilesimportstoragelink, io_storages_redisimportstoragelink, io_storages_s3importstoragelink, is_labeled, last_comment_updated_at, locks, meta, overlap, predictions, project, project_id, total_annotations, total_predictions, unresolved_comment_count, updated_at, updated_by, updated_by_id

Also, for annotation results, this bug seems to only occur when using "Contains". I found a temporary workaround for now by using Regex instead which allows me to do a similar query and this works correctly.

[jombooth]

indeed re: storage_filename being a different bug - I fixed that in https://github.com/HumanSignal/label-studio/pull/5289, we'll be working that into a release soon. That the bug you found occurs only when Contains is used is an excellent clue, thank you for checking this! We'll track this issue on our side, and fix it when we can; I'm glad that you've been able to set up a workaround in the meantime.

[Skier23]

In case it helps, I thought I had filtered by annotation results contains before and it worked correctly. To see if it would fix it, I tried recreating a new label studio db (in case there was some kind of db corruption) and reimporting data and this had the same effect of the bug occuring

[Skier23]

This also applies to "not contains". I'm not entirely sure if the regex filter provides a way to workaround this case.

EDIT: There is a workaround but its not the most pretty: ^(?!.Not included")(?=.Is Included").+$

This will find tasks that dont contain "Not Included" but do contain "Is Included"

[Lain810]

This also applies to "not contains". I'm not entirely sure if the regex filter provides a way to workaround this case.

EDIT: There is a workaround but its not the most pretty: ^(?!.Not included")(?=.Is Included").+$

This will find tasks that dont contain "Not Included" but do contain "Is Included"

Hello, I have encountered the same issue as you. I am currently unable to access the project with the added filter, and I am also unable to create a new project. Have you found any solutions to make the project, which is now inaccessible due to the added filter, available again? Thank you very much.

[Skier23]

If you close label-studio and open it again it will not crash until you go onto that view that has the bugged statement. So you must delete that view while in another view without clicking on it.

[Lain810]

Thank you very much for your response. After re-entering Label Studio, I didn't encounter that error, but when I opened the project corresponding to the filter, the data did not load. I noticed the "database is locked" error in the server backend. I was wondering what you meant by "view." Does it refer to the project?

[Skier23]

The different tabs you can use to look at data for your project. The default one is “Default”. You need to delete the failing one.

[JiashuaiXu]

Let me tell you my solution, none of the above methods solved my problem.But I was unable to access my project directory after using filter and it showed that the database was locked.

django.db.utils.OperationalError: database is locked
[2024-08-01 05:41:15,031] [core.utils.common::custom_exception_handler::91] [ERROR] cb6e4b22-32e5-4acf-a28f-b293626e8460 database is locked
Traceback (most recent call last):
  File "/home/jesse/miniconda3/envs/label_1_13/lib/python3.12/site-packages/django/db/backends/base/base.py", line 242, in _commit
    return self.connection.commit()
           ^^^^^^^^^^^^^^^^^^^^^^^^
sqlite3.OperationalError: database is locked

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/jesse/miniconda3/envs/label_1_13/lib/python3.12/site-packages/rest_framework/views.py", line 506, in dispatch
    response = handler(request, *args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jesse/miniconda3/envs/label_1_13/lib/python3.12/site-packages/label_studio/projects/api.py", line 482, in get
    history = get_label_stream_history(request.user, project)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jesse/miniconda3/envs/label_1_13/lib/python3.12/site-packages/label_studio/projects/functions/stream_history.py", line 38, in get_label_stream_history
    with transaction.atomic():
  File "/home/jesse/miniconda3/envs/label_1_13/lib/python3.12/site-packages/django/db/transaction.py", line 246, in __exit__
    connection.commit()
  File "/home/jesse/miniconda3/envs/label_1_13/lib/python3.12/site-packages/django/utils/asyncio.py", line 33, in inner
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/jesse/miniconda3/envs/label_1_13/lib/python3.12/site-packages/django/db/backends/base/base.py", line 266, in commit
    self._commit()
  File "/home/jesse/miniconda3/envs/label_1_13/lib/python3.12/site-packages/django/db/backends/base/base.py", line 241, in _commit
    with self.wrap_database_errors:
  File "/home/jesse/miniconda3/envs/label_1_13/lib/python3.12/site-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/home/jesse/miniconda3/envs/label_1_13/lib/python3.12/site-packages/django/db/backends/base/base.py", line 242, in _commit
    return self.connection.commit()
           ^^^^^^^^^^^^^^^^^^^^^^^^
django.db.utils.OperationalError: database is locked

I tried all the delete views in the project failed, and the default view can't be deleted.

I know it's an error in the filter on my default page. you could get the page id through the web URL .in the project my default id is 98 ，project id is 9 .if you don't know you can check it through web API

So I updated the filter via the official API provided and it worked!

You need to go to the corresponding web

api address.

Anyway, refer to @Skier23 's suggestion There's must something wrong with the view. I updated the contents of the filter through the interface provided by the web, api. Then solved the problem of view 98 being inaccessible and the database lockup he was causing.

---

import requests

# 配置API URL和请求头
api_url = "http://your-label-project:8080/api/dm/views/98/"
headers = {
    "Authorization": "Token d5c13639bc280904ceccc317a2xxxxxxxxxxxxxxxxx",  # 确保这里的格式为 'Token <your_token>'
    "Content-Type": "application/json"
}
data = {
    "project": 9,  # 替换为您的项目ID
    "data": {
        "title": "Default",
        "type": "list",
        "target": "tasks",
        "hiddenColumns": {
            "explore": [
                "tasks:annotations_ids",
                "tasks:predictions_score",
                "tasks:predictions_model_versions",
                "tasks:predictions_results",
                "tasks:file_upload",
                "tasks:storage_filename",
                "tasks:created_at",
                "tasks:updated_at",
                "tasks:updated_by",
                "tasks:avg_lead_time",
                "tasks:draft_exists"
            ],
            "labeling": [
                "tasks:cancelled_annotations",
                "tasks:total_predictions",
                "tasks:annotations_ids",
                "tasks:predictions_score",
                "tasks:predictions_model_versions",
                "tasks:predictions_results",
                "tasks:file_upload",
                "tasks:storage_filename",
                "tasks:created_at",
                "tasks:updated_at",
                "tasks:updated_by",
                "tasks:avg_lead_time",
                "tasks:completed_at",
                "tasks:annotators",
                "tasks:annotations_results",
                "tasks:draft_exists",
                "tasks:total_annotations"
            ]
        },
        "columnsWidth": {},
        "columnsDisplayType": {},
        "gridWidth": 4,
        "semantic_search": [],
        "filters": {
            "conjunction": "and",
            "items": [
                {
                    "filter": "filter:tasks:inner_id",
                    "operator": "less_or_equal",
                    "type": "Number",
                    "value": 3500
                }
            ]
        },
        "ordering": [
            "tasks:total_annotations"
        ]
    },
    "user": 1,  # 替换为用户ID
    "project": 9  # 替换为项目ID
}

# 执行API请求
response = requests.put(api_url, headers=headers, json=data)

# 输出响应结果
if response.status_code == 200:
    print("视图更新成功")
    print(response.json())  # 打印更新后的视图数据
else:
    print(f"视图更新失败: {response.status_code} - {response.text}")

[SethFalco]

I just encountered this issue, here's a simpler workaround to save your workspace if you can't access the Data Manager.

1. Navigate to /projects/{PROJECT_ID}/settings/danger-zone in your browser
2. Restart Label Studio on your server
3. Refresh the page in your browser
4. Press Drop All Tabs in Label Studio

So long as you don't actually go into the Data Manager, the database shouldn't lock. So just be sure to head to Danger Zone right away after restarting by already being on the page.