search

Azure / KAN

KubeAI Application Nucleus for edge is a solution accelerator for creating, deploying, and operating environment-aware solutions at scale that use artificial intelligence (AI) at the edge with the control and flexibility of open-source natively on your environment.
MIT License
77 stars 21 forks source link

Web portal crashed and fails to restart and recover from failure #191

Closed sreevishnu-damodaran closed 1 year ago

sreevishnu-damodaran commented 1 year ago

We had faced an issue where the kubernetes Webmodule container suddenly crashed.

The automatic container restarts were starting and failing. We tried restarting the cluster and containers, but the issue persisted. It went away after a clean reinstall of POSS. We couldn't reproduce the issue again.

Version of POSS installer used: 0.38.2

Output of the kubectl logs webmodule... command with the error trace:

[
  {
    "cloudName": "AzureCloud",
    "homeTenantId": [REDACTED]
    "id": [REDACTED],
    "isDefault": true,
    "managedByTenants": [],
    "name": [REDACTED],
    "state": "Enabled",
    "tenantId": [REDACTED],
    "user": {
      "name": [REDACTED],
      "type": "servicePrincipal"
    }
  }
]
[16-Nov-2022 16:57:49] INFO vision_on_edge.azure_settings.apps : Azure Settings AppConfig ready while running server
[16-Nov-2022 16:57:49] INFO vision_on_edge.azure_settings.apps : Setting azure http logging level to WARNING
[16-Nov-2022 16:57:49] INFO vision_on_edge.azure_settings.apps : https://westeurope.api.cognitive.microsoft.com/
[16-Nov-2022 16:57:49] INFO vision_on_edge.azure_settings.apps : [REDACTED]
[16-Nov-2022 16:57:49] INFO vision_on_edge.azure_settings.apps : Creating new DEFAULT_SETTING
[16-Nov-2022 16:57:49] INFO vision_on_edge.azure_settings.models : Setting pre_save
[16-Nov-2022 16:57:49] INFO vision_on_edge.azure_settings.models : Setting validating (DEFAULT_SETTING)
[16-Nov-2022 16:57:49] INFO vision_on_edge.azure_settings.models : Setting validate success.
[16-Nov-2022 16:57:49] INFO vision_on_edge.azure_settings.models : Setting validating (DEFAULT_SETTING)
[16-Nov-2022 16:57:49] INFO vision_on_edge.azure_settings.models : Setting validate success.
[16-Nov-2022 16:57:49] INFO vision_on_edge.azure_projects.helpers : Create/update a default project.
[16-Nov-2022 16:57:49] INFO vision_on_edge.azure_projects.models : Project pre_save start
[16-Nov-2022 16:57:49] INFO vision_on_edge.azure_projects.models : Project (id, customvision_id, name): (None, , )
[16-Nov-2022 16:57:49] INFO vision_on_edge.azure_projects.models : Project (id, customvision_id, name): (None, , VisionOnEdge-2022-11-16T16:57:49.362000)
[16-Nov-2022 16:57:49] INFO vision_on_edge.azure_projects.models : Project pre_save end
[16-Nov-2022 16:57:49] WARNING Backend-Training-App-Insight : create_project
[16-Nov-2022 16:57:49] INFO vision_on_edge.azure_projects.models : Project pre_save start
[16-Nov-2022 16:57:49] INFO vision_on_edge.azure_projects.models : Project (id, customvision_id, name): (None, , rtsp)
[16-Nov-2022 16:57:49] INFO vision_on_edge.azure_projects.models : Project (id, customvision_id, name): (None, , rtsp)
[16-Nov-2022 16:57:49] INFO vision_on_edge.azure_projects.models : Project pre_save end
[16-Nov-2022 16:57:49] WARNING Backend-Training-App-Insight : create_project
[16-Nov-2022 16:57:49] INFO vision_on_edge.azure_projects.models : Project pre_save start
[16-Nov-2022 16:57:49] INFO vision_on_edge.azure_projects.models : Project (id, customvision_id, name): (None, , filter_transform)
[16-Nov-2022 16:57:49] INFO vision_on_edge.azure_projects.models : Project (id, customvision_id, name): (None, , filter_transform)
[16-Nov-2022 16:57:49] INFO vision_on_edge.azure_projects.models : Project pre_save end
[16-Nov-2022 16:57:49] WARNING Backend-Training-App-Insight : create_project
[16-Nov-2022 16:57:49] INFO vision_on_edge.azure_projects.models : Project pre_save start
[16-Nov-2022 16:57:49] INFO vision_on_edge.azure_projects.models : Project (id, customvision_id, name): (None, , video_snippet_export)
[16-Nov-2022 16:57:49] INFO vision_on_edge.azure_projects.models : Project (id, customvision_id, name): (None, , video_snippet_export)
[16-Nov-2022 16:57:49] INFO vision_on_edge.azure_projects.models : Project pre_save end
[16-Nov-2022 16:57:49] WARNING Backend-Training-App-Insight : create_project
[16-Nov-2022 16:57:49] INFO vision_on_edge.azure_projects.models : Project pre_save start
[16-Nov-2022 16:57:49] INFO vision_on_edge.azure_projects.models : Project (id, customvision_id, name): (None, , iothub_export)
[16-Nov-2022 16:57:49] INFO vision_on_edge.azure_projects.models : Project (id, customvision_id, name): (None, , iothub_export)
[16-Nov-2022 16:57:49] INFO vision_on_edge.azure_projects.models : Project pre_save end
[16-Nov-2022 16:57:49] WARNING Backend-Training-App-Insight : create_project
[16-Nov-2022 16:57:49] INFO vision_on_edge.azure_projects.models : Project pre_save start
[16-Nov-2022 16:57:49] INFO vision_on_edge.azure_projects.models : Project (id, customvision_id, name): (None, , iotedge_export)
[16-Nov-2022 16:57:49] INFO vision_on_edge.azure_projects.models : Project (id, customvision_id, name): (None, , iotedge_export)
[16-Nov-2022 16:57:49] INFO vision_on_edge.azure_projects.models : Project pre_save end
[16-Nov-2022 16:57:49] WARNING Backend-Training-App-Insight : create_project
[16-Nov-2022 16:57:49] INFO vision_on_edge.azure_projects.models : Project pre_save start
[16-Nov-2022 16:57:49] INFO vision_on_edge.azure_projects.models : Project (id, customvision_id, name): (None, , http_export)
[16-Nov-2022 16:57:49] INFO vision_on_edge.azure_projects.models : Project (id, customvision_id, name): (None, , http_export)
[16-Nov-2022 16:57:49] INFO vision_on_edge.azure_projects.models : Project pre_save end
[16-Nov-2022 16:57:49] WARNING Backend-Training-App-Insight : create_project
[16-Nov-2022 16:57:49] WARNING vision_on_edge.azure_projects.symphony_client : Creating project: [REDACTED]
[16-Nov-2022 16:57:49] INFO vision_on_edge.azure_projects.utils : pull_cv_project_helper
[16-Nov-2022 16:57:49] INFO vision_on_edge.azure_projects.utils : customvision_project_id: [REDACTED]
[16-Nov-2022 16:57:49] INFO vision_on_edge.azure_projects.utils : is_partial True
[16-Nov-2022 16:57:49] INFO vision_on_edge.azure_projects.utils : Deleting all parts and images...
[16-Nov-2022 16:57:49] INFO vision_on_edge.azure_projects.utils : Handle by signals...
[16-Nov-2022 16:57:49] INFO vision_on_edge.azure_projects.utils : Pulling Parts...
[16-Nov-2022 16:57:49] INFO vision_on_edge.azure_projects.models : Project pre_save start
[16-Nov-2022 16:57:49] INFO vision_on_edge.azure_projects.models : Project (id, customvision_id, name): (None, [REDACTED], [REDACTED])
[16-Nov-2022 16:57:49] INFO vision_on_edge.azure_projects.models : Project (id, customvision_id, name): (None, [REDACTED], [REDACTED])
[16-Nov-2022 16:57:49] INFO vision_on_edge.azure_projects.models : Project pre_save end
[16-Nov-2022 16:57:49] WARNING Backend-Training-App-Insight : create_project
[16-Nov-2022 16:57:49] WARNING vision_on_edge.azure_projects.models : endpoint: westeurope.api.cognitive.microsoft.com
[16-Nov-2022 16:57:49] WARNING vision_on_edge.azure_projects.models : customvision_id: [REDACTED]
[16-Nov-2022 16:57:50] WARNING vision_on_edge.general.symphony_client : {'apiVersion': 'ai.symphony/v1', 'kind': 'Model', 'metadata': {'name': '[REDACTED]', 'labels': {}}, 'spec': {'properties': {'model.endpoint': 'westeurope.api.cognitive.microsoft.com', 'model.project': '[REDACTED]', 'model.subtype': 'customvision.ObjectDetection', 'model.type': 'customvision', 'model.symphony.id': '[REDACTED]', 'model.version.1': '[REDACTED]', 'state': 'trained', 'tags': '[[REDACTED], [REDACTED], [REDACTED], [REDACTED]]'}, 'displayName': [REDACTED]}}
[16-Nov-2022 16:57:50] WARNING vision_on_edge.general.symphony_client : fail
[16-Nov-2022 16:57:50] WARNING vision_on_edge.general.symphony_client : (422)
Reason: Unprocessable Entity
HTTP response headers: HTTPHeaderDict({'Audit-Id': 'e835f2cb-826a-407f-9c5c-a203ccea5fd4', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': 'f4d1c1f9-d5da-4a72-bdcd-1721df38a576', 'X-Kubernetes-Pf-Prioritylevel-Uid': '56b51218-639e-401b-a87c-71b75181461b', 'Date': 'Wed, 16 Nov 2022 16:57:50 GMT', 'Content-Length': '601'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"admission webhook \"vmodel.kb.io\" denied the request: Model.ai.symphony \"[REDACTED]\" is invalid: spec.displayName: Invalid value: \"[REDACTED]\": model display name is already taken","reason":"Invalid","details":{"name":"[REDACTED]","group":"ai.symphony","kind":"Model","causes":[{"reason":"FieldValueInvalid","message":"Invalid value: \"[REDACTED]\": model display name is already taken","field":"spec.displayName"}]},"code":422}

[16-Nov-2022 16:57:50] INFO vision_on_edge.azure_projects.models : Project pre_save start
[16-Nov-2022 16:57:50] INFO vision_on_edge.azure_projects.models : Project (id, customvision_id, name): (8, [REDACTED], [REDACTED])
[16-Nov-2022 16:57:51] INFO vision_on_edge.azure_projects.models : Project (id, customvision_id, name): (8, [REDACTED], [REDACTED])
[16-Nov-2022 16:57:51] INFO vision_on_edge.azure_projects.models : Project pre_save end
[16-Nov-2022 16:57:51] WARNING Backend-Training-App-Insight : create_project
[16-Nov-2022 16:57:51] WARNING vision_on_edge.azure_projects.models : endpoint: westeurope.api.cognitive.microsoft.com
[16-Nov-2022 16:57:51] WARNING vision_on_edge.azure_projects.models : customvision_id: [REDACTED]
[16-Nov-2022 16:57:51] INFO vision_on_edge.azure_projects.utils : Creating Part 0: [REDACTED] None
[16-Nov-2022 16:57:52] WARNING Backend-Training-App-Insight : create_part
[16-Nov-2022 16:57:52] INFO vision_on_edge.azure_projects.utils : Create Part 0: [REDACTED] None Success!
[16-Nov-2022 16:57:52] INFO vision_on_edge.azure_projects.utils : Try to download one image as icon.
[16-Nov-2022 16:57:52] INFO vision_on_edge.images.models : Saving as name [REDACTED]-i-a4c8b23573f541b5a58c68e327322396?sv=2020-04-08&se=2022-11-17T16%3A57%3A52Z&sr=b&sp=r&sig=UiPEEksGOr4TqoA2C6%2B%2BiferRU0NghVC3a7uXlUe3YQ%3D
[16-Nov-2022 16:57:52] INFO vision_on_edge.azure_projects.utils : Finding region with tag ([REDACTED], 58d793cd-241d-4bf4-a80d-2d346c827e40)
[16-Nov-2022 16:57:52] INFO vision_on_edge.azure_projects.utils : Region Found
[16-Nov-2022 16:57:52] INFO vision_on_edge.images.models : Setting Image labels
[16-Nov-2022 16:57:52] INFO vision_on_edge.images.models : Setting labels. Image size images/[REDACTED]-i-a4c8b23573f541b5a58c68e327322396sv2020-04-08se2022-11-17T163A573A52Zsrbspr_mzazpRl
[16-Nov-2022 16:57:52] INFO vision_on_edge.images.models : Set image labels success [{"x1": 317, "y1": 83, "x2": 575, "y2": 279, "part": 1}]
[16-Nov-2022 16:57:52] INFO vision_on_edge.images.models : Set image part_ids success ["1"]
[16-Nov-2022 16:57:52] INFO vision_on_edge.azure_projects.utils : Creating Part 1: [REDACTED] None
[16-Nov-2022 16:57:52] WARNING Backend-Training-App-Insight : create_part
[16-Nov-2022 16:57:52] INFO vision_on_edge.azure_projects.utils : Create Part 1: [REDACTED] None Success!
[16-Nov-2022 16:57:52] INFO vision_on_edge.azure_projects.utils : Try to download one image as icon.
[16-Nov-2022 16:57:52] INFO vision_on_edge.images.models : Saving as name [REDACTED]-i-8e208cfb429c401ba7584d312fc0a41c?sv=2020-04-08&se=2022-11-17T16%3A57%3A52Z&sr=b&sp=r&sig=xZKvpHmNqayT5472IJzR9v9WwonCK40N2okITSW7GN4%3D
[16-Nov-2022 16:57:52] INFO vision_on_edge.azure_projects.utils : Finding region with tag ([REDACTED], c8f5b956-300e-4738-92bf-3062fcdb5be9)
[16-Nov-2022 16:57:52] INFO vision_on_edge.azure_projects.utils : Region Found
[16-Nov-2022 16:57:52] INFO vision_on_edge.images.models : Setting Image labels
[16-Nov-2022 16:57:52] INFO vision_on_edge.images.models : Setting labels. Image size images/[REDACTED]-i-8e208cfb429c401ba7584d312fc0a41csv2020-04-08se2022-11-17T163A573A52Zsrbsp_9xhteFj
[16-Nov-2022 16:57:52] INFO vision_on_edge.images.models : Set image labels success [{"x1": 482, "y1": 178, "x2": 623, "y2": 231, "part": 2}]
[16-Nov-2022 16:57:52] INFO vision_on_edge.images.models : Set image part_ids success ["2"]
[16-Nov-2022 16:57:52] INFO vision_on_edge.azure_projects.utils : Creating Part 2: [REDACTED] None
[16-Nov-2022 16:57:52] WARNING Backend-Training-App-Insight : create_part
[16-Nov-2022 16:57:52] INFO vision_on_edge.azure_projects.utils : Create Part 2: [REDACTED] None Success!
[16-Nov-2022 16:57:52] INFO vision_on_edge.azure_projects.utils : Try to download one image as icon.
[16-Nov-2022 16:57:52] INFO vision_on_edge.images.models : Saving as name [REDACTED]-i-a4c8b23573f541b5a58c68e327322396?sv=2020-04-08&se=2022-11-17T16%3A57%3A52Z&sr=b&sp=r&sig=UiPEEksGOr4TqoA2C6%2B%2BiferRU0NghVC3a7uXlUe3YQ%3D
[16-Nov-2022 16:57:52] INFO vision_on_edge.azure_projects.utils : Finding region with tag ([REDACTED], ae029404-ccb0-43a2-8808-3533f318c13b)
[16-Nov-2022 16:57:52] INFO vision_on_edge.azure_projects.utils : Region Found
[16-Nov-2022 16:57:52] INFO vision_on_edge.images.models : Setting Image labels
[16-Nov-2022 16:57:52] INFO vision_on_edge.images.models : Setting labels. Image size images/[REDACTED]-i-a4c8b23573f541b5a58c68e327322396sv2020-04-08se2022-11-17T163A573A52_dUevTvt
[16-Nov-2022 16:57:52] INFO vision_on_edge.images.models : Set image labels success [{"x1": 0, "y1": 0, "x2": 296, "y2": 476, "part": 3}]
[16-Nov-2022 16:57:52] INFO vision_on_edge.images.models : Set image part_ids success ["3"]
[16-Nov-2022 16:57:52] INFO vision_on_edge.azure_projects.utils : Creating Part 3: [REDACTED] None
[16-Nov-2022 16:57:53] WARNING Backend-Training-App-Insight : create_part
[16-Nov-2022 16:57:53] INFO vision_on_edge.azure_projects.utils : Create Part 3: [REDACTED] None Success!
[16-Nov-2022 16:57:53] INFO vision_on_edge.azure_projects.utils : Try to download one image as icon.
[16-Nov-2022 16:57:53] INFO vision_on_edge.images.models : Saving as name [REDACTED]-i-a4c8b23573f541b5a58c68e327322396?sv=2020-04-08&se=2022-11-17T16%3A57%3A53Z&sr=b&sp=r&sig=aWEms3SqSzg27n2AteA9OE7IIgYK3s%2FaXETrfJ3pnv0%3D
[16-Nov-2022 16:57:53] INFO vision_on_edge.azure_projects.utils : Finding region with tag ([REDACTED], 371f5eef-3e92-4dc0-a789-85f4179a4c84)
[16-Nov-2022 16:57:53] INFO vision_on_edge.azure_projects.utils : Region Found
[16-Nov-2022 16:57:53] INFO vision_on_edge.images.models : Setting Image labels
[16-Nov-2022 16:57:53] INFO vision_on_edge.images.models : Setting labels. Image size images/[REDACTED]-i-a4c8b23573f541b5a58c68e327322396sv2020-04-08se2022-11-17T163A573A53Zsrbsp_aIPLg2N
[16-Nov-2022 16:57:53] INFO vision_on_edge.images.models : Set image labels success [{"x1": 368, "y1": 359, "x2": 586, "y2": 480, "part": 4}]
[16-Nov-2022 16:57:53] INFO vision_on_edge.images.models : Set image part_ids success ["4"]
[16-Nov-2022 16:57:53] INFO vision_on_edge.azure_projects.utils : Pulled 4 Parts... End
[16-Nov-2022 16:57:53] WARNING vision_on_edge.azure_projects.symphony_client : Creating project: pedestrian-and-vehicle-detector
[16-Nov-2022 16:57:53] INFO vision_on_edge.azure_projects.models : Project pre_save start
[16-Nov-2022 16:57:53] INFO vision_on_edge.azure_projects.models : Project (id, customvision_id, name): (None, pedestrian-and-vehicle-detector-adas-0001, pedestrian-and-vehicle-detector)
[16-Nov-2022 16:57:53] INFO vision_on_edge.azure_projects.models : Project (id, customvision_id, name): (None, , pedestrian-and-vehicle-detector)
[16-Nov-2022 16:57:53] INFO vision_on_edge.azure_projects.models : Project pre_save end
[16-Nov-2022 16:57:53] WARNING Backend-Training-App-Insight : create_project
[16-Nov-2022 16:57:53] INFO vision_on_edge.azure_projects.symphony_client : Project: pedestrian-and-vehicle-detector.
[16-Nov-2022 16:57:53] WARNING vision_on_edge.azure_projects.symphony_client : Creating project: person-detection
[16-Nov-2022 16:57:53] INFO vision_on_edge.azure_projects.models : Project pre_save start
[16-Nov-2022 16:57:53] INFO vision_on_edge.azure_projects.models : Project (id, customvision_id, name): (None, person-detection-retail-0013, person-detection)
[16-Nov-2022 16:57:53] INFO vision_on_edge.azure_projects.models : Project (id, customvision_id, name): (None, , person-detection)
[16-Nov-2022 16:57:53] INFO vision_on_edge.azure_projects.models : Project pre_save end
[16-Nov-2022 16:57:53] WARNING Backend-Training-App-Insight : create_project
[16-Nov-2022 16:57:53] INFO vision_on_edge.azure_projects.symphony_client : Project: person-detection.
[16-Nov-2022 16:57:53] WARNING vision_on_edge.azure_projects.symphony_client : Creating project: vehicle-attributes-recognition
[16-Nov-2022 16:57:53] INFO vision_on_edge.azure_projects.models : Project pre_save start
[16-Nov-2022 16:57:53] INFO vision_on_edge.azure_projects.models : Project (id, customvision_id, name): (None, vehicle-attributes-recognition-barrier-0039, vehicle-attributes-recognition)
[16-Nov-2022 16:57:53] INFO vision_on_edge.azure_projects.models : Project (id, customvision_id, name): (None, , vehicle-attributes-recognition)
[16-Nov-2022 16:57:53] INFO vision_on_edge.azure_projects.models : Project pre_save end
[16-Nov-2022 16:57:53] WARNING Backend-Training-App-Insight : create_project
[16-Nov-2022 16:57:53] INFO vision_on_edge.azure_projects.symphony_client : Project: vehicle-attributes-recognition.
[16-Nov-2022 16:57:53] INFO vision_on_edge.azure_parts.apps : Azure Part App Config ready while running server
[16-Nov-2022 16:57:53] INFO vision_on_edge.azure_parts.apps : Creating demo parts...
[16-Nov-2022 16:57:53] INFO vision_on_edge.locations.apps : App Config ready start while running server
[16-Nov-2022 16:57:53] INFO vision_on_edge.locations.apps : App Config ready end while running server
[16-Nov-2022 16:57:53] INFO vision_on_edge.cameras.apps : App ready ready while running server
[16-Nov-2022 16:57:55] INFO vision_on_edge.cameras.utils : verify_rtsp rtsp://:@20.4.75.231:8554/live.stream
[16-Nov-2022 16:57:57] WARNING Backend-Training-App-Insight : create_camera
************************************
CONFIGURATION:
  TRAINING_KEY: [REDACTED]
  ENDPOINT: https://westeurope.api.cognitive.microsoft.com/
************************************
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/django/db/models/query.py", line 589, in update_or_create
    obj = self.select_for_update().get(**kwargs)
  File "/usr/local/lib/python3.8/site-packages/django/db/models/query.py", line 429, in get
    raise self.model.DoesNotExist(
vision_on_edge.cameras.models.DoesNotExist: Camera matching query does not exist.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python3.8/site-packages/django/db/backends/sqlite3/base.py", line 413, in execute
    return Database.Cursor.execute(self, query, params)
sqlite3.OperationalError: database is locked

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "manage.py", line 30, in <module>
    main()
  File "manage.py", line 26, in main
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 401, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 377, in execute
    django.setup()
  File "/usr/local/lib/python3.8/site-packages/django/__init__.py", line 24, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "/usr/local/lib/python3.8/site-packages/django/apps/registry.py", line 122, in populate
    app_config.ready()
  File "/app/vision_on_edge/cameras/apps.py", line 35, in ready
    device_client.load_symphony_objects()
  File "/app/vision_on_edge/cameras/symphony_client.py", line 137, in load_symphony_objects
    cam_obj, created = Camera.objects.update_or_create(
  File "/usr/local/lib/python3.8/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/django/db/models/query.py", line 594, in update_or_create
    obj, created = self._create_object_from_params(kwargs, params, lock=True)
  File "/usr/local/lib/python3.8/site-packages/django/db/models/query.py", line 610, in _create_object_from_params
    obj = self.create(**params)
  File "/usr/local/lib/python3.8/site-packages/django/db/models/query.py", line 447, in create
    obj.save(force_insert=True, using=self.db)
  File "/usr/local/lib/python3.8/site-packages/django/db/models/base.py", line 753, in save
    self.save_base(using=using, force_insert=force_insert,
  File "/usr/local/lib/python3.8/site-packages/django/db/models/base.py", line 790, in save_base
    updated = self._save_table(
  File "/usr/local/lib/python3.8/site-packages/django/db/models/base.py", line 895, in _save_table
    results = self._do_insert(cls._base_manager, using, fields, returning_fields, raw)
  File "/usr/local/lib/python3.8/site-packages/django/db/models/base.py", line 933, in _do_insert
    return manager._insert(
  File "/usr/local/lib/python3.8/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/django/db/models/query.py", line 1254, in _insert
    return query.get_compiler(using=using).execute_sql(returning_fields)
  File "/usr/local/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1397, in execute_sql
    cursor.execute(sql, params)
  File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 98, in execute
    return super().execute(sql, params)
  File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 66, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python3.8/site-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python3.8/site-packages/django/db/backends/sqlite3/base.py", line 413, in execute
    return Database.Cursor.execute(self, query, params)
django.db.utils.OperationalError: database is locked
[16-Nov-2022 16:57:57] INFO azure.identity._internal.get_token_mixin : ClientSecretCredential.get_token succeeded
[16-Nov-2022 16:57:58] INFO azure.identity._internal.get_token_mixin : ClientSecretCredential.get_token succeeded
[16-Nov-2022 16:57:59] INFO azure.identity._internal.get_token_mixin : ClientSecretCredential.get_token succeeded
[16-Nov-2022 16:58:00] INFO azure.identity._internal.get_token_mixin : ClientSecretCredential.get_token succeeded
[16-Nov-2022 16:58:01] INFO azure.identity._internal.get_token_mixin : ClientSecretCredential.get_token succeeded
[16-Nov-2022 16:58:02] INFO azure.identity._internal.get_token_mixin : ClientSecretCredential.get_token succeeded
[16-Nov-2022 16:58:02] INFO azure.identity._internal.get_token_mixin : ClientSecretCredential.get_token succeeded
[16-Nov-2022 16:58:03] INFO azure.identity._internal.get_token_mixin : ClientSecretCredential.get_token succeeded
[16-Nov-2022 16:58:04] INFO azure.identity._internal.get_token_mixin : ClientSecretCredential.get_token succeeded
[16-Nov-2022 16:58:04] INFO azure.identity._internal.get_token_mixin : ClientSecretCredential.get_token succeeded
[16-Nov-2022 16:58:04] INFO azure.identity._internal.get_token_mixin : ClientSecretCredential.get_token succeeded
[16-Nov-2022 16:58:04] INFO azure.identity._internal.get_token_mixin : ClientSecretCredential.get_token succeeded
[16-Nov-2022 16:58:04] INFO azure.identity._internal.get_token_mixin : ClientSecretCredential.get_token succeeded
[16-Nov-2022 16:58:04] INFO azure.identity._internal.get_token_mixin : ClientSecretCredential.get_token succeeded
[16-Nov-2022 16:58:04] INFO azure.identity._internal.get_token_mixin : ClientSecretCredential.get_token succeeded
[16-Nov-2022 16:58:04] INFO azure.identity._internal.get_token_mixin : ClientSecretCredential.get_token succeeded
[16-Nov-2022 16:58:04] INFO azure.identity._internal.get_token_mixin : ClientSecretCredential.get_token succeeded
Haishi2016 commented 1 year ago

Close as can't reproduce