Cannot connect to the CVAT server

shintaro-matsui commented 1 year ago

My actions before raising this issue

Searched for previous issues (#5591)

Expected Behaviour

I was expecting CVAT to run in browser at http://localhost:8080, as described in the guidelines.

Current Behaviour

After running docker compose up -d and then accessing http://localhost:8080, the following pop-up error appears

Cannot connect to the server Make sure the CVAT backend and all necessary services (Database, Redis and Open Policy Agent) are running and available. 0 or >earlier, manual actions may be needed, see the Upgrade Guide.

Possible Solution

I started my PC and run CVAT, and now I get the above error. I don't remember doing anything special the last time I did this.

Your Environment

Ubuntu 18.04 LTS
Docker version 23.0.3, build 3e7cbfd

I am not familiar with Docker and I have looked around but could not find a solution. What should I check first?

azhavoro commented 1 year ago

Did you follow the recommendation and check the container logs?

ganindu7 commented 1 year ago

I have the same issue (but I installed with a helm chart)

in the backend server pod I ran health checks

python manage.py health_check
Cache backend: default   ... working
Cache backend: media     ... working
DatabaseBackend          ... working
DiskUsage                ... working
MemoryUsage              ... working
MigrationsHealthCheck    ... working
OPAHealthCheck           ... working
$ python manage.py testserver
Creating test database for alias 'default'...
Getting flipped tasks...
Conversion started...

Start schema migration...
INFO:0024_auto_20191023_1025:
Start schema migration...

Schema migration is finished...
INFO:0024_auto_20191023_1025:
Schema migration is finished...

Start data migration...
INFO:0024_auto_20191023_1025:
Start data migration...
The data migration has been started for creating manifest`s files
The data migration has been started for creating manifest`s files
INFO:0038_manifest:The data migration has been started for creating manifest`s files
Need to update 0 data objects
Need to update 0 data objects
INFO:0038_manifest:Need to update 0 data objects
Migration has been started. Need to create 0 directories.
Migration has been started. Need to create 0 directories.
INFO:0055_jobs_directories:Migration has been started. Need to create 0 directories.
Migration has been finished successfully.
Migration has been finished successfully.
INFO:0055_jobs_directories:Migration has been finished successfully.
Migration has been started. Need to create 0 previews.
Migration has been started. Need to create 0 previews.
INFO:0056_jobs_previews:Migration has been started. Need to create 0 previews.

Deleting Data previews...
INFO:0062_delete_previews:
Deleting Data previews...

Deleting Job previews...
INFO:0062_delete_previews:
Deleting Job previews...

Deleting CloudStorage previews...
INFO:0062_delete_previews:
Deleting CloudStorage previews...

Deleting skeleton Labels without skeletons...
INFO:0064_delete_or_rename_wrong_labels:
Deleting skeleton Labels without skeletons...

Deleting duplicate skeleton sublabels and renaming duplicate Labels...
INFO:0064_delete_or_rename_wrong_labels:
Deleting duplicate skeleton sublabels and renaming duplicate Labels...
CommandError: Error: No database fixture specified. Please provide the path of at least one fixture in the command line.

the admin panel login works fine

http://cvat.gsrv.lan:32608/admin/ Screenshot from 2023-05-26 19-48-13

But this happends when I try to log in to the gui. Screenshot from 2023-05-26 19-50-46

this is the full backend server pod log

g@gsrv:~$ kubectl logs  -n cvat-gsrv  cvat-backend-server-6887d99fbd-t8wph -f
2023-05-26 18:28:05,045 INFO RPC interface 'supervisor' initialized
2023-05-26 18:28:05,045 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2023-05-26 18:28:05,045 INFO supervisord started with pid 1
2023-05-26 18:28:06,049 INFO spawned: 'ssh-agent' with pid 7
2023-05-26 18:28:06,052 INFO spawned: 'clamav_update' with pid 8
2023-05-26 18:28:06,071 INFO spawned: 'runserver' with pid 9
2023-05-26 18:28:06,071 DEBG fd 10 closed, stopped monitoring <POutputDispatcher at 139821883798432 for <Subprocess at 139821883787872 with name clamav_update in state STARTING> (stdout)>
2023-05-26 18:28:06,071 DEBG fd 14 closed, stopped monitoring <POutputDispatcher at 139821883798288 for <Subprocess at 139821883787872 with name clamav_update in state STARTING> (stderr)>
2023-05-26 18:28:06,071 INFO exited: clamav_update (exit status 0; not expected)
2023-05-26 18:28:06,071 DEBG received SIGCHLD indicating a child quit
2023-05-26 18:28:06,074 DEBG 'ssh-agent' stdout output:
SSH_AUTH_SOCK=/tmp/ssh-agent.sock; export SSH_AUTH_SOCK;
echo Agent pid 7;

2023-05-26 18:28:06,074 DEBG 'ssh-agent' stderr output:
debug1: new_socket: type = SOCKET
debug2: fd 3 setting O_NONBLOCK

2023-05-26 18:28:06,083 DEBG 'runserver' stderr output:
wait-for-it.sh: waiting for cvat-postgresql:5432 without a timeout

2023-05-26 18:28:06,089 DEBG 'runserver' stderr output:
wait-for-it.sh: cvat-postgresql:5432 is available after 0 seconds

2023-05-26 18:28:06,090 DEBG 'runserver' stderr output:
bash: cannot set terminal process group (-1): Inappropriate ioctl for device
bash: no job control in this shell

2023-05-26 18:28:06,326 DEBG 'ssh-agent' stderr output:
debug1: new_socket: type = CONNECTION
debug2: fd 4 setting O_NONBLOCK

2023-05-26 18:28:06,330 DEBG 'ssh-agent' stderr output:
debug1: new_socket: type = CONNECTION
debug2: fd 4 setting O_NONBLOCK
debug1: process_message: socket 1 (fd=4) type 11
debug2: process_request_identities: entering
debug2: process_request_identities: replying with 0 allowed of 0 available keys

2023-05-26 18:28:06,339 DEBG 'ssh-agent' stderr output:
debug1: new_socket: type = CONNECTION
debug2: fd 4 setting O_NONBLOCK

2023-05-26 18:28:06,340 DEBG 'ssh-agent' stderr output:
debug1: process_message: socket 1 (fd=4) type 17
debug2: process_add_identity: entering

2023-05-26 18:28:06,342 DEBG 'ssh-agent' stderr output:
debug1: process_add_identity: add ssh-rsa SHA256:EBbB2b6j9z65OMIXjLKm/Tm2zxhjNm3CQwE13KlLXM8 "django@cvat-backend-worker-export-5b4c6dfcbf-6dvrn" (life: 0) (confirm: 0) (provider: none) (destination constraints: 0)

2023-05-26 18:28:06,342 DEBG 'runserver' stderr output:
Identity added: /home/django/.ssh/id_rsa (django@cvat-backend-worker-export-5b4c6dfcbf-6dvrn)

2023-05-26 18:28:07,343 INFO success: ssh-agent entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-05-26 18:28:07,347 INFO spawned: 'clamav_update' with pid 91
2023-05-26 18:28:07,347 INFO success: runserver entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-05-26 18:28:07,372 DEBG fd 10 closed, stopped monitoring <POutputDispatcher at 139821883798384 for <Subprocess at 139821883787872 with name clamav_update in state STARTING> (stdout)>
2023-05-26 18:28:07,372 DEBG fd 16 closed, stopped monitoring <POutputDispatcher at 139821883798480 for <Subprocess at 139821883787872 with name clamav_update in state STARTING> (stderr)>
2023-05-26 18:28:07,372 INFO exited: clamav_update (exit status 0; not expected)
2023-05-26 18:28:07,372 DEBG received SIGCHLD indicating a child quit
2023-05-26 18:28:09,385 INFO spawned: 'clamav_update' with pid 284
2023-05-26 18:28:09,389 DEBG fd 10 closed, stopped monitoring <POutputDispatcher at 139821883798432 for <Subprocess at 139821883787872 with name clamav_update in state STARTING> (stdout)>
2023-05-26 18:28:09,389 DEBG fd 16 closed, stopped monitoring <POutputDispatcher at 139821883787920 for <Subprocess at 139821883787872 with name clamav_update in state STARTING> (stderr)>
2023-05-26 18:28:09,389 INFO exited: clamav_update (exit status 0; not expected)
2023-05-26 18:28:09,389 DEBG received SIGCHLD indicating a child quit
2023-05-26 18:28:12,401 INFO spawned: 'clamav_update' with pid 285
2023-05-26 18:28:12,405 DEBG fd 10 closed, stopped monitoring <POutputDispatcher at 139821883798384 for <Subprocess at 139821883787872 with name clamav_update in state STARTING> (stdout)>
2023-05-26 18:28:12,405 DEBG fd 16 closed, stopped monitoring <POutputDispatcher at 139821883798576 for <Subprocess at 139821883787872 with name clamav_update in state STARTING> (stderr)>
2023-05-26 18:28:12,405 INFO exited: clamav_update (exit status 0; not expected)
2023-05-26 18:28:12,405 DEBG received SIGCHLD indicating a child quit
2023-05-26 18:28:13,408 INFO gave up: clamav_update entered FATAL state, too many start retries too quickly
2023-05-26 18:28:36,921 DEBG 'runserver' stdout output:
SSH keys were not found
Copying them from keys volume

2023-05-26 18:28:36,922 DEBG 'runserver' stderr output:
Traceback (most recent call last):
  File "/opt/venv/lib/python3.10/site-packages/django/db/backends/base/base.py", line 289, in ensure_connection
    self.connect()
  File "/opt/venv/lib/python3.10/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/opt/venv/lib/python3.10/site-packages/django/db/backends/base/base.py", line 270, in connect
    self.connection = self.get_new_connection(conn_params)
  File "/opt/venv/lib/python3.10/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/opt/venv/lib/python3.10/site-packages/django/db/backends/postgresql/base.py", line 275, in get_new_connection
    connection = self.Database.connect(**conn_params)
  File "/opt/venv/lib/python3.10/site-packages/psycopg2/__init__.py", line 122, in connect
    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: connection to server at "cvat-postgresql" (10.106.161.41), port 5432 failed: Connection refused
    Is the server running on that host and accepting TCP/IP connections?

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/django/manage.py", line 21, in <module>
    execute_from_command_line(sys.argv)
  File "/opt/venv/lib/python3.10/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "/opt/venv/lib/python3.10/site-packages/django/core/management/__init__.py", line 436, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/opt/venv/lib/python3.10/site-packages/django/core/management/base.py", line 412, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/opt/venv/lib/python3.10/site-packages/django/core/management/base.py", line 458, in execute
    output = self.handle(*args, **options)
  File "/opt/venv/lib/python3.10/site-packages/django/core/management/base.py", line 106, in wrapper
    res = handle_func(*args, **kwargs)
  File "/opt/venv/lib/python3.10/site-packages/django/core/management/commands/migrate.py", line 117, in handle
    executor = MigrationExecutor(connection, self.migration_progress_callback)
  File "/opt/venv/lib/python3.10/site-packages/django/db/migrations/executor.py", line 18, in __init__
    self.loader = MigrationLoader(self.connection)
  File "/opt/venv/lib/python3.10/site-packages/django/db/migrations/loader.py", line 58, in __init__
    self.build_graph()
  File "/opt/venv/lib/python3.10/site-packages/django/db/migrations/loader.py", line 235, in build_graph
    self.applied_migrations = recorder.applied_migrations()
  File "/opt/venv/lib/python3.10/site-packages/django/db/migrations/recorder.py", line 81, in applied_migrations
    if self.has_table():
  File "/opt/venv/lib/python3.10/site-packages/django/db/migrations/recorder.py", line 57, in has_table
    with self.connection.cursor() as cursor:
  File "/opt/venv/lib/python3.10/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/opt/venv/lib/python3.10/site-packages/django/db/backends/base/base.py", line 330, in cursor
    return self._cursor()
  File "/opt/venv/lib/python3.10/site-packages/django/db/backends/base/base.py", line 306, in _cursor
    self.ensure_connection()
  File "/opt/venv/lib/python3.10/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/opt/venv/lib/python3.10/site-packages/django/db/backends/base/base.py", line 288, in ensure_connection

2023-05-26 18:28:36,922 DEBG 'runserver' stderr output:
    with self.wrap_database_errors:
  File "/opt/venv/lib/python3.10/site-packages/django/db/utils.py", line 91, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/opt/venv/lib/python3.10/site-packages/django/db/backends/base/base.py", line 289, in ensure_connection
    self.connect()
  File "/opt/venv/lib/python3.10/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/opt/venv/lib/python3.10/site-packages/django/db/backends/base/base.py", line 270, in connect
    self.connection = self.get_new_connection(conn_params)
  File "/opt/venv/lib/python3.10/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/opt/venv/lib/python3.10/site-packages/django/db/backends/postgresql/base.py", line 275, in get_new_connection
    connection = self.Database.connect(**conn_params)
  File "/opt/venv/lib/python3.10/site-packages/psycopg2/__init__.py", line 122, in connect
    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: connection to server at "cvat-postgresql" (10.106.161.41), port 5432 failed: Connection refused
    Is the server running on that host and accepting TCP/IP connections?

2023-05-26 18:28:38,014 DEBG fd 15 closed, stopped monitoring <POutputDispatcher at 139821883798768 for <Subprocess at 139821883788064 with name runserver in state RUNNING> (stdout)>
2023-05-26 18:28:38,014 DEBG fd 19 closed, stopped monitoring <POutputDispatcher at 139821883798816 for <Subprocess at 139821883788064 with name runserver in state RUNNING> (stderr)>
2023-05-26 18:28:38,014 INFO exited: runserver (exit status 1; not expected)
2023-05-26 18:28:38,014 DEBG received SIGCHLD indicating a child quit
2023-05-26 18:28:39,040 INFO spawned: 'runserver' with pid 286
2023-05-26 18:28:39,051 DEBG 'runserver' stderr output:
wait-for-it.sh: waiting for cvat-postgresql:5432 without a timeout

2023-05-26 18:28:40,052 INFO success: runserver entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-05-26 18:28:43,080 DEBG 'runserver' stderr output:
wait-for-it.sh: cvat-postgresql:5432 is available after 4 seconds

2023-05-26 18:28:43,080 DEBG 'runserver' stderr output:
bash: cannot set terminal process group (-1): Inappropriate ioctl for device
bash: no job control in this shell

2023-05-26 18:28:43,302 DEBG 'ssh-agent' stderr output:
debug1: new_socket: type = CONNECTION
debug2: fd 4 setting O_NONBLOCK

2023-05-26 18:28:43,302 DEBG 'ssh-agent' stderr output:
debug1: process_message: socket 1 (fd=4) type 17
debug2: process_add_identity: entering

2023-05-26 18:28:43,304 DEBG 'ssh-agent' stderr output:
debug3: identity_permitted: entering: key RSA comment "django@cvat-backend-worker-export-5b4c6dfcbf-6dvrn", 0 socket bindings, 0 constraints
debug1: process_add_identity: add ssh-rsa SHA256:EBbB2b6j9z65OMIXjLKm/Tm2zxhjNm3CQwE13KlLXM8 "django@cvat-backend-worker-export-5b4c6dfcbf-6dvrn" (life: 0) (confirm: 0) (provider: none) (destination constraints: 0)

2023-05-26 18:28:43,307 DEBG 'ssh-agent' stderr output:
debug1: new_socket: type = CONNECTION
debug2: fd 4 setting O_NONBLOCK
debug1: process_message: socket 1 (fd=4) type 11
debug2: process_request_identities: entering
debug3: identity_permitted: entering: key RSA comment "django@cvat-backend-worker-export-5b4c6dfcbf-6dvrn", 0 socket bindings, 0 constraints
debug2: process_request_identities: replying with 1 allowed of 1 available keys

2023-05-26 18:29:29,912 DEBG 'runserver' stdout output:
Operations to perform:
  Apply all migrations: account, admin, auth, authtoken, contenttypes, dataset_repo, db, engine, organizations, sessions, sites, socialaccount, webhooks
Running migrations:
  No migrations to apply.

2023-05-26 18:29:31,826 DEBG 'ssh-agent' stderr output:
debug1: new_socket: type = CONNECTION
debug2: fd 4 setting O_NONBLOCK

2023-05-26 18:29:31,826 DEBG 'ssh-agent' stderr output:
debug1: process_message: socket 1 (fd=4) type 17
debug2: process_add_identity: entering
debug3: identity_permitted: entering: key RSA comment "django@cvat-backend-worker-export-5b4c6dfcbf-6dvrn", 0 socket bindings, 0 constraints
debug1: process_add_identity: add ssh-rsa SHA256:EBbB2b6j9z65OMIXjLKm/Tm2zxhjNm3CQwE13KlLXM8 "django@cvat-backend-worker-export-5b4c6dfcbf-6dvrn" (life: 0) (confirm: 0) (provider: none) (destination constraints: 0)

2023-05-26 18:29:31,828 DEBG 'ssh-agent' stderr output:
debug1: new_socket: type = CONNECTION
debug2: fd 4 setting O_NONBLOCK
debug1: process_message: socket 1 (fd=4) type 11
debug2: process_request_identities: entering
debug3: identity_permitted: entering: key RSA comment "django@cvat-backend-worker-export-5b4c6dfcbf-6dvrn", 0 socket bindings, 0 constraints
debug2: process_request_identities: replying with 1 allowed of 1 available keys

2023-05-26 18:29:32,791 DEBG 'runserver' stdout output:

165 static files copied to '/home/django/static'.

2023-05-26 18:29:33,339 DEBG 'ssh-agent' stderr output:
debug1: new_socket: type = CONNECTION
debug2: fd 4 setting O_NONBLOCK

2023-05-26 18:29:33,340 DEBG 'ssh-agent' stderr output:
debug1: process_message: socket 1 (fd=4) type 17
debug2: process_add_identity: entering

2023-05-26 18:29:33,342 DEBG 'ssh-agent' stderr output:
debug3: identity_permitted: entering: key RSA comment "django@cvat-backend-worker-export-5b4c6dfcbf-6dvrn", 0 socket bindings, 0 constraints
debug1: process_add_identity: add ssh-rsa SHA256:EBbB2b6j9z65OMIXjLKm/Tm2zxhjNm3CQwE13KlLXM8 "django@cvat-backend-worker-export-5b4c6dfcbf-6dvrn" (life: 0) (confirm: 0) (provider: none) (destination constraints: 0)

2023-05-26 18:29:33,345 DEBG 'ssh-agent' stderr output:
debug1: new_socket: type = CONNECTION
debug2: fd 4 setting O_NONBLOCK
debug1: process_message: socket 1 (fd=4) type 11
debug2: process_request_identities: entering
debug3: identity_permitted: entering: key RSA comment "django@cvat-backend-worker-export-5b4c6dfcbf-6dvrn", 0 socket bindings, 0 constraints
debug2: process_request_identities: replying with 1 allowed of 1 available keys

2023-05-26 18:30:48,562 DEBG 'runserver' stdout output:
Successfully ran command.
Server URL         : http://localhost:8080/
Server Root        : /tmp/cvat-server
Server Conf        : /tmp/cvat-server/httpd.conf
Error Log File     : /dev/stderr (INFO)
Operating Mode     : daemon
Request Capacity   : 5 (1 process * 5 threads)
Request Timeout    : 60 (seconds)
Startup Timeout    : 15 (seconds)
Queue Backlog      : 100 (connections)
Queue Timeout      : 45 (seconds)
Server Capacity    : 20 (event/worker), 20 (prefork)
Server Backlog     : 500 (connections)
Locale Setting     : C.UTF-8

2023-05-26 18:30:48,679 DEBG 'runserver' stderr output:
[Fri May 26 18:30:48.678678 2023] [mpm_event:notice] [pid 286:tid 140099398150016] AH00489: Apache/2.4.52 (Ubuntu) mod_wsgi/4.9.4 Python/3.10 configured -- resuming normal operations
[Fri May 26 18:30:48.678746 2023] [mpm_event:info] [pid 286:tid 140099398150016] AH00490: Server built: 2023-03-01T22:43:55
[Fri May 26 18:30:48.678749 2023] [core:notice] [pid 286:tid 140099398150016] AH00094: Command line: 'apache2 (mod_wsgi-express) -f /tmp/cvat-server/httpd.conf -D MOD_WSGI_KEEP_ALIVE -D MOD_WSGI_MPM_ENABLE_EVENT_MODULE -D MOD_WSGI_MPM_EXISTS_EVENT_MODULE -D MOD_WSGI_MPM_EXISTS_WORKER_MODULE -D MOD_WSGI_MPM_EXISTS_PREFORK_MODULE -D FOREGROUND'

2023-05-26 18:30:48,679 DEBG 'runserver' stderr output:
[Fri May 26 18:30:48.678697 2023] [wsgi:info] [pid 1164:tid 140099398150016] mod_wsgi (pid=1164): Starting process 'localhost:8080' with threads=5.
[Fri May 26 18:30:48.679233 2023] [wsgi:info] [pid 1164:tid 140099398150016] mod_wsgi (pid=1164): Python home /opt/venv.
[Fri May 26 18:30:48.679280 2023] [wsgi:info] [pid 1164:tid 140099398150016] mod_wsgi (pid=1164): Initializing Python.

2023-05-26 18:30:48,699 DEBG 'runserver' stderr output:
[Fri May 26 18:30:48.699323 2023] [wsgi:info] [pid 1164:tid 140099398150016] mod_wsgi (pid=1164): Attach interpreter ''.

2023-05-26 18:30:48,703 DEBG 'runserver' stderr output:
[Fri May 26 18:30:48.703031 2023] [wsgi:info] [pid 1164:tid 140099398150016] mod_wsgi (pid=1164): Imported 'mod_wsgi'.
[Fri May 26 18:30:48.703070 2023] [wsgi:info] [pid 1164:tid 140099398150016] mod_wsgi (pid=1164, process='localhost:8080', application=''): Loading Python script file '/tmp/cvat-server/handler.wsgi'.

2023-05-26 18:30:49,026 DEBG 'ssh-agent' stderr output:
debug1: new_socket: type = CONNECTION
debug2: fd 4 setting O_NONBLOCK

2023-05-26 18:30:49,026 DEBG 'ssh-agent' stderr output:
debug1: process_message: socket 1 (fd=4) type 17
debug2: process_add_identity: entering
debug3: identity_permitted: entering: key RSA comment "django@cvat-backend-worker-export-5b4c6dfcbf-6dvrn", 0 socket bindings, 0 constraints
debug1: process_add_identity: add ssh-rsa SHA256:EBbB2b6j9z65OMIXjLKm/Tm2zxhjNm3CQwE13KlLXM8 "django@cvat-backend-worker-export-5b4c6dfcbf-6dvrn" (life: 0) (confirm: 0) (provider: none) (destination constraints: 0)

2023-05-26 18:30:49,029 DEBG 'ssh-agent' stderr output:
debug1: new_socket: type = CONNECTION
debug2: fd 4 setting O_NONBLOCK
debug1: process_message: socket 1 (fd=4) type 11
debug2: process_request_identities: entering
debug3: identity_permitted: entering: key RSA comment "django@cvat-backend-worker-export-5b4c6dfcbf-6dvrn", 0 socket bindings, 0 constraints
debug2: process_request_identities: replying with 1 allowed of 1 available keys

2023-05-26 18:31:01,333 DEBG 'runserver' stderr output:
[Fri May 26 18:31:01.333018 2023] [wsgi:info] [pid 1164:tid 140096294712896] mod_wsgi (pid=1164): Application startup timer triggered 'localhost:8080'.

2023-05-26 18:31:01,333 DEBG 'runserver' stderr output:
[Fri May 26 18:31:01.333154 2023] [wsgi:info] [pid 1164:tid 140096294712896] mod_wsgi (pid=1164): Application startup timer cancelled 'localhost:8080'.

2023-05-26 18:31:22,809 DEBG 'runserver' stderr output:
[Fri May 26 18:31:22.808466 2023] [wsgi:error] [pid 1164:tid 140096294712896] [remote 192.168.251.158:44468] [2023-05-26 18:31:22,808] WARNING django.request: Unauthorized: /api/events

2023-05-26 18:31:22,809 DEBG 'runserver' stderr output:
[Fri May 26 18:31:22.808565 2023] [wsgi:error] [pid 1164:tid 140096294712896] [remote 192.168.251.158:44468] WARNING:django.request:Unauthorized: /api/events

2023-05-26 18:32:12,839 DEBG 'runserver' stderr output:
[Fri May 26 18:32:12.838749 2023] [wsgi:error] [pid 1164:tid 140096261142080] [remote 192.168.251.158:45660] [2023-05-26 18:32:12,838] WARNING django.request: Unauthorized: /api/events

2023-05-26 18:32:12,841 DEBG 'runserver' stderr output:
[Fri May 26 18:32:12.841036 2023] [wsgi:error] [pid 1164:tid 140096261142080] [remote 192.168.251.158:45660] WARNING:django.request:Unauthorized: /api/events

2023-05-26 18:32:52,799 DEBG 'runserver' stderr output:
[Fri May 26 18:32:52.799619 2023] [wsgi:error] [pid 1164:tid 140096261142080] [remote 192.168.251.158:52798] [2023-05-26 18:32:52,799] WARNING django.request: Unauthorized: /api/events

2023-05-26 18:32:52,800 DEBG 'runserver' stderr output:
[Fri May 26 18:32:52.800489 2023] [wsgi:error] [pid 1164:tid 140096261142080] [remote 192.168.251.158:52798] WARNING:django.request:Unauthorized: /api/events

2023-05-26 18:33:42,838 DEBG 'runserver' stderr output:
[Fri May 26 18:33:42.838814 2023] [wsgi:error] [pid 1164:tid 140096294712896] [remote 192.168.251.158:52110] [2023-05-26 18:33:42,838] WARNING django.request: Unauthorized: /api/events

2023-05-26 18:33:42,840 DEBG 'runserver' stderr output:
[Fri May 26 18:33:42.840722 2023] [wsgi:error] [pid 1164:tid 140096294712896] [remote 192.168.251.158:52110] WARNING:django.request:Unauthorized: /api/events

2023-05-26 18:34:22,800 DEBG 'runserver' stderr output:
[Fri May 26 18:34:22.800785 2023] [wsgi:error] [pid 1164:tid 140096277927488] [remote 192.168.251.158:46566] [2023-05-26 18:34:22,800] WARNING django.request: Unauthorized: /api/events

2023-05-26 18:34:22,802 DEBG 'runserver' stderr output:
[Fri May 26 18:34:22.802621 2023] [wsgi:error] [pid 1164:tid 140096277927488] [remote 192.168.251.158:46566] WARNING:django.request:Unauthorized: /api/events

2023-05-26 18:35:49,116 DEBG 'runserver' stderr output:
[Fri May 26 18:35:49.112634 2023] [wsgi:error] [pid 1164:tid 140096269534784] [remote 192.168.251.158:55496] [2023-05-26 18:35:49,112] WARNING django.request: Unauthorized: /api/events

2023-05-26 18:35:49,125 DEBG 'runserver' stderr output:
[Fri May 26 18:35:49.124806 2023] [wsgi:error] [pid 1164:tid 140096269534784] [remote 192.168.251.158:55496] WARNING:django.request:Unauthorized: /api/events

2023-05-26 18:35:53,017 DEBG 'runserver' stderr output:
[Fri May 26 18:35:53.017364 2023] [wsgi:error] [pid 1164:tid 140096269534784] [remote 192.168.251.158:58484] [2023-05-26 18:35:53,017] WARNING django.request: Unauthorized: /api/events

2023-05-26 18:35:53,022 DEBG 'runserver' stderr output:
[Fri May 26 18:35:53.022368 2023] [wsgi:error] [pid 1164:tid 140096269534784] [remote 192.168.251.158:58484] WARNING:django.request:Unauthorized: /api/events

2023-05-26 18:36:59,839 DEBG 'runserver' stderr output:
[Fri May 26 18:36:59.839811 2023] [wsgi:error] [pid 1164:tid 140096286320192] [remote 192.168.251.158:49144] [2023-05-26 18:36:59,839] WARNING django.request: Unauthorized: /api/events

2023-05-26 18:36:59,840 DEBG 'runserver' stderr output:
[Fri May 26 18:36:59.839910 2023] [wsgi:error] [pid 1164:tid 140096286320192] [remote 192.168.251.158:49144] WARNING:django.request:Unauthorized: /api/events

2023-05-26 18:37:22,801 DEBG 'runserver' stderr output:
[Fri May 26 18:37:22.801042 2023] [wsgi:error] [pid 1164:tid 140096277927488] [remote 192.168.251.158:45464] [2023-05-26 18:37:22,800] WARNING django.request: Unauthorized: /api/events

2023-05-26 18:37:22,801 DEBG 'runserver' stderr output:
[Fri May 26 18:37:22.801919 2023] [wsgi:error] [pid 1164:tid 140096277927488] [remote 192.168.251.158:45464] WARNING:django.request:Unauthorized: /api/events

2023-05-26 18:38:29,841 DEBG 'runserver' stderr output:
[Fri May 26 18:38:29.840507 2023] [wsgi:error] [pid 1164:tid 140096277927488] [remote 192.168.251.158:57048] [2023-05-26 18:38:29,840] WARNING django.request: Unauthorized: /api/events

2023-05-26 18:38:29,842 DEBG 'runserver' stderr output:
[Fri May 26 18:38:29.842033 2023] [wsgi:error] [pid 1164:tid 140096277927488] [remote 192.168.251.158:57048] WARNING:django.request:Unauthorized: /api/events

2023-05-26 18:38:52,801 DEBG 'runserver' stderr output:
[Fri May 26 18:38:52.801036 2023] [wsgi:error] [pid 1164:tid 140096286320192] [remote 192.168.251.158:40356] [2023-05-26 18:38:52,800] WARNING django.request: Unauthorized: /api/events

2023-05-26 18:38:52,801 DEBG 'runserver' stderr output:
[Fri May 26 18:38:52.801146 2023] [wsgi:error] [pid 1164:tid 140096286320192] [remote 192.168.251.158:40356] WARNING:django.request:Unauthorized: /api/events

2023-05-26 18:39:59,840 DEBG 'runserver' stderr output:
[Fri May 26 18:39:59.839951 2023] [wsgi:error] [pid 1164:tid 140096286320192] [remote 192.168.251.158:45170] [2023-05-26 18:39:59,839] WARNING django.request: Unauthorized: /api/events

2023-05-26 18:39:59,842 DEBG 'runserver' stderr output:
[Fri May 26 18:39:59.842491 2023] [wsgi:error] [pid 1164:tid 140096286320192] [remote 192.168.251.158:45170] WARNING:django.request:Unauthorized: /api/events

2023-05-26 18:40:22,801 DEBG 'runserver' stderr output:
[Fri May 26 18:40:22.801096 2023] [wsgi:error] [pid 1164:tid 140096277927488] [remote 192.168.251.158:54500] [2023-05-26 18:40:22,801] WARNING django.request: Unauthorized: /api/events

2023-05-26 18:40:22,802 DEBG 'runserver' stderr output:
[Fri May 26 18:40:22.802407 2023] [wsgi:error] [pid 1164:tid 140096277927488] [remote 192.168.251.158:54500] WARNING:django.request:Unauthorized: /api/events

2023-05-26 18:41:29,839 DEBG 'runserver' stderr output:
[Fri May 26 18:41:29.839216 2023] [wsgi:error] [pid 1164:tid 140096277927488] [remote 192.168.251.158:47696] [2023-05-26 18:41:29,839] WARNING django.request: Unauthorized: /api/events

2023-05-26 18:41:29,841 DEBG 'runserver' stderr output:
[Fri May 26 18:41:29.841389 2023] [wsgi:error] [pid 1164:tid 140096277927488] [remote 192.168.251.158:47696] WARNING:django.request:Unauthorized: /api/events

2023-05-26 18:41:52,801 DEBG 'runserver' stderr output:
[Fri May 26 18:41:52.801148 2023] [wsgi:error] [pid 1164:tid 140096286320192] [remote 192.168.251.158:57128] [2023-05-26 18:41:52,801] WARNING django.request: Unauthorized: /api/events

2023-05-26 18:41:52,802 DEBG 'runserver' stderr output:
[Fri May 26 18:41:52.802091 2023] [wsgi:error] [pid 1164:tid 140096286320192] [remote 192.168.251.158:57128] WARNING:django.request:Unauthorized: /api/events

2023-05-26 18:42:59,837 DEBG 'runserver' stderr output:
[Fri May 26 18:42:59.837380 2023] [wsgi:error] [pid 1164:tid 140096286320192] [remote 192.168.251.158:40368] [2023-05-26 18:42:59,837] WARNING django.request: Unauthorized: /api/events

2023-05-26 18:42:59,839 DEBG 'runserver' stderr output:
[Fri May 26 18:42:59.839210 2023] [wsgi:error] [pid 1164:tid 140096286320192] [remote 192.168.251.158:40368] WARNING:django.request:Unauthorized: /api/events

2023-05-26 18:43:22,800 DEBG 'runserver' stderr output:
[Fri May 26 18:43:22.800607 2023] [wsgi:error] [pid 1164:tid 140096277927488] [remote 192.168.251.158:54246] [2023-05-26 18:43:22,800] WARNING django.request: Unauthorized: /api/events

2023-05-26 18:43:22,801 DEBG 'runserver' stderr output:
[Fri May 26 18:43:22.801906 2023] [wsgi:error] [pid 1164:tid 140096277927488] [remote 192.168.251.158:54246] WARNING:django.request:Unauthorized: /api/events

2023-05-26 18:44:52,801 DEBG 'runserver' stderr output:
[Fri May 26 18:44:52.800717 2023] [wsgi:error] [pid 1164:tid 140096286320192] [remote 192.168.251.158:35104] [2023-05-26 18:44:52,800] WARNING django.request: Unauthorized: /api/events

2023-05-26 18:44:52,803 DEBG 'runserver' stderr output:
[Fri May 26 18:44:52.803067 2023] [wsgi:error] [pid 1164:tid 140096286320192] [remote 192.168.251.158:35104] WARNING:django.request:Unauthorized: /api/events

2023-05-26 18:46:22,803 DEBG 'runserver' stderr output:
[Fri May 26 18:46:22.801908 2023] [wsgi:error] [pid 1164:tid 140096286320192] [remote 192.168.251.158:48514] [2023-05-26 18:46:22,801] WARNING django.request: Unauthorized: /api/events

2023-05-26 18:46:22,804 DEBG 'runserver' stderr output:
[Fri May 26 18:46:22.803849 2023] [wsgi:error] [pid 1164:tid 140096286320192] [remote 192.168.251.158:48514] WARNING:django.request:Unauthorized: /api/events

2023-05-26 18:47:52,802 DEBG 'runserver' stderr output:
[Fri May 26 18:47:52.801545 2023] [wsgi:error] [pid 1164:tid 140096277927488] [remote 192.168.251.158:46980] [2023-05-26 18:47:52,801] WARNING django.request: Unauthorized: /api/events

2023-05-26 18:47:52,803 DEBG 'runserver' stderr output:
[Fri May 26 18:47:52.803562 2023] [wsgi:error] [pid 1164:tid 140096277927488] [remote 192.168.251.158:46980] WARNING:django.request:Unauthorized: /api/events

2023-05-26 18:49:22,810 DEBG 'runserver' stderr output:
[Fri May 26 18:49:22.803981 2023] [wsgi:error] [pid 1164:tid 140096269534784] [remote 192.168.251.158:60084] [2023-05-26 18:49:22,803] WARNING django.request: Unauthorized: /api/events

2023-05-26 18:49:22,811 DEBG 'runserver' stderr output:
[Fri May 26 18:49:22.811722 2023] [wsgi:error] [pid 1164:tid 140096269534784] [remote 192.168.251.158:60084] WARNING:django.request:Unauthorized: /api/events

the ingress seem to be working as well

Name:             cvat
Labels:           app.kubernetes.io/instance=cvat
                  app.kubernetes.io/managed-by=Helm
                  app.kubernetes.io/name=cvat
                  app.kubernetes.io/version=latest
                  helm.sh/chart=cvat
Namespace:        cvat-gsrv
Address:          
Ingress Class:    <none>
Default backend:  <default>
Rules:
  Host           Path  Backends
  ----           ----  --------
  cvat.gsrv.lan  
                 /api         cvat-backend-service:8080 (192.168.251.152:8080)
                 /admin       cvat-backend-service:8080 (192.168.251.152:8080)
                 /static      cvat-backend-service:8080 (192.168.251.152:8080)
                 /django-rq   cvat-backend-service:8080 (192.168.251.152:8080)
                 /git         cvat-backend-service:8080 (192.168.251.152:8080)
                 /opencv      cvat-backend-service:8080 (192.168.251.152:8080)
                 /profiler    cvat-backend-service:8080 (192.168.251.152:8080)
                 /            cvat-frontend-service:80 (192.168.251.148:80)
Annotations:     kubernetes.io/ingress.class: traefik
                 meta.helm.sh/release-name: cvat
                 meta.helm.sh/release-namespace: cvat-gsrv
                 traefik.ingress.kubernetes.io/router.entrypoints: web
Events:          <none>

ganindu7 commented 1 year ago

Here is a backend server log again (this time I was using the latest release branch)

kubectl logs --namespace cvat-gsrv cvat-backend-server-d64969648-5zxg8 Defaulted container "cvat-app-backend-server-container" out of: cvat-app-backend-server-container, user-data-permission-fix (init) 2023-05-27 17:45:11,642 INFO RPC interface 'supervisor' initialized 2023-05-27 17:45:11,642 CRIT Server 'unix_http_server' running without any HTTP authentication checking 2023-05-27 17:45:11,642 INFO supervisord started with pid 1 2023-05-27 17:45:12,648 INFO spawned: 'ssh-agent' with pid 7 2023-05-27 17:45:12,652 INFO spawned: 'clamav_update' with pid 8 2023-05-27 17:45:12,673 INFO spawned: 'runserver' with pid 9 2023-05-27 17:45:12,674 DEBG fd 10 closed, stopped monitoring <POutputDispatcher at 140061064098720 for <Subprocess at 140061064088160 with name clamav_update in state STARTING> (stdout)> 2023-05-27 17:45:12,674 DEBG fd 14 closed, stopped monitoring <POutputDispatcher at 140061064098576 for <Subprocess at 140061064088160 with name clamav_update in state STARTING> (stderr)> 2023-05-27 17:45:12,674 INFO exited: clamav_update (exit status 0; not expected) 2023-05-27 17:45:12,676 DEBG received SIGCHLD indicating a child quit 2023-05-27 17:45:12,677 DEBG 'ssh-agent' stdout output: SSH_AUTH_SOCK=/tmp/ssh-agent.sock; export SSH_AUTH_SOCK; echo Agent pid 7;

2023-05-27 17:45:12,677 DEBG 'ssh-agent' stderr output: debug1: new_socket: type = SOCKET debug2: fd 3 setting O_NONBLOCK

2023-05-27 17:45:12,682 DEBG 'runserver' stderr output: wait-for-it.sh: waiting for cvat-postgresql:5432 without a timeout

2023-05-27 17:45:13,683 INFO success: ssh-agent entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2023-05-27 17:45:13,687 INFO spawned: 'clamav_update' with pid 16 2023-05-27 17:45:13,687 INFO success: runserver entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2023-05-27 17:45:13,692 DEBG fd 10 closed, stopped monitoring <POutputDispatcher at 140061064098672 for <Subprocess at 140061064088160 with name clamav_update in state STARTING> (stdout)> 2023-05-27 17:45:13,692 DEBG fd 16 closed, stopped monitoring <POutputDispatcher at 140061064098768 for <Subprocess at 140061064088160 with name clamav_update in state STARTING> (stderr)> 2023-05-27 17:45:13,693 INFO exited: clamav_update (exit status 0; not expected) 2023-05-27 17:45:13,693 DEBG received SIGCHLD indicating a child quit 2023-05-27 17:45:15,700 INFO spawned: 'clamav_update' with pid 19 2023-05-27 17:45:15,705 DEBG fd 10 closed, stopped monitoring <POutputDispatcher at 140061064098720 for <Subprocess at 140061064088160 with name clamav_update in state STARTING> (stdout)> 2023-05-27 17:45:15,705 DEBG fd 16 closed, stopped monitoring <POutputDispatcher at 140061064088208 for <Subprocess at 140061064088160 with name clamav_update in state STARTING> (stderr)> 2023-05-27 17:45:15,705 INFO exited: clamav_update (exit status 0; not expected) 2023-05-27 17:45:15,705 DEBG received SIGCHLD indicating a child quit 2023-05-27 17:45:18,715 INFO spawned: 'clamav_update' with pid 23 2023-05-27 17:45:18,719 DEBG fd 10 closed, stopped monitoring <POutputDispatcher at 140061064098672 for <Subprocess at 140061064088160 with name clamav_update in state STARTING> (stdout)> 2023-05-27 17:45:18,719 DEBG fd 16 closed, stopped monitoring <POutputDispatcher at 140061064098864 for <Subprocess at 140061064088160 with name clamav_update in state STARTING> (stderr)> 2023-05-27 17:45:18,719 INFO exited: clamav_update (exit status 0; not expected) 2023-05-27 17:45:18,719 DEBG received SIGCHLD indicating a child quit 2023-05-27 17:45:19,722 INFO gave up: clamav_update entered FATAL state, too many start retries too quickly 2023-05-27 17:45:48,985 DEBG 'runserver' stderr output: wait-for-it.sh: cvat-postgresql:5432 is available after 36 seconds

2023-05-27 17:45:48,985 DEBG 'runserver' stderr output: bash: cannot set terminal process group (-1): Inappropriate ioctl for device bash: no job control in this shell

2023-05-27 17:45:49,200 DEBG 'ssh-agent' stderr output: debug1: new_socket: type = CONNECTION debug2: fd 4 setting O_NONBLOCK

2023-05-27 17:45:49,203 DEBG 'ssh-agent' stderr output: debug1: new_socket: type = CONNECTION debug2: fd 4 setting O_NONBLOCK debug1: process_message: socket 1 (fd=4) type 11 debug2: process_request_identities: entering debug2: process_request_identities: replying with 0 allowed of 0 available keys

2023-05-27 17:45:49,221 DEBG 'ssh-agent' stderr output: debug1: new_socket: type = CONNECTION debug2: fd 4 setting O_NONBLOCK

2023-05-27 17:45:49,222 DEBG 'ssh-agent' stderr output: debug1: process_message: socket 1 (fd=4) type 17 debug2: process_add_identity: entering

2023-05-27 17:45:49,224 DEBG 'ssh-agent' stderr output: debug1: process_add_identity: add ssh-rsa SHA256:kx1H/Kp+1xknufZlSVxut3W0wwaC+QBVtwyLgOOLQbs "django@cvat-backend-worker-export-57b7c75c79-679cz" (life: 0) (confirm: 0) (provider: none) (destination constraints: 0)

2023-05-27 17:45:49,224 DEBG 'runserver' stderr output: Identity added: /home/django/.ssh/id_rsa (django@cvat-backend-worker-export-57b7c75c79-679cz)

2023-05-27 17:47:23,876 DEBG 'runserver' stdout output: SSH keys were not found Copying them from keys volume Operations to perform: Apply all migrations: account, admin, auth, authtoken, contenttypes, dataset_repo, db, engine, organizations, sessions, sites, socialaccount, webhooks Running migrations: Applying contenttypes.0001_initial... 2023-05-27 17:47:23,974 DEBG 'runserver' stdout output: OK Applying auth.0001_initial... 2023-05-27 17:47:24,874 DEBG 'runserver' stdout output: OK Applying account.0001_initial... 2023-05-27 17:47:25,219 DEBG 'runserver' stdout output: OK Applying account.0002_email_max_length... 2023-05-27 17:47:25,244 DEBG 'runserver' stdout output: OK Applying admin.0001_initial... 2023-05-27 17:47:25,413 DEBG 'runserver' stdout output: OK Applying admin.0002_logentry_remove_auto_add... 2023-05-27 17:47:25,423 DEBG 'runserver' stdout output: OK Applying admin.0003_logentry_add_action_flag_choices... 2023-05-27 17:47:25,432 DEBG 'runserver' stdout output: OK Applying contenttypes.0002_remove_content_type_name... 2023-05-27 17:47:25,447 DEBG 'runserver' stdout output: OK Applying auth.0002_alter_permission_name_max_length... 2023-05-27 17:47:25,455 DEBG 'runserver' stdout output: OK Applying auth.0003_alter_user_email_max_length... 2023-05-27 17:47:25,466 DEBG 'runserver' stdout output: OK Applying auth.0004_alter_user_username_opts... 2023-05-27 17:47:25,473 DEBG 'runserver' stdout output: OK Applying auth.0005_alter_user_last_login_null... 2023-05-27 17:47:25,482 DEBG 'runserver' stdout output: OK Applying auth.0006_require_contenttypes_0002... 2023-05-27 17:47:25,488 DEBG 'runserver' stdout output: OK Applying auth.0007_alter_validators_add_error_messages... 2023-05-27 17:47:25,498 DEBG 'runserver' stdout output: OK Applying auth.0008_alter_user_username_max_length... 2023-05-27 17:47:25,556 DEBG 'runserver' stdout output: OK Applying auth.0009_alter_user_last_name_max_length... 2023-05-27 17:47:25,563 DEBG 'runserver' stdout output: OK Applying auth.0010_alter_group_name_max_length... 2023-05-27 17:47:25,579 DEBG 'runserver' stdout output: OK Applying auth.0011_update_proxy_permissions... 2023-05-27 17:47:25,585 DEBG 'runserver' stdout output: OK Applying auth.0012_alter_user_first_name_max_length... 2023-05-27 17:47:25,597 DEBG 'runserver' stdout output: OK Applying authtoken.0001_initial... 2023-05-27 17:47:25,743 DEBG 'runserver' stdout output: OK Applying authtoken.0002_auto_20160226_1747... 2023-05-27 17:47:25,764 DEBG 'runserver' stdout output: OK Applying authtoken.0003_tokenproxy... 2023-05-27 17:47:25,772 DEBG 'runserver' stdout output: OK Applying engine.0001_release_v0_1_0... 2023-05-27 17:47:27,543 DEBG 'runserver' stdout output: OK Applying engine.0002_labeledpoints_labeledpointsattributeval_labeledpolygon_labeledpolygonattributeval_labeledpolyline_la... 2023-05-27 17:47:29,562 DEBG 'runserver' stdout output: OK Applying engine.0003_objectpath_shapes... 2023-05-27 17:47:29,583 DEBG 'runserver' stdout output: OK Applying engine.0004_task_z_order... 2023-05-27 17:47:29,598 DEBG 'runserver' stdout output: OK Applying engine.0005_auto_20180609_1512... 2023-05-27 17:47:29,806 DEBG 'runserver' stdout output: OK Applying engine.0006_auto_20180629_1501... 2023-05-27 17:47:29,865 DEBG 'runserver' stdout output: OK Applying engine.0007_task_flipped... 2023-05-27 17:47:29,875 DEBG 'runserver' stdout output: OK Applying engine.0008_auto_20180917_1424... 2023-05-27 17:47:29,891 DEBG 'runserver' stdout output: OK Applying engine.0009_auto_20180917_1424... 2023-05-27 17:47:35,154 DEBG 'runserver' stdout output: OK Applying engine.0010_auto_20181011_1517... 2023-05-27 17:47:35,705 DEBG 'runserver' stdout output: OK Applying engine.0011_add_task_source_and_safecharfield... 2023-05-27 17:47:35,779 DEBG 'runserver' stdout output: OK Applying engine.0012_auto_20181025_1618... 2023-05-27 17:47:35,797 DEBG 'runserver' stdout output: OK Applying engine.0013_auth_no_default_permissions... 2023-05-27 17:47:36,065 DEBG 'runserver' stdout output: OK Applying engine.0014_job_max_shape_id... 2023-05-27 17:47:36,266 DEBG 'runserver' stdout output: OK Applying dataset_repo.0001_initial... 2023-05-27 17:47:36,356 DEBG 'runserver' stdout output: OK Applying dataset_repo.0002_auto_20190123_1305... 2023-05-27 17:47:36,365 DEBG 'runserver' stdout output: OK Applying dataset_repo.0003_gitdata_lfs... 2023-05-27 17:47:36,372 DEBG 'runserver' stdout output: OK Applying dataset_repo.0004_rename... 2023-05-27 17:47:36,408 DEBG 'runserver' stdout output: OK Applying dataset_repo.0005_auto_20201019_1100... 2023-05-27 17:47:36,416 DEBG 'runserver' stdout output: OK Applying dataset_repo.0006_gitdata_format... 2023-05-27 17:47:36,443 DEBG 'runserver' stdout output: OK Applying db.0001_initial... 2023-05-27 17:47:36,490 DEBG 'runserver' stdout output: OK Applying organizations.0001_initial... 2023-05-27 17:47:36,978 DEBG 'runserver' stdout output: OK Applying engine.0015_db_redesign_20190217... 2023-05-27 17:47:38,085 DEBG 'runserver' stdout output: OK Applying engine.0016_attribute_spec_20190217... 2023-05-27 17:47:38,178 DEBG 'runserver' stdout output: OK Applying engine.0017_db_redesign_20190221... 2023-05-27 17:47:41,246 DEBG 'runserver' stdout output: OK Applying engine.0018_jobcommit... 2023-05-27 17:47:41,396 DEBG 'runserver' stdout output: OK Applying engine.0019_frame_selection... 2023-05-27 17:47:41,465 DEBG 'runserver' stdout output: OK Applying engine.0020_remove_task_flipped... 2023-05-27 17:47:41,496 DEBG 'runserver' stdout output: Getting flipped tasks... Conversion started... OK Applying engine.0021_auto_20190826_1827... 2023-05-27 17:47:41,508 DEBG 'runserver' stdout output: OK Applying engine.0022_auto_20191004_0817... 2023-05-27 17:47:41,698 DEBG 'runserver' stdout output: OK Applying engine.0023_auto_20200113_1323... 2023-05-27 17:47:41,843 DEBG 'runserver' stdout output: OK Applying engine.0024_auto_20191023_1025... 2023-05-27 17:47:42,254 DEBG 'runserver' stdout output:

Start schema migration...

2023-05-27 17:47:42,258 DEBG 'runserver' stderr output: INFO:0024_auto_20191023_1025: Start schema migration...

2023-05-27 17:47:42,258 DEBG 'runserver' stdout output:

Schema migration is finished...

Start data migration...

2023-05-27 17:47:42,258 DEBG 'runserver' stderr output: INFO:0024_auto_20191023_1025: Schema migration is finished... INFO:0024_auto_20191023_1025: Start data migration...

2023-05-27 17:47:42,629 DEBG 'runserver' stdout output: OK Applying engine.0025_auto_20200324_1222... 2023-05-27 17:47:42,636 DEBG 'runserver' stdout output: OK Applying engine.0026_auto_20200719_1511... 2023-05-27 17:47:42,649 DEBG 'runserver' stdout output: OK Applying engine.0027_auto_20200719_1552... 2023-05-27 17:47:42,673 DEBG 'runserver' stdout output: OK Applying engine.0028_labelcolor... 2023-05-27 17:47:42,706 DEBG 'runserver' stdout output: OK Applying engine.0029_data_storage_method... 2023-05-27 17:47:42,732 DEBG 'runserver' stdout output: OK Applying engine.0030_auto_20200914_1331... 2023-05-27 17:47:42,854 DEBG 'runserver' stdout output: OK Applying engine.0031_auto_20201011_0220... 2023-05-27 17:47:42,870 DEBG 'runserver' stdout output: OK Applying engine.0032_remove_task_z_order... 2023-05-27 17:47:42,886 DEBG 'runserver' stdout output: OK Applying engine.0033_projects_adjastment... 2023-05-27 17:47:42,964 DEBG 'runserver' stdout output: OK Applying engine.0034_auto_20201125_1426... 2023-05-27 17:47:43,587 DEBG 'runserver' stdout output: OK Applying engine.0035_data_storage... 2023-05-27 17:47:43,595 DEBG 'runserver' stdout output: OK Applying engine.0036_auto_20201216_0943... 2023-05-27 17:47:43,706 DEBG 'runserver' stdout output: OK Applying engine.0037_task_subset... 2023-05-27 17:47:43,723 DEBG 'runserver' stdout output: OK Applying engine.0038_manifest... 2023-05-27 17:47:43,741 DEBG 'runserver' stdout output: The data migration has been started for creating manifest`s files

2023-05-27 17:47:43,741 DEBG 'runserver' stderr output: The data migration has been started for creating manifests files INFO:0038_manifest:The data migration has been started for creating manifests files

2023-05-27 17:47:43,743 DEBG 'runserver' stdout output: Need to update 0 data objects

2023-05-27 17:47:43,743 DEBG 'runserver' stderr output: Need to update 0 data objects INFO:0038_manifest:Need to update 0 data objects

2023-05-27 17:47:43,746 DEBG 'runserver' stdout output: OK Applying engine.0039_auto_training... 2023-05-27 17:47:44,168 DEBG 'runserver' stdout output: OK Applying engine.0040_cloud_storage... 2023-05-27 17:47:44,288 DEBG 'runserver' stdout output: OK Applying engine.0041_auto_20210827_0258... 2023-05-27 17:47:44,312 DEBG 'runserver' stdout output: OK Applying engine.0042_auto_20210830_1056... 2023-05-27 17:47:44,401 DEBG 'runserver' stdout output: OK Applying engine.0043_auto_20211027_0718... 2023-05-27 17:47:44,415 DEBG 'runserver' stdout output: OK Applying engine.0044_auto_20211115_0858... 2023-05-27 17:47:44,431 DEBG 'runserver' stdout output: OK Applying engine.0045_auto_20211123_0824... 2023-05-27 17:47:44,459 DEBG 'runserver' stdout output: OK Applying engine.0046_data_sorting_method... 2023-05-27 17:47:44,469 DEBG 'runserver' stdout output: OK Applying engine.0047_auto_20211110_1938... 2023-05-27 17:47:44,669 DEBG 'runserver' stdout output: OK Applying engine.0048_auto_20211112_1918... 2023-05-27 17:47:44,848 DEBG 'runserver' stdout output: OK Applying engine.0049_auto_20220202_0710... 2023-05-27 17:47:44,863 DEBG 'runserver' stdout output: OK Applying engine.0050_auto_20220211_1425... 2023-05-27 17:47:45,001 DEBG 'runserver' stdout output: OK Applying engine.0051_auto_20220220_1824... 2023-05-27 17:47:45,248 DEBG 'runserver' stdout output: OK Applying engine.0052_alter_cloudstorage_specific_attributes... 2023-05-27 17:47:45,262 DEBG 'runserver' stdout output: OK Applying engine.0053_data_deleted_frames... 2023-05-27 17:47:45,286 DEBG 'runserver' stdout output: OK Applying engine.0054_auto_20220610_1829... 2023-05-27 17:47:45,474 DEBG 'runserver' stdout output: OK Applying engine.0055_jobs_directories... 2023-05-27 17:47:45,495 DEBG 'runserver' stdout output: Migration has been started. Need to create 0 directories.

2023-05-27 17:47:45,495 DEBG 'runserver' stderr output: Migration has been started. Need to create 0 directories. INFO:0055_jobs_directories:Migration has been started. Need to create 0 directories.

2023-05-27 17:47:45,495 DEBG 'runserver' stdout output: Migration has been finished successfully.

2023-05-27 17:47:45,496 DEBG 'runserver' stderr output: Migration has been finished successfully. INFO:0055_jobs_directories:Migration has been finished successfully.

2023-05-27 17:47:45,498 DEBG 'runserver' stdout output: OK Applying engine.0056_jobs_previews... 2023-05-27 17:47:45,517 DEBG 'runserver' stdout output: Migration has been started. Need to create 0 previews.

2023-05-27 17:47:45,517 DEBG 'runserver' stderr output: Migration has been started. Need to create 0 previews. INFO:0056_jobs_previews:Migration has been started. Need to create 0 previews.

2023-05-27 17:47:45,521 DEBG 'runserver' stdout output: OK Applying engine.0057_auto_20220726_0926... 2023-05-27 17:47:46,075 DEBG 'runserver' stdout output: OK Applying engine.0058_auto_20220809_1236... 2023-05-27 17:47:46,632 DEBG 'runserver' stdout output: OK Applying engine.0059_labeledshape_outside... 2023-05-27 17:47:46,647 DEBG 'runserver' stdout output: OK Applying engine.0060_alter_label_parent... 2023-05-27 17:47:46,669 DEBG 'runserver' stdout output: OK Applying engine.0061_auto_20221130_0844... 2023-05-27 17:47:46,685 DEBG 'runserver' stdout output: OK Applying engine.0062_delete_previews... 2023-05-27 17:47:46,704 DEBG 'runserver' stdout output:

Deleting Data previews...

2023-05-27 17:47:46,704 DEBG 'runserver' stderr output: INFO:0062_delete_previews: Deleting Data previews...

2023-05-27 17:47:46,708 DEBG 'runserver' stdout output:

Deleting Job previews...

2023-05-27 17:47:46,708 DEBG 'runserver' stderr output: INFO:0062_delete_previews: Deleting Job previews...

2023-05-27 17:47:46,708 DEBG 'runserver' stdout output:

Deleting CloudStorage previews...

2023-05-27 17:47:46,708 DEBG 'runserver' stderr output: INFO:0062_delete_previews: Deleting CloudStorage previews...

2023-05-27 17:47:46,717 DEBG 'runserver' stdout output: OK Applying engine.0063_delete_jobcommit... 2023-05-27 17:47:46,758 DEBG 'runserver' stdout output: OK Applying engine.0064_delete_or_rename_wrong_labels... 2023-05-27 17:47:46,777 DEBG 'runserver' stdout output:

Deleting skeleton Labels without skeletons...

2023-05-27 17:47:46,777 DEBG 'runserver' stderr output: INFO:0064_delete_or_rename_wrong_labels: Deleting skeleton Labels without skeletons...

2023-05-27 17:47:46,780 DEBG 'runserver' stdout output:

Deleting duplicate skeleton sublabels and renaming duplicate Labels...

2023-05-27 17:47:46,780 DEBG 'runserver' stderr output: INFO:0064_delete_or_rename_wrong_labels: Deleting duplicate skeleton sublabels and renaming duplicate Labels...

2023-05-27 17:47:46,785 DEBG 'runserver' stdout output: OK Applying engine.0065_auto_20230221_0931... 2023-05-27 17:47:46,884 DEBG 'runserver' stdout output: OK Applying engine.0066_auto_20230319_1252... 2023-05-27 17:47:46,918 DEBG 'runserver' stdout output: OK Applying engine.0067_alter_cloudstorage_credentials_type... 2023-05-27 17:47:46,931 DEBG 'runserver' stdout output: OK Applying sessions.0001_initial... 2023-05-27 17:47:46,984 DEBG 'runserver' stdout output: OK Applying sites.0001_initial... 2023-05-27 17:47:47,006 DEBG 'runserver' stdout output: OK Applying sites.0002_alter_domain_unique... 2023-05-27 17:47:47,032 DEBG 'runserver' stdout output: OK Applying socialaccount.0001_initial... 2023-05-27 17:47:47,327 DEBG 'runserver' stdout output: OK Applying socialaccount.0002_token_max_lengths... 2023-05-27 17:47:47,363 DEBG 'runserver' stdout output: OK Applying socialaccount.0003_extra_data_default_dict... 2023-05-27 17:47:47,377 DEBG 'runserver' stdout output: OK Applying webhooks.0001_initial... 2023-05-27 17:47:47,588 DEBG 'runserver' stdout output: OK Applying webhooks.0002_alter_webhookdelivery_status_code... 2023-05-27 17:47:47,663 DEBG 'runserver' stdout output: OK Applying webhooks.0003_alter_webhookdelivery_status_code... 2023-05-27 17:47:47,673 DEBG 'runserver' stdout output: OK Applying webhooks.0004_alter_webhook_target_url... 2023-05-27 17:47:47,878 DEBG 'runserver' stdout output: OK

2023-05-27 17:47:49,114 DEBG 'ssh-agent' stderr output: debug1: new_socket: type = CONNECTION debug2: fd 4 setting O_NONBLOCK

2023-05-27 17:47:49,115 DEBG 'ssh-agent' stderr output: debug1: process_message: socket 1 (fd=4) type 17 debug2: process_add_identity: entering

2023-05-27 17:47:49,116 DEBG 'ssh-agent' stderr output: debug3: identity_permitted: entering: key RSA comment "django@cvat-backend-worker-export-57b7c75c79-679cz", 0 socket bindings, 0 constraints

2023-05-27 17:47:49,116 DEBG 'ssh-agent' stderr output: debug1: process_add_identity: add ssh-rsa SHA256:kx1H/Kp+1xknufZlSVxut3W0wwaC+QBVtwyLgOOLQbs "django@cvat-backend-worker-export-57b7c75c79-679cz" (life: 0) (confirm: 0) (provider: none) (destination constraints: 0)

2023-05-27 17:47:49,119 DEBG 'ssh-agent' stderr output: debug1: new_socket: type = CONNECTION debug2: fd 4 setting O_NONBLOCK debug1: process_message: socket 1 (fd=4) type 11 debug2: process_request_identities: entering debug3: identity_permitted: entering: key RSA comment "django@cvat-backend-worker-export-57b7c75c79-679cz", 0 socket bindings, 0 constraints debug2: process_request_identities: replying with 1 allowed of 1 available keys

2023-05-27 17:47:50,046 DEBG 'runserver' stdout output:

166 static files copied to '/home/django/static'.

2023-05-27 17:47:50,566 DEBG 'ssh-agent' stderr output: debug1: new_socket: type = CONNECTION debug2: fd 4 setting O_NONBLOCK

2023-05-27 17:47:50,567 DEBG 'ssh-agent' stderr output: debug1: process_message: socket 1 (fd=4) type 17 debug2: process_add_identity: entering

2023-05-27 17:47:50,568 DEBG 'ssh-agent' stderr output: debug3: identity_permitted: entering: key RSA comment "django@cvat-backend-worker-export-57b7c75c79-679cz", 0 socket bindings, 0 constraints debug1: process_add_identity: add ssh-rsa SHA256:kx1H/Kp+1xknufZlSVxut3W0wwaC+QBVtwyLgOOLQbs "django@cvat-backend-worker-export-57b7c75c79-679cz" (life: 0) (confirm: 0) (provider: none) (destination constraints: 0)

2023-05-27 17:47:50,572 DEBG 'ssh-agent' stderr output: debug1: new_socket: type = CONNECTION debug2: fd 4 setting O_NONBLOCK debug1: process_message: socket 1 (fd=4) type 11 debug2: process_request_identities: entering debug3: identity_permitted: entering: key RSA comment "django@cvat-backend-worker-export-57b7c75c79-679cz", 0 socket bindings, 0 constraints debug2: process_request_identities: replying with 1 allowed of 1 available keys

2023-05-27 17:48:16,264 DEBG 'runserver' stdout output: Successfully ran command. Server URL : http://localhost:8080/ Server Root : /tmp/cvat-server Server Conf : /tmp/cvat-server/httpd.conf Error Log File : /dev/stderr (INFO) Operating Mode : daemon Request Capacity : 5 (1 process * 5 threads) Request Timeout : 60 (seconds) Startup Timeout : 15 (seconds) Queue Backlog : 100 (connections) Queue Timeout : 45 (seconds) Server Capacity : 20 (event/worker), 20 (prefork) Server Backlog : 500 (connections) Locale Setting : C.UTF-8

2023-05-27 17:48:16,351 DEBG 'runserver' stderr output: [Sat May 27 17:48:16.350446 2023] [mpm_event:notice] [pid 9:tid 139633266231168] AH00489: Apache/2.4.52 (Ubuntu) mod_wsgi/4.9.4 Python/3.10 configured -- resuming normal operations [Sat May 27 17:48:16.350500 2023] [mpm_event:info] [pid 9:tid 139633266231168] AH00490: Server built: 2023-03-01T22:43:55 [Sat May 27 17:48:16.350504 2023] [core:notice] [pid 9:tid 139633266231168] AH00094: Command line: 'apache2 (mod_wsgi-express) -f /tmp/cvat-server/httpd.conf -D MOD_WSGI_KEEP_ALIVE -D MOD_WSGI_MPM_ENABLE_EVENT_MODULE -D MOD_WSGI_MPM_EXISTS_EVENT_MODULE -D MOD_WSGI_MPM_EXISTS_WORKER_MODULE -D MOD_WSGI_MPM_EXISTS_PREFORK_MODULE -D FOREGROUND'

2023-05-27 17:48:16,351 DEBG 'runserver' stderr output: [Sat May 27 17:48:16.350453 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): Starting process 'localhost:8080' with threads=5. [Sat May 27 17:48:16.350980 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): Python home /opt/venv. [Sat May 27 17:48:16.351027 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): Initializing Python.

2023-05-27 17:48:16,371 DEBG 'runserver' stderr output: [Sat May 27 17:48:16.371743 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): Attach interpreter ''.

2023-05-27 17:48:16,375 DEBG 'runserver' stderr output: [Sat May 27 17:48:16.375441 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): Imported 'mod_wsgi'. [Sat May 27 17:48:16.375480 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662, process='localhost:8080', application=''): Loading Python script file '/tmp/cvat-server/handler.wsgi'.

2023-05-27 17:48:16,689 DEBG 'ssh-agent' stderr output: debug1: new_socket: type = CONNECTION debug2: fd 4 setting O_NONBLOCK

2023-05-27 17:48:16,690 DEBG 'ssh-agent' stderr output: debug1: process_message: socket 1 (fd=4) type 17 debug2: process_add_identity: entering

2023-05-27 17:48:16,691 DEBG 'ssh-agent' stderr output: debug3: identity_permitted: entering: key RSA comment "django@cvat-backend-worker-export-57b7c75c79-679cz", 0 socket bindings, 0 constraints debug1: process_add_identity: add ssh-rsa SHA256:kx1H/Kp+1xknufZlSVxut3W0wwaC+QBVtwyLgOOLQbs "django@cvat-backend-worker-export-57b7c75c79-679cz" (life: 0) (confirm: 0) (provider: none) (destination constraints: 0)

2023-05-27 17:48:16,694 DEBG 'ssh-agent' stderr output: debug1: new_socket: type = CONNECTION debug2: fd 4 setting O_NONBLOCK debug1: process_message: socket 1 (fd=4) type 11 debug2: process_request_identities: entering debug3: identity_permitted: entering: key RSA comment "django@cvat-backend-worker-export-57b7c75c79-679cz", 0 socket bindings, 0 constraints debug2: process_request_identities: replying with 1 allowed of 1 available keys

2023-05-27 17:48:17,700 DEBG 'runserver' stderr output: [Sat May 27 17:48:17.700799 2023] [wsgi:info] [pid 662:tid 139630162835008] mod_wsgi (pid=662): Application startup timer triggered 'localhost:8080'.

2023-05-27 17:48:17,701 DEBG 'runserver' stderr output: [Sat May 27 17:48:17.700943 2023] [wsgi:info] [pid 662:tid 139630162835008] mod_wsgi (pid=662): Application startup timer cancelled 'localhost:8080'.

2023-05-27 17:48:27,043 DEBG 'runserver' stderr output: [Sat May 27 17:48:27.043573 2023] [wsgi:error] [pid 662:tid 139630154442304] [remote 192.168.251.158:35490] [2023-05-27 17:48:27,043] WARNING django.request: Unauthorized: /api/events

2023-05-27 17:48:27,043 DEBG 'runserver' stderr output: [Sat May 27 17:48:27.043677 2023] [wsgi:error] [pid 662:tid 139630154442304] [remote 192.168.251.158:35490] WARNING:django.request:Unauthorized: /api/events

2023-05-27 17:49:45,445 DEBG 'runserver' stderr output: [Sat May 27 17:49:45.443724 2023] [mpm_event:error] [pid 9:tid 139633266231168] AH10159: server is within MinSpareThreads of MaxRequestWorkers, consider raising the MaxRequestWorkers setting

2023-05-27 17:49:47,571 DEBG 'runserver' stderr output: [Sat May 27 17:49:47.571790 2023] [wsgi:error] [pid 744:tid 139633258665536] [client 192.168.251.158:47226] Timeout when reading response headers from daemon process 'localhost:8080': /tmp/cvat-server/htdocs/api, referer: http://cvat.gsrv.lan:32296/

2023-05-27 17:49:55,627 DEBG 'runserver' stderr output: [Sat May 27 17:49:55.626886 2023] [wsgi:error] [pid 744:tid 139633253938752] [client 192.168.251.158:36670] Timeout when reading response headers from daemon process 'localhost:8080': /tmp/cvat-server/htdocs/api, referer: http://cvat.gsrv.lan:32296/

2023-05-27 17:50:03,707 DEBG 'runserver' stderr output: [Sat May 27 17:50:03.706884 2023] [wsgi:error] [pid 663:tid 139633253406272] [client 192.168.251.158:60206] Timeout when reading response headers from daemon process 'localhost:8080': /tmp/cvat-server/htdocs/api, referer: http://cvat.gsrv.lan:32296/

2023-05-27 17:50:03,804 DEBG 'runserver' stderr output: [Sat May 27 17:50:03.804297 2023] [wsgi:info] [pid 662:tid 139630179620416] mod_wsgi (pid=662): Daemon process request time limit exceeded, stopping process 'localhost:8080'. [Sat May 27 17:50:03.804366 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): Shutdown requested 'localhost:8080'.

2023-05-27 17:50:08,804 DEBG 'runserver' stderr output: [Sat May 27 17:50:08.804525 2023] [wsgi:info] [pid 662:tid 139625561585216] mod_wsgi (pid=662): Aborting process 'localhost:8080'. [Sat May 27 17:50:08.804548 2023] [wsgi:info] [pid 662:tid 139625561585216] mod_wsgi (pid=662): Exiting process 'localhost:8080'.

2023-05-27 17:50:11,772 DEBG 'runserver' stderr output: [Sat May 27 17:50:11.772011 2023] [wsgi:error] [pid 663:tid 139633252607552] [client 192.168.251.158:38672] Timeout when reading response headers from daemon process 'localhost:8080': /tmp/cvat-server/htdocs/api, referer: http://cvat.gsrv.lan:32296/

2023-05-27 17:50:19,843 DEBG 'runserver' stderr output: [Sat May 27 17:50:19.843528 2023] [wsgi:error] [pid 744:tid 139633252873792] [client 192.168.251.158:38686] Timeout when reading response headers from daemon process 'localhost:8080': /tmp/cvat-server/htdocs/api, referer: http://cvat.gsrv.lan:32296/

2023-05-27 17:50:24,371 DEBG 'runserver' stderr output: [Sat May 27 17:50:24.371529 2023] [wsgi:error] [pid 663:tid 139633258665536] [client 192.168.251.152:32782] Timeout when reading response headers from daemon process 'localhost:8080': /tmp/cvat-server/htdocs/api

2023-05-27 17:50:27,971 DEBG 'runserver' stderr output: [Sat May 27 17:50:27.970867 2023] [wsgi:error] [pid 744:tid 139633252607552] [client 192.168.251.158:42802] Timeout when reading response headers from daemon process 'localhost:8080': /tmp/cvat-server/htdocs/api, referer: http://cvat.gsrv.lan:32296/

2023-05-27 17:50:32,264 DEBG 'runserver' stderr output: [Sat May 27 17:50:32.264410 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): Dumping stack trace for active Python threads. [Sat May 27 17:50:32.264429 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): Thread 139624345232960 executing file "/usr/lib/python3.10/concurrent/futures/thread.py", line 81, in _worker [Sat May 27 17:50:32.264433 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/threading.py", line 953, in run, [Sat May 27 17:50:32.264435 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner, [Sat May 27 17:50:32.264437 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/threading.py", line 973, in _bootstrap. [Sat May 27 17:50:32.264438 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): Thread 139624353625664 executing file "/usr/lib/python3.10/concurrent/futures/thread.py", line 81, in _worker [Sat May 27 17:50:32.264440 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/threading.py", line 953, in run, [Sat May 27 17:50:32.264441 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner, [Sat May 27 17:50:32.264443 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/threading.py", line 973, in _bootstrap. [Sat May 27 17:50:32.264444 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): Thread 139624907249216 executing file "/usr/lib/python3.10/concurrent/futures/thread.py", line 81, in _worker [Sat May 27 17:50:32.264446 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/threading.py", line 953, in run, [Sat May 27 17:50:32.264447 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner, [Sat May 27 17:50:32.264449 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/threading.py", line 973, in _bootstrap. [Sat May 27 17:50:32.264451 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): Thread 139624915641920 executing file "/opt/venv/lib/python3.10/site-packages/diskcache/core.py", line 666, in _execute_with_retry [Sat May 27 17:50:32.264454 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/diskcache/core.py", line 499, in init, [Sat May 27 17:50:32.264456 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/diskcache/fanout.py", line 43, in , [Sat May 27 17:50:32.264458 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/diskcache/fanout.py", line 42, in init, [Sat May 27 17:50:32.264460 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/diskcache/djangocache.py", line 31, in init, [Sat May 27 17:50:32.264463 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/cache/init.py", line 44, in create_connection, [Sat May 27 17:50:32.264465 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/utils/connection.py", line 62, in getitem, [Sat May 27 17:50:32.264467 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/home/django/cvat/apps/health/backends.py", line 42, in check_status, [Sat May 27 17:50:32.264469 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/health_check/backends.py", line 30, in run_check, [Sat May 27 17:50:32.264473 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/health_check/mixins.py", line 35, in _run, [Sat May 27 17:50:32.264475 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run, [Sat May 27 17:50:32.264476 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/concurrent/futures/thread.py", line 83, in _worker, [Sat May 27 17:50:32.264478 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/threading.py", line 953, in run, [Sat May 27 17:50:32.264479 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner, [Sat May 27 17:50:32.264480 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/threading.py", line 973, in _bootstrap. [Sat May 27 17:50:32.264482 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): Thread 139630146049600 executing file "/usr/lib/python3.10/threading.py", line 320, in wait [Sat May 27 17:50:32.264483 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/concurrent/futures/_base.py", line 453, in result, [Sat May 27 17:50:32.264485 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/concurrent/futures/_base.py", line 319, in _result_or_cancel, [Sat May 27 17:50:32.264487 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/concurrent/futures/_base.py", line 621, in result_iterator, [Sat May 27 17:50:32.264489 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/health_check/mixins.py", line 44, in run_check, [Sat May 27 17:50:32.264490 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/health_check/mixins.py", line 16, in errors, [Sat May 27 17:50:32.264492 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/health_check/views.py", line 91, in get, [Sat May 27 17:50:32.264493 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/views/decorators/cache.py", line 44, in _wrapped_view_func, [Sat May 27 17:50:32.264496 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/utils/decorators.py", line 43, in _wrapper, [Sat May 27 17:50:32.264497 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/views/generic/base.py", line 98, in dispatch, [Sat May 27 17:50:32.264499 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/views/generic/base.py", line 70, in view, [Sat May 27 17:50:32.264501 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/base.py", line 181, in _get_response, [Sat May 27 17:50:32.264503 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner, [Sat May 27 17:50:32.264504 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/home/django/cvat/apps/iam/views.py", line 91, in call, [Sat May 27 17:50:32.264506 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner, [Sat May 27 17:50:32.264509 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 117, in call,

2023-05-27 17:50:32,264 DEBG 'runserver' stderr output: [Sat May 27 17:50:32.264510 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner, [Sat May 27 17:50:32.264521 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 117, in call, [Sat May 27 17:50:32.264523 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner, [Sat May 27 17:50:32.264524 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 117, in call, [Sat May 27 17:50:32.264526 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner, [Sat May 27 17:50:32.264527 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/crum/init.py", line 97, in call, [Sat May 27 17:50:32.264529 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner, [Sat May 27 17:50:32.264531 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/home/django/cvat/apps/engine/middleware.py", line 17, in call, [Sat May 27 17:50:32.264532 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner, [Sat May 27 17:50:32.264533 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 117, in call, [Sat May 27 17:50:32.264535 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner, [Sat May 27 17:50:32.264536 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 117, in call, [Sat May 27 17:50:32.264537 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner, [Sat May 27 17:50:32.264539 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 117, in call, [Sat May 27 17:50:32.264540 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner, [Sat May 27 17:50:32.264541 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 117, in call, [Sat May 27 17:50:32.264543 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner, [Sat May 27 17:50:32.264544 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 117, in call, [Sat May 27 17:50:32.264545 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner, [Sat May 27 17:50:32.264547 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 117, in call, [Sat May 27 17:50:32.264549 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner, [Sat May 27 17:50:32.264551 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/base.py", line 130, in get_response, [Sat May 27 17:50:32.264552 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/wsgi.py", line 133, in call, [Sat May 27 17:50:32.264554 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/mod_wsgi/server/init.py", line 1568, in handle_request. [Sat May 27 17:50:32.264556 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): Thread 139624924034624 executing file "/usr/lib/python3.10/concurrent/futures/thread.py", line 81, in _worker [Sat May 27 17:50:32.264557 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/threading.py", line 953, in run, [Sat May 27 17:50:32.264559 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner, [Sat May 27 17:50:32.264560 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/threading.py", line 973, in _bootstrap. [Sat May 27 17:50:32.264561 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): Thread 139624932427328 executing file "/usr/lib/python3.10/concurrent/futures/thread.py", line 81, in _worker [Sat May 27 17:50:32.264562 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/threading.py", line 953, in run, [Sat May 27 17:50:32.264564 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner, [Sat May 27 17:50:32.264565 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/threading.py", line 973, in _bootstrap. [Sat May 27 17:50:32.264566 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): Thread 139624940820032 executing file "/usr/lib/python3.10/concurrent/futures/thread.py", line 81, in _worker [Sat May 27 17:50:32.264567 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/threading.py", line 953, in run, [Sat May 27 17:50:32.264569 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner, [Sat May 27 17:50:32.264570 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/threading.py", line 973, in _bootstrap. [Sat May 27 17:50:32.264571 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): Thread 139624949212736 executing file "/usr/lib/python3.10/concurrent/futures/thread.py", line 81, in _worker [Sat May 27 17:50:32.264572 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/threading.py", line 953, in run, [Sat May 27 17:50:32.264574 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner, [Sat May 27 17:50:32.264575 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/threading.py", line 973, in _bootstrap. [Sat May 27 17:50:32.264576 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): Thread 139624957605440 executing file "/opt/venv/lib/python3.10/site-packages/diskcache/core.py", line 666, in _execute_with_retry [Sat May 27 17:50:32.264578 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/diskcache/core.py", line 499, in init, [Sat May 27 17:50:32.264580 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/diskcache/fanout.py", line 43, in , [Sat May 27 17:50:32.264581 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/diskcache/fanout.py", line 42, in init, [Sat May 27 17:50:32.264582 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/diskcache/djangocache.py", line 31, in init, [Sat May 27 17:50:32.264584 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/cache/init.py", line 44, in create_connection, [Sat May 27 17:50:32.264585 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/utils/connection.py", line 62, in getitem, [Sat May 27 17:50:32.264586 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/home/django/cvat/apps/health/backends.py", line 42, in check_status, [Sat May 27 17:50:32.264588 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/health_check/backends.py", line 30, in run_check, [Sat May 27 17:50:32.264589 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/health_check/mixins.py", line 35, in _run, [Sat May 27 17:50:32.264590 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run, [Sat May 27 17:50:32.264592 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/concurrent/futures/thread.py", line 83, in _worker, [Sat May 27 17:50:32.264593 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/threading.py", line 953, in run, [Sat May 27 17:50:32.264594 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner, [Sat May 27 17:50:32.264595 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/threading.py", line 973, in _bootstrap. [Sat May 27 17:50:32.264596 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): Thread 139625511228992 executing file "/usr/lib/python3.10/concurrent/futures/thread.py", line 81, in _worker [Sat May 27 17:50:32.264598 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/threading.py", line 953, in run, [Sat May 27 17:50:32.264599 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner, [Sat May 27 17:50:32.264600 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/threading.py", line 973, in _bootstrap. [Sat May 27 17:50:32.264601 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): Thread 139625519621696 executing file "/usr/lib/python3.10/concurrent/futures/thread.py", line 81, in _worker [Sat May 27 17:50:32.264603 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/threading.py", line 953, in run, [Sat May 27 17:50:32.264604 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner, [Sat May 27 17:50:32.264606 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/threading.py", line 973, in _bootstrap. [Sat May 27 17:50:32.264607 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): Thread 139625528014400 executing file "/usr/lib/python3.10/concurrent/futures/thread.py", line 81, in _worker [Sat May 27 17:50:32.264608 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/threading.py", line 953, in run, [Sat May 27 17:50:32.264610 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner, [Sat May 27 17:50:32.264611 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/threading.py", line 973, in _bootstrap. [Sat May 27 17:50:32.264612 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): Thread 139625536407104 executing file "/opt/venv/lib/python3.10/site-packages/diskcache/core.py", line 721, in _transact [Sat May 27 17:50:32.264614 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/contextlib.py", line 135, in enter, [Sat May 27 17:50:32.264616 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/diskcache/core.py", line 796, in set, [Sat May 27 17:50:32.264618 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/diskcache/fanout.py", line 123, in set, [Sat May 27 17:50:32.264619 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/diskcache/djangocache.py", line 165, in set, [Sat May 27 17:50:32.264621 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/home/django/cvat/apps/health/backends.py", line 44, in check_status, [Sat May 27 17:50:32.264622 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/health_check/backends.py", line 30, in run_check, [Sat May 27 17:50:32.264623 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/health_check/mixins.py", line 35, in _run, [Sat May 27 17:50:32.264624 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run, [Sat May 27 17:50:32.264626 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/concurrent/futures/thread.py", line 83, in _worker, [Sat May 27 17:50:32.264627 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/threading.py", line 953, in run, [Sat May 27 17:50:32.264628 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner, [Sat May 27 17:50:32.264629 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/threading.py", line 973, in _bootstrap. [Sat May 27 17:50:32.264630 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): Thread 139630137656896 executing file "/usr/lib/python3.10/threading.py", line 320, in wait [Sat May 27 17:50:32.264632 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/concurrent/futures/_base.py", line 453, in result, [Sat May 27 17:50:32.264633 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/concurrent/futures/_base.py", line 319, in _result_or_cancel, [Sat May 27 17:50:32.264635 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/concurrent/futures/_base.py", line 621, in result_iterator, [Sat May 27 17:50:32.264636 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/health_check/mixins.py", line 44, in run_check, [Sat May 27 17:50:32.264638 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/health_check/mixins.py", line 16, in errors, [Sat May 27 17:50:32.264639 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/health_check/views.py", line 91, in get, [Sat May 27 17:50:32.264640 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/views/decorators/cache.py", line 44, in _wrapped_view_func, [Sat May 27 17:50:32.264641 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/utils/decorators.py", line 43, in _wrapper, [Sat May 27 17:50:32.264643 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/views/generic/base.py", line 98, in dispatch, [Sat May 27 17:50:32.264644 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/views/generic/base.py", line 70, in view, [Sat May 27 17:50:32.264645 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/base.py", line 181, in _get_response, [Sat May 27 17:50:32.264647 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner, [Sat May 27 17:50:32.264648 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/home/django/cvat/apps/iam/views.py", line 91, in call, [Sat May 27 17:50:32.264649 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner, [Sat May 27 17:50:32.264651 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 117, in call, [Sat May 27 17:50:32.264652 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner, [Sat May 27 17:50:32.264653 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 117, in call, [Sat May 27 17:50:32.264655 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner, [Sat May 27 17:50:32.264656 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 117, in call, [Sat May 27 17:50:32.264657 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner, [Sat May 27 17:50:32.264658 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/crum/init.py", line 97, in call, [Sat May 27 17:50:32.264660 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner, [Sat May 27 17:50:32.264662 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/home/django/cvat/apps/engine/middleware.py", line 17, in call, [Sat May 27 17:50:32.264663 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner, [Sat May 27 17:50:32.264664 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 117, in call, [Sat May 27 17:50:32.264666 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner, [Sat May 27 17:50:32.264667 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 117, in call, [Sat May 27 17:50:32.264668 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner, [Sat May 27 17:50:32.264670 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 117, in call, [Sat May 27 17:50:32.264671 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner, [Sat May 27 17:50:32.264672 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 117, in call, [Sat May 27 17:50:32.264674 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner, [Sat May 27 17:50:32.264675 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 117, in call, [Sat May 27 17:50:32.264676 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner, [Sat May 27 17:50:32.264678 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 117, in call, [Sat May 27 17:50:32.264679 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner, [Sat May 27 17:50:32.264680 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/base.py", line 130, in get_response, [Sat May 27 17:50:32.264682 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/wsgi.py", line 133, in call, [Sat May 27 17:50:32.264683 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/mod_wsgi/server/init.py", line 1568, in handle_request. [Sat May 27 17:50:32.264684 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): Thread 139630129264192 executing file "/usr/lib/python3.10/threading.py", line 320, in wait [Sat May 27 17:50:32.264686 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/concurrent/futures/_base.py", line 453, in result, [Sat May 27 17:50:32.264687 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/concurrent/futures/_base.py", line 319, in _result_or_cancel, [Sat May 27 17:50:32.264689 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/concurrent/futures/_base.py", line 621, in result_iterator, [Sat May 27 17:50:32.264690 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/health_check/mixins.py", line 44, in run_check, [Sat May 27 17:50:32.264691 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/health_check/mixins.py", line 16, in errors, [Sat May 27 17:50:32.264693 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/health_check/views.py", line 91, in get, [Sat May 27 17:50:32.264694 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/views/decorators/cache.py", line 44, in _wrapped_view_func, [Sat May 27 17:50:32.264695 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/utils/decorators.py", line 43, in _wrapper, [Sat May 27 17:50:32.264697 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/views/generic/base.py", line 98, in dispatch, [Sat May 27 17:50:32.264698 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/views/generic/base.py", line 70, in view, [Sat May 27 17:50:32.264699 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/base.py", line 181, in _get_response, [Sat May 27 17:50:32.264701 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner, [Sat May 27 17:50:32.264702 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/home/django/cvat/apps/iam/views.py", line 91, in call, [Sat May 27 17:50:32.264703 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner, [Sat May 27 17:50:32.264704 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 117, in call, [Sat May 27 17:50:32.264706 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner, [Sat May 27 17:50:32.264707 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 117, in call, [Sat May 27 17:50:32.264708 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner, [Sat May 27 17:50:32.264710 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 117, in call, [Sat May 27 17:50:32.264711 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner, [Sat May 27 17:50:32.264712 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/crum/init.py", line 97, in call, [Sat May 27 17:50:32.264713 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner, [Sat May 27 17:50:32.264715 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/home/django/cvat/apps/engine/middleware.py", line 17, in call, [Sat May 27 17:50:32.264717 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner, [Sat May 27 17:50:32.264718 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 117, in call, [Sat May 27 17:50:32.264719 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner, [Sat May 27 17:50:32.264720 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 117, in call, [Sat May 27 17:50:32.264722 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner, [Sat May 27 17:50:32.264723 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 117, in call, [Sat May 27 17:50:32.264724 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner, [Sat May 27 17:50:32.264726 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 117, in call, [Sat May 27 17:50:32.264727 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner, [Sat May 27 17:50:32.264728 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 117, in call, [Sat May 27 17:50:32.264730 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner, [Sat May 27 17:50:32.264731 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 117, in call, [Sat May 27 17:50:32.264732 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner, [Sat May 27 17:50:32.264734 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/base.py", line 130, in get_response, [Sat May 27 17:50:32.264735 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/wsgi.py", line 133, in call, [Sat May 27 17:50:32.264736 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/mod_wsgi/server/init.py", line 1568, in handle_request. [Sat May 27 17:50:32.264737 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): Thread 139626062329408 executing file "/usr/lib/python3.10/concurrent/futures/thread.py", line 81, in _worker [Sat May 27 17:50:32.264739 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/threading.py", line 953, in run, [Sat May 27 17:50:32.264740 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner, [Sat May 27 17:50:32.264743 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/threading.py", line 973, in _bootstrap. [Sat May 27 17:50:32.264744 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): Thread 139626070722112 executing file "/usr/lib/python3.10/concurrent/futures/thread.py", line 81, in _worker [Sat May 27 17:50:32.264745 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/threading.py", line 953, in run, [Sat May 27 17:50:32.264747 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner, [Sat May 27 17:50:32.264748 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/threading.py", line 973, in _bootstrap. [Sat May 27 17:50:32.264749 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): Thread 139626079114816 executing file "/usr/lib/python3.10/concurrent/futures/thread.py", line 81, in _worker [Sat May 27 17:50:32.264751 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/threading.py", line 953, in run, [Sat May 27 17:50:32.264752 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner, [Sat May 27 17:50:32.264753 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/threading.py", line 973, in _bootstrap. [Sat May 27 17:50:32.264755 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): Thread 139626087507520 executing file "/opt/venv/lib/python3.10/site-packages/diskcache/core.py", line 744, in _transact [Sat May 27 17:50:32.264757 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/contextlib.py", line 142, in exit, [Sat May 27 17:50:32.264758 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/diskcache/core.py", line 796, in set, [Sat May 27 17:50:32.264759 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/diskcache/fanout.py", line 123, in set, [Sat May 27 17:50:32.264761 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/diskcache/djangocache.py", line 165, in set, [Sat May 27 17:50:32.264762 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/home/django/cvat/apps/health/backends.py", line 44, in check_status, [Sat May 27 17:50:32.264763 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/health_check/backends.py", line 30, in run_check, [Sat May 27 17:50:32.264765 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/health_check/mixins.py", line 35, in _run, [Sat May 27 17:50:32.264766 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run, [Sat May 27 17:50:32.264767 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/concurrent/futures/thread.py", line 83, in _worker, [Sat May 27 17:50:32.264768 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/threading.py", line 953, in run, [Sat May 27 17:50:32.264770 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner, [Sat May 27 17:50:32.264771 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/threading.py", line 973, in _bootstrap. [Sat May 27 17:50:32.264773 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): Thread 139630154442304 executing file "/usr/lib/python3.10/threading.py", line 320, in wait [Sat May 27 17:50:32.264774 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/concurrent/futures/_base.py", line 453, in result, [Sat May 27 17:50:32.264776 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/concurrent/futures/_base.py", line 319, in _result_or_cancel, [Sat May 27 17:50:32.264777 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/concurrent/futures/_base.py", line 621, in result_iterator, [Sat May 27 17:50:32.264778 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/health_check/mixins.py", line 44, in run_check, [Sat May 27 17:50:32.264780 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/health_check/mixins.py", line 16, in errors, [Sat May 27 17:50:32.264781 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/health_check/views.py", line 91, in get, [Sat May 27 17:50:32.264782 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/views/decorators/cache.py", line 44, in _wrapped_view_func, [Sat May 27 17:50:32.264784 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/utils/decorators.py", line 43, in _wrapper, [Sat May 27 17:50:32.264785 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/views/generic/base.py", line 98, in dispatch, [Sat May 27 17:50:32.264786 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/views/generic/base.py", line 70, in view, [Sat May 27 17:50:32.264788 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/base.py", line 181, in _get_response, [Sat May 27 17:50:32.264789 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner, [Sat May 27 17:50:32.264790 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/home/django/cvat/apps/iam/views.py", line 91, in call, [Sat May 27 17:50:32.264792 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner, [Sat May 27 17:50:32.264793 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 117, in call, [Sat May 27 17:50:32.264794 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner, [Sat May 27 17:50:32.264795 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 117, in call, [Sat May 27 17:50:32.264797 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner, [Sat May 27 17:50:32.264798 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 117, in call, [Sat May 27 17:50:32.264800 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner, [Sat May 27 17:50:32.264802 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/crum/init.py", line 97, in call, [Sat May 27 17:50:32.264803 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner, [Sat May 27 17:50:32.264804 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/home/django/cvat/apps/engine/middleware.py", line 17, in call, [Sat May 27 17:50:32.264805 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner, [Sat May 27 17:50:32.264807 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 117, in call, [Sat May 27 17:50:32.264808 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner, [Sat May 27 17:50:32.264809 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 117, in call, [Sat May 27 17:50:32.264811 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner, [Sat May 27 17:50:32.264812 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 117, in call, [Sat May 27 17:50:32.264813 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner, [Sat May 27 17:50:32.264814 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 117, in call, [Sat May 27 17:50:32.264816 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner, [Sat May 27 17:50:32.264817 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 117, in call, [Sat May 27 17:50:32.264818 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner, [Sat May 27 17:50:32.264820 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 117, in call, [Sat May 27 17:50:32.264821 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner,

2023-05-27 17:50:32,265 DEBG 'runserver' stderr output: [Sat May 27 17:50:32.264822 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/base.py", line 130, in get_response, [Sat May 27 17:50:32.264833 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/wsgi.py", line 133, in call, [Sat May 27 17:50:32.264834 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/mod_wsgi/server/init.py", line 1568, in handle_request. [Sat May 27 17:50:32.264836 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): Thread 139630162835008 executing file "/usr/lib/python3.10/threading.py", line 1116, in _wait_for_tstate_lock [Sat May 27 17:50:32.264838 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/threading.py", line 1096, in join, [Sat May 27 17:50:32.264839 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/concurrent/futures/thread.py", line 235, in shutdown, [Sat May 27 17:50:32.264841 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/usr/lib/python3.10/concurrent/futures/_base.py", line 649, in exit, [Sat May 27 17:50:32.264842 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/health_check/mixins.py", line 43, in run_check, [Sat May 27 17:50:32.264844 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/health_check/mixins.py", line 16, in errors, [Sat May 27 17:50:32.264845 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/health_check/views.py", line 91, in get, [Sat May 27 17:50:32.264847 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/views/decorators/cache.py", line 44, in _wrapped_view_func, [Sat May 27 17:50:32.264848 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/utils/decorators.py", line 43, in _wrapper, [Sat May 27 17:50:32.264849 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/views/generic/base.py", line 98, in dispatch, [Sat May 27 17:50:32.264851 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/views/generic/base.py", line 70, in view, [Sat May 27 17:50:32.264852 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/base.py", line 181, in _get_response, [Sat May 27 17:50:32.264853 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner, [Sat May 27 17:50:32.264854 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/home/django/cvat/apps/iam/views.py", line 91, in call, [Sat May 27 17:50:32.264856 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner, [Sat May 27 17:50:32.264857 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 117, in call, [Sat May 27 17:50:32.264858 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner, [Sat May 27 17:50:32.264860 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 117, in call, [Sat May 27 17:50:32.264862 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner, [Sat May 27 17:50:32.264863 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 117, in call, [Sat May 27 17:50:32.264865 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner, [Sat May 27 17:50:32.264866 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/crum/init.py", line 97, in call, [Sat May 27 17:50:32.264868 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner, [Sat May 27 17:50:32.264869 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/home/django/cvat/apps/engine/middleware.py", line 17, in call, [Sat May 27 17:50:32.264870 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner, [Sat May 27 17:50:32.264871 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 117, in call, [Sat May 27 17:50:32.264873 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner, [Sat May 27 17:50:32.264874 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 117, in call, [Sat May 27 17:50:32.264875 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner, [Sat May 27 17:50:32.264877 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 117, in call, [Sat May 27 17:50:32.264878 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner, [Sat May 27 17:50:32.264879 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 117, in call, [Sat May 27 17:50:32.264881 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner, [Sat May 27 17:50:32.264882 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 117, in call, [Sat May 27 17:50:32.264883 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner, [Sat May 27 17:50:32.264884 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 117, in call, [Sat May 27 17:50:32.264886 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner, [Sat May 27 17:50:32.264887 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/base.py", line 130, in get_response, [Sat May 27 17:50:32.264889 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/django/core/handlers/wsgi.py", line 133, in call, [Sat May 27 17:50:32.264890 2023] [wsgi:info] [pid 662:tid 139633266231168] mod_wsgi (pid=662): called from file "/opt/venv/lib/python3.10/site-packages/mod_wsgi/server/init.py", line 1568, in handle_request.

2023-05-27 17:50:32,493 DEBG 'runserver' stderr output: [Sat May 27 17:50:32.493121 2023] [wsgi:info] [pid 9:tid 139633266231168] mod_wsgi (pid=662): Process 'localhost:8080' has died, deregister and restart it. [Sat May 27 17:50:32.493148 2023] [wsgi:info] [pid 9:tid 139633266231168] mod_wsgi (pid=662): Process 'localhost:8080' has been deregistered and will no longer be monitored.

2023-05-27 17:50:32,493 DEBG 'runserver' stderr output: [Sat May 27 17:50:32.493678 2023] [wsgi:info] [pid 985:tid 139633266231168] mod_wsgi (pid=985): Starting process 'localhost:8080' with threads=5.

2023-05-27 17:50:32,494 DEBG 'runserver' stderr output: [Sat May 27 17:50:32.494175 2023] [wsgi:info] [pid 985:tid 139633266231168] mod_wsgi (pid=985): Python home /opt/venv.

2023-05-27 17:50:32,494 DEBG 'runserver' stderr output: [Sat May 27 17:50:32.494230 2023] [wsgi:info] [pid 985:tid 139633266231168] mod_wsgi (pid=985): Initializing Python.

2023-05-27 17:50:32,515 DEBG 'runserver' stderr output: [Sat May 27 17:50:32.515504 2023] [wsgi:info] [pid 985:tid 139633266231168] mod_wsgi (pid=985): Attach interpreter ''.

2023-05-27 17:50:32,519 DEBG 'runserver' stderr output: [Sat May 27 17:50:32.519334 2023] [wsgi:info] [pid 985:tid 139633266231168] mod_wsgi (pid=985): Imported 'mod_wsgi'. [Sat May 27 17:50:32.519374 2023] [wsgi:info] [pid 985:tid 139633266231168] mod_wsgi (pid=985, process='localhost:8080', application=''): Loading Python script file '/tmp/cvat-server/handler.wsgi'.

2023-05-27 17:50:32,833 DEBG 'ssh-agent' stderr output: debug1: new_socket: type = CONNECTION debug2: fd 4 setting O_NONBLOCK

2023-05-27 17:50:32,834 DEBG 'ssh-agent' stderr output: debug1: process_message: socket 1 (fd=4) type 17 debug2: process_add_identity: entering

2023-05-27 17:50:32,835 DEBG 'ssh-agent' stderr output: debug3: identity_permitted: entering: key RSA comment "django@cvat-backend-worker-export-57b7c75c79-679cz", 0 socket bindings, 0 constraints debug1: process_add_identity: add ssh-rsa SHA256:kx1H/Kp+1xknufZlSVxut3W0wwaC+QBVtwyLgOOLQbs "django@cvat-backend-worker-export-57b7c75c79-679cz" (life: 0) (confirm: 0) (provider: none) (destination constraints: 0)

2023-05-27 17:50:32,839 DEBG 'ssh-agent' stderr output: debug1: new_socket: type = CONNECTION debug2: fd 4 setting O_NONBLOCK debug1: process_message: socket 1 (fd=4) type 11 debug2: process_request_identities: entering debug3: identity_permitted: entering: key RSA comment "django@cvat-backend-worker-export-57b7c75c79-679cz", 0 socket bindings, 0 constraints debug2: process_request_identities: replying with 1 allowed of 1 available keys

2023-05-27 17:50:33,827 DEBG 'runserver' stderr output: [Sat May 27 17:50:33.827240 2023] [wsgi:error] [pid 985:tid 139630162835008] [remote 192.168.251.152:32782] mod_wsgi (pid=985): Queue timeout expired for WSGI daemon process 'localhost:8080'. [Sat May 27 17:50:33.827330 2023] [wsgi:error] [pid 985:tid 139630154442304] [remote 192.168.251.158:42802] mod_wsgi (pid=985): Queue timeout expired for WSGI daemon process 'localhost:8080'.

2023-05-27 17:50:33,827 DEBG 'runserver' stderr output: [Sat May 27 17:50:33.827380 2023] [wsgi:error] [pid 985:tid 139630162835008] [remote 192.168.251.158:46290] mod_wsgi (pid=985): Queue timeout expired for WSGI daemon process 'localhost:8080'. [Sat May 27 17:50:33.827391 2023] [wsgi:error] [pid 985:tid 139630146049600] [remote 192.168.251.152:58362] mod_wsgi (pid=985): Queue timeout expired for WSGI daemon process 'localhost:8080'. [Sat May 27 17:50:33.827418 2023] [wsgi:error] [pid 985:tid 139630129264192] [remote 192.168.251.158:41920] mod_wsgi (pid=985): Queue timeout expired for WSGI daemon process 'localhost:8080'. [Sat May 27 17:50:33.827516 2023] [wsgi:info] [pid 985:tid 139630154442304] mod_wsgi (pid=985): Application startup timer triggered 'localhost:8080'. [Sat May 27 17:50:33.827517 2023] [wsgi:error] [pid 985:tid 139630137656896] [remote 192.168.251.152:56128] mod_wsgi (pid=985): Queue timeout expired for WSGI daemon process 'localhost:8080'.

2023-05-27 17:50:33,827 DEBG 'runserver' stderr output: [Sat May 27 17:50:33.827621 2023] [wsgi:info] [pid 985:tid 139630154442304] mod_wsgi (pid=985): Application startup timer cancelled 'localhost:8080'.

2023-05-27 17:50:37,891 DEBG 'runserver' stderr output: [Sat May 27 17:50:37.891695 2023] [wsgi:error] [pid 985:tid 139630129264192] [remote 192.168.251.158:37642] [2023-05-27 17:50:37,891] WARNING django.request: Unauthorized: /api/events

2023-05-27 17:50:37,894 DEBG 'runserver' stderr output: [Sat May 27 17:50:37.893960 2023] [wsgi:error] [pid 985:tid 139630129264192] [remote 192.168.251.158:37642] WARNING:django.request:Unauthorized: /api/events

2023-05-27 17:51:33,887 DEBG 'runserver' stderr output: [Sat May 27 17:51:33.887711 2023] [wsgi:error] [pid 744:tid 139633253672512] [client 192.168.251.158:49386] Timeout when reading response headers from daemon process 'localhost:8080': /tmp/cvat-server/htdocs/api, referer: http://cvat.gsrv.lan:32296/

2023-05-27 17:51:33,888 DEBG 'runserver' stderr output: [Sat May 27 17:51:33.888291 2023] [wsgi:error] [pid 744:tid 139633253140032] [client 192.168.251.158:49396] Timeout when reading response headers from daemon process 'localhost:8080': /tmp/cvat-server/htdocs/api, referer: http://cvat.gsrv.lan:32296/

2023-05-27 17:51:33,891 DEBG 'runserver' stderr output: [Sat May 27 17:51:33.891583 2023] [wsgi:error] [pid 663:tid 139633253140032] [client 192.168.251.158:46128] Timeout when reading response headers from daemon process 'localhost:8080': /tmp/cvat-server/htdocs/api, referer: http://cvat.gsrv.lan:32296/

ganindu7 commented 1 year ago

Screenshot from 2023-05-27 18-54-55

ganindu7 commented 1 year ago

Control panel is working fine

Screenshot from 2023-05-27 18-58-58

my values.override.yaml

  secret:
    password: password***
    postgres_password: password***
    replication_password: password***

analytics:
  enabled: false

Apart from that everything else is stock, (used a NFS storage provisoner)

ganindu7 commented 1 year ago

I Think the cause may be because of Screenshot from 2023-05-27 19-40-20

shintaro-matsui commented 1 year ago

@azhavoro

Is it the following steps? My CVAT root directory seems to have no .env.

Open a new terminal window. Run VScode from the virtual environment (run the following command from CVAT root directory): source .env/bin/activate && code Inside VScode, Open CVAT root dir Select server: debug configuration and run it (F5) to run REST server and its workers

Results of docker ps

CONTAINER ID   IMAGE                                      COMMAND                   CREATED          STATUS          PORTS                                                                                          NAMES
1e063d97e199   cvat/ui:dev                                "/docker-entrypoint.…"   11 minutes ago   Up 11 minutes   80/tcp                                                                                         cvat_ui
6931beed6959   cvat/server:dev                            "/usr/bin/supervisor…"   11 minutes ago   Up 11 minutes   8080/tcp                                                                                       cvat_worker_annotation
5a02e6ed0edc   cvat/server:dev                            "/usr/bin/supervisor…"   11 minutes ago   Up 11 minutes   8080/tcp                                                                                       cvat_worker_import
0484b4928b93   cvat/server:dev                            "/usr/bin/supervisor…"   11 minutes ago   Up 11 minutes   8080/tcp                                                                                       cvat_worker_export
216f7056cc9a   cvat/server:dev                            "/usr/bin/supervisor…"   11 minutes ago   Up 11 minutes   8080/tcp                                                                                       cvat_server
9444ef2509f5   cvat/server:dev                            "/usr/bin/supervisor…"   11 minutes ago   Up 11 minutes   8080/tcp                                                                                       cvat_utils
ad1bb01b1524   cvat/server:dev                            "/usr/bin/supervisor…"   11 minutes ago   Up 11 minutes   8080/tcp                                                                                       cvat_worker_webhooks
06fabbcbaac1   timberio/vector:0.26.0-alpine              "/usr/local/bin/vect…"   11 minutes ago   Up 11 minutes                                                                                                  cvat_vector
a8b39d7e884f   clickhouse/clickhouse-server:22.3-alpine   "/entrypoint.sh"          11 minutes ago   Up 11 minutes   8123/tcp, 9000/tcp, 9009/tcp                                                                   cvat_clickhouse
32c1ba705e9e   traefik:v2.9                               "/entrypoint.sh --pr…"   11 minutes ago   Up 11 minutes   0.0.0.0:8080->8080/tcp, :::8080->8080/tcp, 80/tcp, 0.0.0.0:8090->8090/tcp, :::8090->8090/tcp   traefik
e2a8a54fc330   postgres:15-alpine                         "docker-entrypoint.s…"   11 minutes ago   Up 11 minutes   5432/tcp                                                                                       cvat_db
03df36df1f3f   grafana/grafana-oss:9.3.6                  "sh -euc 'mkdir -p /…"   11 minutes ago   Up 11 minutes   3000/tcp                                                                                       cvat_grafana
d4b97986f685   redis:4.0-alpine                           "docker-entrypoint.s…"   11 minutes ago   Up 11 minutes   6379/tcp                                                                                       cvat_redis
97ef4c98312c   openpolicyagent/opa:0.45.0-rootless        "/opa run --server -…"   11 minutes ago   Up 11 minutes                                                                                                  cvat_opa

Results of docker logs cvat_opa

{"level":"error","msg":"Bundle load failed: request failed: Get \"http://cvat-server:8080/api/auth/rules\": dial tcp 172.18.0.9:8080: connect: connection refused","name":"cvat","plugin":"bundle","time":"2023-05-29T00:08:51Z"}
{"level":"error","msg":"Bundle load failed: request failed: Get \"http://cvat-server:8080/api/auth/rules\": dial tcp 172.18.0.9:8080: connect: connection refused","name":"cvat","plugin":"bundle","time":"2023-05-29T00:08:51Z"}
{"level":"error","msg":"Bundle load failed: request failed: Get \"http://cvat-server:8080/api/auth/rules\": dial tcp 172.18.0.9:8080: connect: connection refused","name":"cvat","plugin":"bundle","time":"2023-05-29T00:08:51Z"}
{"level":"error","msg":"Bundle load failed: request failed: Get \"http://cvat-server:8080/api/auth/rules\": dial tcp 172.18.0.9:8080: connect: connection refused","name":"cvat","plugin":"bundle","time":"2023-05-29T00:08:51Z"}
{"level":"error","msg":"Bundle load failed: request failed: Get \"http://cvat-server:8080/api/auth/rules\": dial tcp 172.18.0.9:8080: connect: connection refused","name":"cvat","plugin":"bundle","time":"2023-05-29T00:08:52Z"}
{"level":"error","msg":"Bundle load failed: request failed: Get \"http://cvat-server:8080/api/auth/rules\": dial tcp 172.18.0.9:8080: connect: connection refused","name":"cvat","plugin":"bundle","time":"2023-05-29T00:08:52Z"}
{"level":"error","msg":"Bundle load failed: request failed: Get \"http://cvat-server:8080/api/auth/rules\": dial tcp 172.18.0.9:8080: connect: connection refused","name":"cvat","plugin":"bundle","time":"2023-05-29T00:08:53Z"}
{"level":"error","msg":"Bundle load failed: request failed: Get \"http://cvat-server:8080/api/auth/rules\": dial tcp 172.18.0.9:8080: connect: connection refused","name":"cvat","plugin":"bundle","time":"2023-05-29T00:08:55Z"}
{"level":"error","msg":"Bundle load failed: request failed: Get \"http://cvat-server:8080/api/auth/rules\": dial tcp 172.18.0.9:8080: connect: connection refused","name":"cvat","plugin":"bundle","time":"2023-05-29T00:08:58Z"}
{"level":"error","msg":"Bundle load failed: request failed: Get \"http://cvat-server:8080/api/auth/rules\": dial tcp 172.18.0.9:8080: connect: connection refused","name":"cvat","plugin":"bundle","time":"2023-05-29T00:09:02Z"}
{"level":"error","msg":"Bundle load failed: request failed: Get \"http://cvat-server:8080/api/auth/rules\": dial tcp 172.18.0.9:8080: connect: connection refused","name":"cvat","plugin":"bundle","time":"2023-05-29T00:09:08Z"}

shintaro-matsui commented 1 year ago

I checked #5922 and confirmed that there is no problem with the memory limit.

pepijnob commented 1 year ago

I have the same error. Tried installing it on a new PC, it was working in the beginning, then after deploying Segment anything on Nuclio I got the same error on the new PC as well. When I delete the function in the nuclio dashboard, it is still not working.

AdimasBarelang commented 1 year ago

I got the same problems, at first it will works fine but then. The error appears to be said the server is error like posr above.

trandinh commented 1 year ago

I got the same problems and had already checked and confirmed that there is no problem with the memory. For me I found out that the problems may related to cvat_db -> there are no table created ( no List of relations found in cvat_db) but still not yet found out where the problems come from

cvat_db log:


fixing permissions on existing directory /var/lib/postgresql/data ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... UTC
creating configuration files ... ok
running bootstrap script ... ok
sh: locale: not found
2023-06-26 10:04:26.717 UTC [31] WARNING:  no usable system locales were found
performing post-bootstrap initialization ... ok
syncing data to disk ... ok

initdb: warning: enabling "trust" authentication for local connections
initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.

Success. You can now start the database server using:

    pg_ctl -D /var/lib/postgresql/data -l logfile start

waiting for server to start....2023-06-26 10:05:10.292 UTC [37] LOG:  starting PostgreSQL 15.3 on x86_64-pc-linux-musl, compiled by gcc (Alpine 12.2.1_git20220924-r10) 12.2.1 20220924, 64-bit
2023-06-26 10:05:10.584 UTC [37] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
.2023-06-26 10:05:10.960 UTC [40] LOG:  database system was shut down at 2023-06-26 10:04:31 UTC
2023-06-26 10:05:11.312 UTC [37] LOG:  database system is ready to accept connections
 done
server started
CREATE DATABASE

/usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*

waiting for server to shut down....2023-06-26 10:05:12.280 UTC [37] LOG:  received fast shutdown request
2023-06-26 10:05:12.485 UTC [37] LOG:  aborting any active transactions
2023-06-26 10:05:12.489 UTC [37] LOG:  background worker "logical replication launcher" (PID 43) exited with exit code 1
2023-06-26 10:05:12.489 UTC [38] LOG:  shutting down
2023-06-26 10:05:12.850 UTC [38] LOG:  checkpoint starting: shutdown immediate
...........2023-06-26 10:05:23.427 UTC [38] LOG:  checkpoint complete: wrote 918 buffers (5.6%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.792 s, sync=8.863 s, total=10.938 s; sync files=250, longest=0.350 s, average=0.036 s; distance=4222 kB, estimate=4222 kB
2023-06-26 10:05:23.435 UTC [37] LOG:  database system is shut down
 done
server stopped

PostgreSQL init process complete; ready for start up.

2023-06-26 10:05:23.842 UTC [1] LOG:  starting PostgreSQL 15.3 on x86_64-pc-linux-musl, compiled by gcc (Alpine 12.2.1_git20220924-r10) 12.2.1 20220924, 64-bit
2023-06-26 10:05:23.842 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2023-06-26 10:05:23.842 UTC [1] LOG:  listening on IPv6 address "::", port 5432
2023-06-26 10:05:24.307 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2023-06-26 10:05:24.567 UTC [54] LOG:  incomplete startup packet
2023-06-26 10:05:24.567 UTC [53] LOG:  database system was shut down at 2023-06-26 10:05:23 UTC
2023-06-26 10:05:24.881 UTC [1] LOG:  database system is ready to accept connections
2023-06-26 10:06:28.728 UTC [61] ERROR:  relation "authtoken_token" does not exist at character 363
2023-06-26 10:06:28.728 UTC [61] STATEMENT:  SELECT "authtoken_token"."key", "authtoken_token"."user_id", "authtoken_token"."created", "auth_user"."id", "auth_user"."password", "auth_user"."last_login", "auth_user"."is_superuser", "auth_user"."username", "auth_user"."first_name", "auth_user"."last_name", "auth_user"."email", "auth_user"."is_staff", "auth_user"."is_active", "auth_user"."date_joined" FROM "authtoken_token" INNER JOIN "auth_user" ON ("authtoken_token"."user_id" = "auth_user"."id") WHERE "authtoken_token"."key" = '607c2de23bd84928f8a397d3ca8a96a2836351a7' LIMIT 21

look like problems start from CREATE DATABASE



/usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*

waiting for server to shut down....2023-06-26 10:05:12.280 UTC [37] LOG:  received fast shutdown request
2023-06-26 10:05:12.485 UTC [37] LOG:  aborting any active transactions
2023-06-26 10:05:12.489 UTC [37] LOG:  background worker "logical replication launcher" (PID 43) exited with exit code 1
2023-06-26 10:05:12.489 UTC [38] LOG:  shutting down
2023-06-26 10:05:12.850 UTC [38] LOG:  checkpoint starting: shutdown immediate
...........2023-06-26 10:05:23.427 UTC [38] LOG:  checkpoint complete: wrote 918 buffers (5.6%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.792 s, sync=8.863 s, total=10.938 s; sync files=250, longest=0.350 s, average=0.036 s; distance=4222 kB, estimate=4222 kB
2023-06-26 10:05:23.435 UTC [37] LOG:  database system is shut down
 done
server stopped

PostgreSQL init process complete; ready for start up.```

SjoerdTimovanVliet commented 1 year ago

@shintaro-matsui did you solve it? I have the same issue I am using ubuntu 22.04 LTS and cvat version v2.6.0 and also had this issue with v2.5.0

may109 commented 1 year ago

same problem here, with v246 and v260. It works at the beginning for few hours and starts to fail, with a slightly different error.

cvat_db                    FATAL:  password authentication failed for user "root"
cvat_db                  DETAIL:  User "root" has no password assigned.
cvat_db                   |     Connection matched pg_hba.conf line 102: "host all all all md5"
cvat_server               | Traceback (most recent call last):
cvat_server               |   File "/opt/venv/lib/python3.10/site-packages/django/db/backends/base/base.py", line 289, in ensure_connection
cvat_server               |     self.connect()
cvat_server               |   File "/opt/venv/lib/python3.10/site-packages/django/utils/asyncio.py", line 26, in inner
cvat_server               |     return func(*args, **kwargs)
cvat_server               |   File "/opt/venv/lib/python3.10/site-packages/django/db/backends/base/base.py", line 270, in connect
cvat_server               |     self.connection = self.get_new_connection(conn_params)
cvat_server               |   File "/opt/venv/lib/python3.10/site-packages/django/utils/asyncio.py", line 26, in inner
cvat_server               |     return func(*args, **kwargs)
cvat_server               |   File "/opt/venv/lib/python3.10/site-packages/django/db/backends/postgresql/base.py", line 275, in get_new_connection
cvat_server               |     connection = self.Database.connect(**conn_params)
cvat_server               |   File "/opt/venv/lib/python3.10/site-packages/psycopg2/__init__.py", line 122, in connect
cvat_server               |     conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
cvat_server               | psycopg2.OperationalError: connection to server at "cvat_db" (192.168.16.3), port 5432 failed: fe_sendauth: no password supplied

SjoerdTimovanVliet commented 1 year ago

Any workarounds found?

shintaro-matsui commented 1 year ago

Hi. Unfortunately, this issue has not been resolved. I am not working on this issue at this time as I reinstalled the OS once due to a number of other issues.

I am sorry that I could not be of help.

Petros626 commented 1 year ago

isnt it just docker compose down and docker compose up -d which can help to avoid this. I've read this in some issues here

bartmch commented 11 months ago

I exactly have the same issue as the OP. Unfortunately docker compose down/up did not work.

When I ran CVAT as per OP's description for the first time, it ran fine and was able to annotate an image. Later, when I wanted to start with the next image I got the same "Cannot connect to server" error. I deleted everything to start from scratch, but it was giving me the "Cannot connect to server" from the start.

leotor4 commented 10 months ago

cvat_server takes time to initialize. It's normal to see the following error message sometimes during initialization:

{"level":"error","msg":"Bundle load failed: request failed: Get \"http://cvat-server:8080/api/auth/rules\": dial tcp 172.18.0.9:8080: connect: connection refused","name":"cvat","plugin":"bundle","time":"2023-05-29T00:08:51Z"}

Wait for a few minutes, and then check if you can access your application.

*Remember to persist your CVAT_HOST environment variable.

jtziebarth commented 2 months ago

@bsekachev Just jumping in to say that I have this issue too running v2.16.2. Currently the only solution I know is to wipe docker clean and rebuild all over again, which is very inconvenient. Is there any other solution that exists?

jpfleischer commented 1 month ago

cvat_server takes time to initialize. It's normal to see the following error message sometimes during initialization:

{"level":"error","msg":"Bundle load failed: request failed: Get \"http://cvat-server:8080/api/auth/rules\": dial tcp 172.18.0.9:8080: connect: connection refused","name":"cvat","plugin":"bundle","time":"2023-05-29T00:08:51Z"}

Wait for a few minutes, and then check if you can access your application.

*Remember to persist your CVAT_HOST environment variable.

Thanks, the solution was to export CVAT_HOST=mysite.com

cvat-ai / cvat