search

cvat-ai / cvat

Annotate better with CVAT, the industry-leading data engine for machine learning. Used and trusted by teams at any scale, for data of any scale.
https://cvat.ai
MIT License
12.57k stars 3k forks source link

OPA cannot receive rules from CVAT (request failed: Get \"http://cvat-server:8080/api/auth/rules\") #5922

Open linClubs opened 1 year ago

linClubs commented 1 year ago

Hi, I open the Google Chrome browser and go to localhost:8080.

the page show:

Cannot connect to the server Make sure the CVAT backend and all necessary services (Database, Redis and Open Policy Agent) are running and available. If you upgraded from version 2.2.0 or earlier, manual actions may be needed, see the Upgrade Guide.

I git clone v2.4.0.

There's not login page and the Login button, how can i solve it? Thanks

azhavoro commented 1 year ago

Please provide logs from cvat_server and cvat_db containers

emanuelevivoli commented 1 year ago

I have the same problems. My cvat_server is:

2023-04-03 14:59:32,675 INFO RPC interface 'supervisor' initialized
2023-04-03 14:59:32,676 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2023-04-03 14:59:32,676 INFO supervisord started with pid 1
2023-04-03 14:59:33,679 INFO spawned: 'clamav_update' with pid 7
2023-04-03 14:59:33,681 INFO spawned: 'runserver' with pid 8
2023-04-03 14:59:33,688 DEBG fd 7 closed, stopped monitoring <POutputDispatcher at 140465471488352 for <Subprocess at 140465471488448 with name clamav_update in state STARTING> (stdout)>
2023-04-03 14:59:33,689 DEBG fd 9 closed, stopped monitoring <POutputDispatcher at 140465471275168 for <Subprocess at 140465471488448 with name clamav_update in state STARTING> (stderr)>
2023-04-03 14:59:33,689 INFO exited: clamav_update (exit status 0; not expected)
2023-04-03 14:59:33,689 DEBG received SIGCHLD indicating a child quit
2023-04-03 14:59:33,696 DEBG 'runserver' stderr output:
wait-for-it.sh: waiting for cvat_db:5432 without a timeout

2023-04-03 14:59:33,701 DEBG 'runserver' stderr output:
wait-for-it.sh: cvat_db:5432 is available after 0 seconds

2023-04-03 14:59:33,704 DEBG 'runserver' stderr output:
bash: cannot set terminal process group (-1): Inappropriate ioctl for device
bash: no job control in this shell

2023-04-03 14:59:34,707 INFO spawned: 'clamav_update' with pid 20
2023-04-03 14:59:34,708 INFO success: runserver entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-04-03 14:59:34,717 DEBG fd 7 closed, stopped monitoring <POutputDispatcher at 140465471488352 for <Subprocess at 140465471488448 with name clamav_update in state STARTING> (stdout)>
2023-04-03 14:59:34,717 DEBG fd 11 closed, stopped monitoring <POutputDispatcher at 140465471275216 for <Subprocess at 140465471488448 with name clamav_update in state STARTING> (stderr)>
2023-04-03 14:59:34,717 INFO exited: clamav_update (exit status 0; not expected)
2023-04-03 14:59:34,718 DEBG received SIGCHLD indicating a child quit
2023-04-03 14:59:36,722 INFO spawned: 'clamav_update' with pid 53
2023-04-03 14:59:36,732 DEBG fd 7 closed, stopped monitoring <POutputDispatcher at 140465471488352 for <Subprocess at 140465471488448 with name clamav_update in state STARTING> (stdout)>
2023-04-03 14:59:36,732 DEBG fd 11 closed, stopped monitoring <POutputDispatcher at 140465471275456 for <Subprocess at 140465471488448 with name clamav_update in state STARTING> (stderr)>
2023-04-03 14:59:36,732 INFO exited: clamav_update (exit status 0; not expected)
2023-04-03 14:59:36,732 DEBG received SIGCHLD indicating a child quit
2023-04-03 14:59:37,926 DEBG 'runserver' stderr output:
WARNING:root:Failed to import module 'datumaro.plugins.tf_detection_api_format.converter': The TensorFlow library was compiled to use AVX instructions, but these aren't available on your machine.

2023-04-03 14:59:39,537 DEBG 'runserver' stdout output:
Operations to perform:
  Apply all migrations: account, admin, auth, authtoken, contenttypes, dataset_repo, db, engine, organizations, sessions, sites, socialaccount, webhooks
Running migrations:
  No migrations to apply.

2023-04-03 14:59:40,540 INFO spawned: 'clamav_update' with pid 59
2023-04-03 14:59:40,547 DEBG fd 7 closed, stopped monitoring <POutputDispatcher at 140465471488352 for <Subprocess at 140465471488448 with name clamav_update in state STARTING> (stdout)>
2023-04-03 14:59:40,547 DEBG fd 11 closed, stopped monitoring <POutputDispatcher at 140465471275744 for <Subprocess at 140465471488448 with name clamav_update in state STARTING> (stderr)>
2023-04-03 14:59:40,547 INFO exited: clamav_update (exit status 0; not expected)
2023-04-03 14:59:40,547 DEBG received SIGCHLD indicating a child quit
2023-04-03 14:59:41,549 INFO gave up: clamav_update entered FATAL state, too many start retries too quickly
2023-04-03 14:59:42,135 DEBG 'runserver' stdout output:

165 static files copied to '/home/django/static'.

2023-04-03 14:59:45,434 DEBG 'runserver' stderr output:
WARNING:root:Failed to import module 'datumaro.plugins.tf_detection_api_format.converter': The TensorFlow library was compiled to use AVX instructions, but these aren't available on your machine.

2023-04-03 14:59:46,256 DEBG 'runserver' stdout output:
Successfully ran command.

2023-04-03 14:59:46,262 DEBG 'runserver' stdout output:
Server URL         : http://localhost:8080/
Server Root        : /tmp/cvat-server
Server Conf        : /tmp/cvat-server/httpd.conf
Error Log File     : /dev/stderr (INFO)
Request Capacity   : 5 (1 process * 5 threads)
Request Timeout    : 60 (seconds)
Startup Timeout    : 15 (seconds)
Queue Backlog      : 100 (connections)
Queue Timeout      : 45 (seconds)
Server Capacity    : 20 (event/worker), 20 (prefork)
Server Backlog     : 500 (connections)
Locale Setting     : C.UTF-8

2023-04-03 14:59:46,316 DEBG 'runserver' stderr output:
[Mon Apr 03 14:59:46.315985 2023] [wsgi:info] [pid 114:tid 139995224788032] mod_wsgi (pid=114): Starting process 'localhost:8080' with threads=5.
[Mon Apr 03 14:59:46.316035 2023] [mpm_event:notice] [pid 8:tid 139995224788032] AH00489: Apache/2.4.41 (Ubuntu) mod_wsgi/4.7.1 Python/3.8 configured -- resuming normal operations
[Mon Apr 03 14:59:46.316114 2023] [mpm_event:info] [pid 8:tid 139995224788032] AH00490: Server built: 2023-03-08T17:32:54
[Mon Apr 03 14:59:46.316122 2023] [core:notice] [pid 8:tid 139995224788032] AH00094: Command line: 'apache2 (mod_wsgi-express) -f /tmp/cvat-server/httpd.conf -D MOD_WSGI_MPM_ENABLE_EVENT_MODULE -D MOD_WSGI_MPM_EXISTS_EVENT_MODULE -D MOD_WSGI_MPM_EXISTS_WORKER_MODULE -D MOD_WSGI_MPM_EXISTS_PREFORK_MODULE -D FOREGROUND'

2023-04-03 14:59:46,316 DEBG 'runserver' stderr output:
[Mon Apr 03 14:59:46.316593 2023] [wsgi:info] [pid 114:tid 139995224788032] mod_wsgi (pid=114): Python home /opt/venv.

2023-04-03 14:59:46,316 DEBG 'runserver' stderr output:
[Mon Apr 03 14:59:46.316668 2023] [wsgi:info] [pid 114:tid 139995224788032] mod_wsgi (pid=114): Initializing Python.

2023-04-03 14:59:46,354 DEBG 'runserver' stderr output:
[Mon Apr 03 14:59:46.354421 2023] [wsgi:info] [pid 114:tid 139995224788032] mod_wsgi (pid=114): Attach interpreter ''.

2023-04-03 14:59:46,360 DEBG 'runserver' stderr output:
[Mon Apr 03 14:59:46.360286 2023] [wsgi:info] [pid 114:tid 139995224788032] mod_wsgi (pid=114): Imported 'mod_wsgi'.

2023-04-03 14:59:46,360 DEBG 'runserver' stderr output:
[Mon Apr 03 14:59:46.360359 2023] [wsgi:info] [pid 114:tid 139995224788032] mod_wsgi (pid=114, process='localhost:8080', application=''): Loading Python script file '/tmp/cvat-server/handler.wsgi'.

and my cvat_db is:

PostgreSQL Database directory appears to contain a database; Skipping initialization

2023-04-03 14:59:30.261 UTC [1] LOG:  starting PostgreSQL 15.2 on x86_64-pc-linux-musl, compiled by gcc (Alpine 12.2.1_git20220924-r4) 12.2.1 20220924, 64-bit
2023-04-03 14:59:30.261 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2023-04-03 14:59:30.261 UTC [1] LOG:  listening on IPv6 address "::", port 5432
2023-04-03 14:59:30.278 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2023-04-03 14:59:30.298 UTC [24] LOG:  database system was shut down at 2023-04-03 14:52:30 UTC
2023-04-03 14:59:30.307 UTC [1] LOG:  database system is ready to accept connections
2023-04-03 14:59:33.700 UTC [28] LOG:  incomplete startup packet

I had this problem and I thought was because of memory limit, but apparently it's not.

My docker logs cvat_opa is the following:

{"addrs":[":8181"],"diagnostic-addrs":[],"level":"info","msg":"Initializing server.","time":"2023-04-03T15:13:09Z"}
{"level":"info","msg":"Starting bundle loader.","name":"cvat","plugin":"bundle","time":"2023-04-03T15:13:09Z"}
{"level":"info","msg":"Starting decision logger.","plugin":"decision_logs","time":"2023-04-03T15:13:09Z"}
{"level":"error","msg":"Bundle load failed: request failed: Get \"http://cvat-server:8080/api/auth/rules\": dial tcp: lookup cvat-server: Temporary failure in name resolution","name":"cvat","plugin":"bundle","time":"2023-04-03T15:13:09Z"}
{"level":"error","msg":"Bundle load failed: request failed: Get \"http://cvat-server:8080/api/auth/rules\": dial tcp: lookup cvat-server: Temporary failure in name resolution","name":"cvat","plugin":"bundle","time":"2023-04-03T15:13:09Z"}
{"level":"error","msg":"Bundle load failed: request failed: Get \"http://cvat-server:8080/api/auth/rules\": dial tcp: lookup cvat-server: Temporary failure in name resolution","name":"cvat","plugin":"bundle","time":"2023-04-03T15:13:09Z"}
{"level":"error","msg":"Bundle load failed: request failed: Get \"http://cvat-server:8080/api/auth/rules\": dial tcp: lookup cvat-server: Temporary failure in name resolution","name":"cvat","plugin":"bundle","time":"2023-04-03T15:13:09Z"}
{"current_version":"0.45.0","download_opa":"https://openpolicyagent.org/downloads/v0.51.0/opa_linux_amd64","latest_version":"0.51.0","level":"info","msg":"OPA is out of date.","release_notes":"https://github.com/open-policy-agent/opa/releases/tag/v0.51.0","time":"2023-04-03T15:13:09Z"}
{"level":"error","msg":"Bundle load failed: request failed: Get \"http://cvat-server:8080/api/auth/rules\": dial tcp: lookup cvat-server: Temporary failure in name resolution","name":"cvat","plugin":"bundle","time":"2023-04-03T15:13:10Z"}
{"level":"error","msg":"Bundle load failed: request failed: Get \"http://cvat-server:8080/api/auth/rules\": dial tcp 172.19.0.11:8080: connect: connection refused","name":"cvat","plugin":"bundle","time":"2023-04-03T15:13:11Z"}
{"level":"error","msg":"Bundle load failed: request failed: Get \"http://cvat-server:8080/api/auth/rules\": dial tcp 172.19.0.11:8080: connect: connection refused","name":"cvat","plugin":"bundle","time":"2023-04-03T15:13:12Z"}
{"level":"error","msg":"Bundle load failed: request failed: Get \"http://cvat-server:8080/api/auth/rules\": dial tcp 172.19.0.11:8080: connect: connection refused","name":"cvat","plugin":"bundle","time":"2023-04-03T15:13:14Z"}
{"level":"error","msg":"Bundle load failed: request failed: Get \"http://cvat-server:8080/api/auth/rules\": dial tcp 172.19.0.11:8080: connect: connection refused","name":"cvat","plugin":"bundle","time":"2023-04-03T15:13:16Z"}
{"level":"error","msg":"Bundle load failed: request failed: Get \"http://cvat-server:8080/api/auth/rules\": dial tcp 172.19.0.11:8080: connect: connection refused","name":"cvat","plugin":"bundle","time":"2023-04-03T15:13:20Z"}
{"level":"error","msg":"Bundle load failed: request failed: Get \"http://cvat-server:8080/api/auth/rules\": dial tcp 172.19.0.11:8080: connect: connection refused","name":"cvat","plugin":"bundle","time":"2023-04-03T15:13:28Z"}
Lee4396 commented 1 year ago

same issue

remivanbel commented 1 year ago

Same issue after fresh install on Ubuntu 20.04

EDIT: my problem was >90% disk storage (https://github.com/opencv/cvat/issues/5449). Fixed by clearing up some space.

Check if disk storage is limiting with docker exec -t cvat_server python manage.py health_check (from installation guide )

kikislater commented 1 year ago

Same issue after fresh install on Ubuntu 20.04

EDIT: my problem was >90% disk storage (#5449). Fixed by clearing up some space.

Check if disk storage is limiting with docker exec -t cvat_server python manage.py health_check (from installation guide )

Same behaviour with 10% disk remaining on 2To storage ... I didn't think it was the problem ! It should be nice if user could remove this limitation.

jenfredwell commented 1 year ago

I'm also seeing this but it does NOT appear to be the disk storage issue.

python manage.py health_check
WARNING:HEALTH:getting Cache backend: default health status
WARNING:HEALTH:getting Cache backend: media health status
WARNING:HEALTH:gettin OPA health status
WARNING:HEALTH:DONE getting Cache backend: default health status
ERROR:health-check:unknown error: 500 Server Error: Internal Server Error for url: http://opa:8181/health?bundles
Traceback (most recent call last):
  File "/home/django/cvat/apps/health/backends.py", line 29, in check_status
    response.raise_for_status()
  File "/opt/venv/lib/python3.8/site-packages/requests/models.py", line 953, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http://opa:8181/health?bundles
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/opt/venv/lib/python3.8/site-packages/health_check/backends.py", line 30, in run_check
    self.check_status()
  File "/home/django/cvat/apps/health/backends.py", line 31, in check_status
    raise HealthCheckException(str(e))
health_check.exceptions.HealthCheckException: unknown error: 500 Server Error: Internal Server Error for url: http://opa:8181/health?bundles
WARNING:HEALTH:DONE getting Cache backend: media health status
Cache backend: default   ... working
Cache backend: media     ... working
DatabaseBackend          ... working
DiskUsage                ... working
MigrationsHealthCheck    ... working
OPAHealthCheck           ... unknown error: 500 Server Error: Internal Server Error for url: http://opa:8181/health?bundles

My cvat_server logs are repeatedly print out this and then restarting

2023-05-04 17:58:24,717 DEBG 'runserver' stderr output:
[Thu May 04 17:58:24.717105 2023] [wsgi:error] [pid 178:tid 139838808360704] WARNING:HEALTH:getting Cache backend: default health status

2023-05-04 17:58:24,720 DEBG 'runserver' stderr output:
[Thu May 04 17:58:24.720766 2023] [wsgi:error] [pid 178:tid 139838850307840] WARNING:HEALTH:getting Cache backend: media health status

2023-05-04 17:58:24,722 DEBG 'runserver' stderr output:
[Thu May 04 17:58:24.722321 2023] [wsgi:error] [pid 178:tid 139837506574080] WARNING:HEALTH:getting Cache backend: media health status

2023-05-04 17:58:24,723 DEBG 'runserver' stderr output:
[Thu May 04 17:58:24.723420 2023] [wsgi:error] [pid 178:tid 139837155833600] WARNING:HEALTH:getting Cache backend: media health status

2023-05-04 17:58:24,724 DEBG 'runserver' stderr output:
[Thu May 04 17:58:24.724269 2023] [wsgi:error] [pid 178:tid 139837122262784] WARNING:HEALTH:gettin OPA health status

2023-05-04 17:58:24,725 DEBG 'runserver' stderr output:
[Thu May 04 17:58:24.725515 2023] [wsgi:error] [pid 178:tid 139837113870080] WARNING:HEALTH:gettin OPA health status

2023-05-04 17:58:24,726 DEBG 'runserver' stderr output:
[Thu May 04 17:58:24.726481 2023] [wsgi:error] [pid 178:tid 139837105477376] WARNING:HEALTH:gettin OPA health status

2023-05-04 17:58:24,757 DEBG 'runserver' stderr output:
[Thu May 04 17:58:24.757136 2023] [wsgi:error] [pid 178:tid 139837113870080] ERROR:health-check:unknown error: 500 Server Error: Internal Server Error for url: http://opa:8181/health?bundles

2023-05-04 17:58:24,758 DEBG 'runserver' stderr output:
[Thu May 04 17:58:24.758408 2023] [wsgi:error] [pid 178:tid 139837113870080] Traceback (most recent call last):

2023-05-04 17:58:24,759 DEBG 'runserver' stderr output:
[Thu May 04 17:58:24.759369 2023] [wsgi:error] [pid 178:tid 139837113870080]   File "/home/django/cvat/apps/health/backends.py", line 29, in check_status

2023-05-04 17:58:24,762 DEBG 'runserver' stderr output:
[Thu May 04 17:58:24.762101 2023] [wsgi:error] [pid 178:tid 139837113870080]     response.raise_for_status()

2023-05-04 17:58:24,764 DEBG 'runserver' stderr output:
[Thu May 04 17:58:24.764181 2023] [wsgi:error] [pid 178:tid 139837113870080]   File "/opt/venv/lib/python3.8/site-packages/requests/models.py", line 953, in raise_for_status

2023-05-04 17:58:24,764 DEBG 'runserver' stderr output:
[Thu May 04 17:58:24.764181 2023] [wsgi:error] [pid 178:tid 139837113870080]   File "/opt/venv/lib/python3.8/site-packages/requests/models.py", line 953, in raise_for_status

2023-05-04 17:58:24,764 DEBG 'runserver' stderr output:
[Thu May 04 17:58:24.764389 2023] [wsgi:error] [pid 178:tid 139837113870080]     raise HTTPError(http_error_msg, response=self)
[Thu May 04 17:58:24.764422 2023] [wsgi:error] [pid 178:tid 139837113870080] requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http://opa:8181/health?bundles
[Thu May 04 17:58:24.764613 2023] [wsgi:error] [pid 178:tid 139837113870080] 
[Thu May 04 17:58:24.764646 2023] [wsgi:error] [pid 178:tid 139837113870080] During handling of the above exception, another exception occurred:
[Thu May 04 17:58:24.764651 2023] [wsgi:error] [pid 178:tid 139837113870080] 
[Thu May 04 17:58:24.764654 2023] [wsgi:error] [pid 178:tid 139837113870080] Traceback (most recent call last):
[Thu May 04 17:58:24.764657 2023] [wsgi:error] [pid 178:tid 139837113870080]   File "/opt/venv/lib/python3.8/site-packages/health_check/backends.py", line 30, in run_check
[Thu May 04 17:58:24.764660 2023] [wsgi:error] [pid 178:tid 139837113870080]     self.check_status()

2023-05-04 17:58:24,764 DEBG 'runserver' stderr output:
[Thu May 04 17:58:24.764761 2023] [wsgi:error] [pid 178:tid 139837113870080]   File "/home/django/cvat/apps/health/backends.py", line 31, in check_status
[Thu May 04 17:58:24.764812 2023] [wsgi:error] [pid 178:tid 139837113870080]     raise HealthCheckException(str(e))
[Thu May 04 17:58:24.764816 2023] [wsgi:error] [pid 178:tid 139837113870080] health_check.exceptions.HealthCheckException: unknown error: 500 Server Error: Internal Server Error for url: http://opa:8181/health?bundles

When cvat_opa boots up I get these logs:

{"addrs":[":8181"],"diagnostic-addrs":[],"level":"info","msg":"Initializing server.","time":"2023-05-04T17:34:50Z"}
{"level":"info","msg":"Starting bundle loader.","name":"cvat","plugin":"bundle","time":"2023-05-04T17:34:50Z"}
{"level":"info","msg":"Starting decision logger.","plugin":"decision_logs","time":"2023-05-04T17:34:50Z"}
{"level":"error","msg":"Bundle load failed: request failed: Get \"http://cvat-server:8080/api/auth/rules\": dial tcp 10.28.10.242:8080: connect: connection refused","name":"cvat","plugin":"bundle","time":"2023-05-04T17:34:50Z"}
{"level":"error","msg":"Bundle load failed: request failed: Get \"http://cvat-server:8080/api/auth/rules\": dial tcp 10.28.10.242:8080: connect: connection refused","name":"cvat","plugin":"bundle","time":"2023-05-04T17:34:50Z"}
{"level":"error","msg":"Bundle load failed: request failed: Get \"http://cvat-server:8080/api/auth/rules\": dial tcp 10.28.10.242:8080: connect: connection refused","name":"cvat","plugin":"bundle","time":"2023-05-04T17:34:50Z"}
{"level":"error","msg":"Bundle load failed: request failed: Get \"http://cvat-server:8080/api/auth/rules\": dial tcp 10.28.10.242:8080: connect: connection refused","name":"cvat","plugin":"bundle","time":"2023-05-04T17:34:50Z"}
{"level":"error","msg":"Bundle load failed: request failed: Get \"http://cvat-server:8080/api/auth/rules\": dial tcp 10.28.10.242:8080: connect: connection refused","name":"cvat","plugin":"bundle","time":"2023-05-04T17:34:51Z"}
{"level":"error","msg":"Bundle load failed: server replied with Service Unavailable","name":"cvat","plugin":"bundle","time":"2023-05-04T17:34:51Z"}
{"level":"error","msg":"Bundle load failed: server replied with Service Unavailable","name":"cvat","plugin":"bundle","time":"2023-05-04T17:34:52Z"}
{"level":"error","msg":"Bundle load failed: server replied with Service Unavailable","name":"cvat","plugin":"bundle","time":"2023-05-04T17:34:54Z"}
{"level":"error","msg":"Bundle load failed: server replied with Service Unavailable","name":"cvat","plugin":"bundle","time":"2023-05-04T17:34:57Z"}
{"level":"error","msg":"Bundle load failed: server replied with Service Unavailable","name":"cvat","plugin":"bundle","time":"2023-05-04T17:35:00Z"}
{"level":"error","msg":"Bundle load failed: server replied with Service Unavailable","name":"cvat","plugin":"bundle","time":"2023-05-04T17:35:06Z"}
{"level":"error","msg":"Bundle load failed: server replied with Service Unavailable","name":"cvat","plugin":"bundle","time":"2023-05-04T17:35:18Z"}
ryunosuke-sakamaki commented 1 year ago

traefikのログ

time="2023-06-23T08:44:35Z" level=info msg="Configuration loaded from flags."
time="2023-06-23T08:44:35Z" level=error msg="entryPoint \"websecure\" doesn't exist" routerName=grafana_https@file entryPointName=websecure
time="2023-06-23T08:44:35Z" level=error msg="no valid entryPoint for this router" routerName=grafana_https@file
time="2023-06-23T08:44:35Z" level=error msg="entryPoint \"websecure\" doesn't exist" routerName=grafana_https@file entryPointName=websecure
time="2023-06-23T08:44:35Z" level=error msg="no valid entryPoint for this router" routerName=grafana_https@file
time="2023-06-23T08:44:37Z" level=error msg="entryPoint \"websecure\" doesn't exist" routerName=grafana_https@file entryPointName=websecure
time="2023-06-23T08:44:37Z" level=error msg="no valid entryPoint for this router" routerName=grafana_https@file
time="2023-06-23T08:44:37Z" level=error msg="entryPoint \"websecure\" doesn't exist" routerName=grafana_https@file entryPointName=websecure
time="2023-06-23T08:44:37Z" level=error msg="no valid entryPoint for this router" routerName=grafana_https@file
time="2023-06-23T08:44:39Z" level=error msg="entryPoint \"websecure\" doesn't exist" routerName=grafana_https@file entryPointName=websecure
time="2023-06-23T08:44:39Z" level=error msg="no valid entryPoint for this router" routerName=grafana_https@file
time="2023-06-23T08:44:39Z" level=error msg="entryPoint \"websecure\" doesn't exist" routerName=grafana_https@file entryPointName=websecure
time="2023-06-23T08:44:39Z" level=error msg="no valid entryPoint for this router" routerName=grafana_https@file
ajagadi1 commented 1 year ago

@azhavoro This issue with cvat_opa that many users in this thread have mentioned still happens in CVAT v2.7.4 Can you please help with this error?

Output of docker exec -t cvat_server python manage.py health_check command:

[2023-10-12 16:10:35,846] ERROR health-check: unknown error: 500 Server Error: Internal Server Error for url: http://opa:8181/health?bundles
Traceback (most recent call last):
  File "/home/django/cvat/apps/health/backends.py", line 25, in check_status
    response.raise_for_status()
  File "/opt/venv/lib/python3.10/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http://opa:8181/health?bundles

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/venv/lib/python3.10/site-packages/health_check/backends.py", line 30, in run_check
    self.check_status()
  File "/home/django/cvat/apps/health/backends.py", line 27, in check_status
    raise HealthCheckException(str(e))
health_check.exceptions.HealthCheckException: unknown error: 500 Server Error: Internal Server Error for url: http://opa:8181/health?bundles
Cache backend: default   ... working
Cache backend: media     ... working
DatabaseBackend          ... working
DiskUsage                ... working
MemoryUsage              ... working
MigrationsHealthCheck    ... working
OPAHealthCheck           ... unknown error: 500 Server Error: Internal Server Error for url: http://opa:8181/health?bundles

cvat_server logs:

waiting for migrations to complete...
2023-10-12 16:09:26,033 INFO Creating socket unix:///tmp/uvicorn.sock
2023-10-12 16:09:26,033 INFO Closing socket unix:///tmp/uvicorn.sock
2023-10-12 16:09:26,035 INFO RPC interface 'supervisor' initialized
2023-10-12 16:09:26,036 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2023-10-12 16:09:26,036 INFO supervisord started with pid 1
2023-10-12 16:09:27,039 INFO spawned: 'clamav_update' with pid 128
2023-10-12 16:09:27,041 INFO spawned: 'nginx-0' with pid 129
2023-10-12 16:09:27,043 INFO spawned: 'smokescreen' with pid 130
2023-10-12 16:09:27,043 INFO Creating socket unix:///tmp/uvicorn.sock
2023-10-12 16:09:27,044 DEBG fd 20 closed, stopped monitoring <PInputDispatcher at 140003956306944 for <Subprocess at 140003956295568 with name uvicorn-0 in state STARTING> (stdin)>
2023-10-12 16:09:27,048 INFO spawned: 'uvicorn-0' with pid 131
2023-10-12 16:09:27,088 DEBG fd 24 closed, stopped monitoring <PInputDispatcher at 140003956307520 for <Subprocess at 140003956295424 with name uvicorn-1 in state STARTING> (stdin)>
2023-10-12 16:09:27,090 INFO spawned: 'uvicorn-1' with pid 145
2023-10-12 16:09:27,136 DEBG fd 7 closed, stopped monitoring <POutputDispatcher at 140003955831136 for <Subprocess at 140003955831040 with name clamav_update in state STARTING> (stdout)>
2023-10-12 16:09:27,136 DEBG fd 9 closed, stopped monitoring <POutputDispatcher at 140003956305360 for <Subprocess at 140003955831040 with name clamav_update in state STARTING> (stderr)>
2023-10-12 16:09:27,136 INFO exited: clamav_update (exit status 0; expected)
2023-10-12 16:09:27,136 DEBG received SIGCHLD indicating a child quit
2023-10-12 16:09:27,136 DEBG 'nginx-0' stderr output:
nginx: [alert] could not open error log file: open() "/var/log/nginx/error.log" failed (13: Permission denied)

2023-10-12 16:09:27,137 DEBG 'smokescreen' stderr output:
{"level":"info","msg":"starting","time":"2023-10-12T16:09:27Z"}

2023-10-12 16:09:27,137 DEBG fd 14 closed, stopped monitoring <POutputDispatcher at 140003956305600 for <Subprocess at 140003956294848 with name nginx-0 in state STARTING> (stderr)>
2023-10-12 16:09:28,138 INFO success: nginx-0 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-10-12 16:09:28,138 INFO success: smokescreen entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-10-12 16:09:28,138 INFO success: uvicorn-0 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-10-12 16:09:28,138 INFO success: uvicorn-1 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-10-12 16:09:37,175 DEBG 'uvicorn-1' stderr output:
INFO:     Started server process [145]
INFO:     Waiting for application startup.

2023-10-12 16:09:37,175 DEBG 'uvicorn-1' stderr output:
INFO:     ASGI 'lifespan' protocol appears unsupported.
INFO:     Application startup complete.

2023-10-12 16:09:37,175 DEBG 'uvicorn-1' stderr output:
INFO:     Uvicorn running on socket /tmp/uvicorn.sock (Press CTRL+C to quit)

2023-10-12 16:09:37,204 DEBG 'uvicorn-0' stderr output:
INFO:     Started server process [131]
INFO:     Waiting for application startup.

2023-10-12 16:09:37,204 DEBG 'uvicorn-0' stderr output:
INFO:     ASGI 'lifespan' protocol appears unsupported.
INFO:     Application startup complete.

2023-10-12 16:09:37,204 DEBG 'uvicorn-0' stderr output:
INFO:     Uvicorn running on socket /tmp/uvicorn.sock (Press CTRL+C to quit)

2023-10-12 16:09:37,338 DEBG 'uvicorn-1' stderr output:
[2023-10-12 16:09:37,338] WARNING django.request: Unauthorized: /api/events

2023-10-12 16:09:37,338 DEBG 'uvicorn-1' stdout output:
INFO:     10.215.197.219:0 - "POST /api/events?org= HTTP/1.0" 401 Unauthorized

2023-10-12 16:09:37,413 DEBG 'uvicorn-1' stderr output:
[2023-10-12 16:09:37,411] ERROR health-check: unknown error: 500 Server Error: Internal Server Error for url: http://opa:8181/health?bundles
Traceback (most recent call last):
  File "/home/django/cvat/apps/health/backends.py", line 25, in check_status
    response.raise_for_status()
  File "/opt/venv/lib/python3.10/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http://opa:8181/health?bundles

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/venv/lib/python3.10/site-packages/health_check/backends.py", line 30, in run_check
    self.check_status()
  File "/home/django/cvat/apps/health/backends.py", line 27, in check_status
    raise HealthCheckException(str(e))
health_check.exceptions.HealthCheckException: unknown error: 500 Server Error: Internal Server Error for url: http://opa:8181/health?bundles

2023-10-12 16:09:37,460 DEBG 'uvicorn-1' stderr output:
[2023-10-12 16:09:37,460] ERROR django.request: Internal Server Error: /api/server/health/

2023-10-12 16:09:37,460 DEBG 'uvicorn-1' stdout output:
INFO:     10.215.197.219:0 - "GET /api/server/health/?format=json&org= HTTP/1.0" 500 Internal Server Error

2023-10-12 16:09:40,662 DEBG 'uvicorn-0' stderr output:
[2023-10-12 16:09:40,661] ERROR health-check: unknown error: 500 Server Error: Internal Server Error for url: http://opa:8181/health?bundles
Traceback (most recent call last):
  File "/home/django/cvat/apps/health/backends.py", line 25, in check_status
    response.raise_for_status()
  File "/opt/venv/lib/python3.10/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http://opa:8181/health?bundles

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/venv/lib/python3.10/site-packages/health_check/backends.py", line 30, in run_check
    self.check_status()
  File "/home/django/cvat/apps/health/backends.py", line 27, in check_status
    raise HealthCheckException(str(e))
health_check.exceptions.HealthCheckException: unknown error: 500 Server Error: Internal Server Error for url: http://opa:8181/health?bundles

2023-10-12 16:09:40,700 DEBG 'uvicorn-0' stderr output:
[2023-10-12 16:09:40,700] ERROR django.request: Internal Server Error: /api/server/health/

2023-10-12 16:09:40,700 DEBG 'uvicorn-0' stdout output:
INFO:     10.215.197.219:0 - "GET /api/server/health/?format=json&org= HTTP/1.0" 500 Internal Server Error

2023-10-12 16:09:43,791 DEBG 'uvicorn-0' stderr output:
[2023-10-12 16:09:43,790] ERROR health-check: unknown error: 500 Server Error: Internal Server Error for url: http://opa:8181/health?bundles
Traceback (most recent call last):
  File "/home/django/cvat/apps/health/backends.py", line 25, in check_status
    response.raise_for_status()
  File "/opt/venv/lib/python3.10/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http://opa:8181/health?bundles

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/venv/lib/python3.10/site-packages/health_check/backends.py", line 30, in run_check
    self.check_status()
  File "/home/django/cvat/apps/health/backends.py", line 27, in check_status
    raise HealthCheckException(str(e))
health_check.exceptions.HealthCheckException: unknown error: 500 Server Error: Internal Server Error for url: http://opa:8181/health?bundles

2023-10-12 16:09:43,815 DEBG 'uvicorn-0' stderr output:
[2023-10-12 16:09:43,814] ERROR django.request: Internal Server Error: /api/server/health/

2023-10-12 16:09:43,815 DEBG 'uvicorn-0' stdout output:
INFO:     10.215.197.219:0 - "GET /api/server/health/?format=json&org= HTTP/1.0" 500 Internal Server Error

2023-10-12 16:09:46,865 DEBG 'uvicorn-0' stderr output:
[2023-10-12 16:09:46,863] ERROR health-check: unknown error: 500 Server Error: Internal Server Error for url: http://opa:8181/health?bundles
Traceback (most recent call last):
  File "/home/django/cvat/apps/health/backends.py", line 25, in check_status
    response.raise_for_status()
  File "/opt/venv/lib/python3.10/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http://opa:8181/health?bundles

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/venv/lib/python3.10/site-packages/health_check/backends.py", line 30, in run_check
    self.check_status()
  File "/home/django/cvat/apps/health/backends.py", line 27, in check_status
    raise HealthCheckException(str(e))
health_check.exceptions.HealthCheckException: unknown error: 500 Server Error: Internal Server Error for url: http://opa:8181/health?bundles

2023-10-12 16:09:46,893 DEBG 'uvicorn-0' stderr output:
[2023-10-12 16:09:46,893] ERROR django.request: Internal Server Error: /api/server/health/

2023-10-12 16:09:46,894 DEBG 'uvicorn-0' stdout output:
INFO:     10.215.197.219:0 - "GET /api/server/health/?format=json&org= HTTP/1.0" 500 Internal Server Error

2023-10-12 16:09:49,977 DEBG 'uvicorn-0' stderr output:
[2023-10-12 16:09:49,976] ERROR health-check: unknown error: 500 Server Error: Internal Server Error for url: http://opa:8181/health?bundles
Traceback (most recent call last):
  File "/home/django/cvat/apps/health/backends.py", line 25, in check_status
    response.raise_for_status()
  File "/opt/venv/lib/python3.10/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http://opa:8181/health?bundles

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/venv/lib/python3.10/site-packages/health_check/backends.py", line 30, in run_check
    self.check_status()
  File "/home/django/cvat/apps/health/backends.py", line 27, in check_status
    raise HealthCheckException(str(e))
health_check.exceptions.HealthCheckException: unknown error: 500 Server Error: Internal Server Error for url: http://opa:8181/health?bundles

2023-10-12 16:09:50,003 DEBG 'uvicorn-0' stderr output:
[2023-10-12 16:09:50,003] ERROR django.request: Internal Server Error: /api/server/health/

2023-10-12 16:09:50,003 DEBG 'uvicorn-0' stdout output:
INFO:     10.215.197.219:0 - "GET /api/server/health/?format=json&org= HTTP/1.0" 500 Internal Server Error

2023-10-12 16:09:53,051 DEBG 'uvicorn-0' stderr output:
[2023-10-12 16:09:53,050] ERROR health-check: unknown error: 500 Server Error: Internal Server Error for url: http://opa:8181/health?bundles
Traceback (most recent call last):
  File "/home/django/cvat/apps/health/backends.py", line 25, in check_status
    response.raise_for_status()
  File "/opt/venv/lib/python3.10/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http://opa:8181/health?bundles

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/venv/lib/python3.10/site-packages/health_check/backends.py", line 30, in run_check
    self.check_status()
  File "/home/django/cvat/apps/health/backends.py", line 27, in check_status
    raise HealthCheckException(str(e))
health_check.exceptions.HealthCheckException: unknown error: 500 Server Error: Internal Server Error for url: http://opa:8181/health?bundles

2023-10-12 16:09:53,077 DEBG 'uvicorn-0' stderr output:
[2023-10-12 16:09:53,076] ERROR django.request: Internal Server Error: /api/server/health/

2023-10-12 16:09:53,077 DEBG 'uvicorn-0' stdout output:
INFO:     10.215.197.219:0 - "GET /api/server/health/?format=json&org= HTTP/1.0" 500 Internal Server Error

2023-10-12 16:09:53,409 DEBG 'uvicorn-0' stderr output:
[2023-10-12 16:09:53,408] WARNING django.request: Unauthorized: /api/events

2023-10-12 16:09:53,409 DEBG 'uvicorn-0' stdout output:
INFO:     10.215.197.219:0 - "POST /api/events?org= HTTP/1.0" 401 Unauthorized

2023-10-12 16:09:56,170 DEBG 'uvicorn-0' stderr output:
[2023-10-12 16:09:56,170] ERROR health-check: unknown error: 500 Server Error: Internal Server Error for url: http://opa:8181/health?bundles
Traceback (most recent call last):
  File "/home/django/cvat/apps/health/backends.py", line 25, in check_status
    response.raise_for_status()
  File "/opt/venv/lib/python3.10/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http://opa:8181/health?bundles

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/venv/lib/python3.10/site-packages/health_check/backends.py", line 30, in run_check
    self.check_status()
  File "/home/django/cvat/apps/health/backends.py", line 27, in check_status
    raise HealthCheckException(str(e))
health_check.exceptions.HealthCheckException: unknown error: 500 Server Error: Internal Server Error for url: http://opa:8181/health?bundles

2023-10-12 16:09:56,200 DEBG 'uvicorn-0' stderr output:
[2023-10-12 16:09:56,200] ERROR django.request: Internal Server Error: /api/server/health/

2023-10-12 16:09:56,200 DEBG 'uvicorn-0' stdout output:
INFO:     10.215.197.219:0 - "GET /api/server/health/?format=json&org= HTTP/1.0" 500 Internal Server Error

2023-10-12 16:09:59,254 DEBG 'uvicorn-0' stderr output:
[2023-10-12 16:09:59,254] ERROR health-check: unknown error: 500 Server Error: Internal Server Error for url: http://opa:8181/health?bundles
Traceback (most recent call last):
  File "/home/django/cvat/apps/health/backends.py", line 25, in check_status
    response.raise_for_status()
  File "/opt/venv/lib/python3.10/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http://opa:8181/health?bundles

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/venv/lib/python3.10/site-packages/health_check/backends.py", line 30, in run_check
    self.check_status()
  File "/home/django/cvat/apps/health/backends.py", line 27, in check_status
    raise HealthCheckException(str(e))
health_check.exceptions.HealthCheckException: unknown error: 500 Server Error: Internal Server Error for url: http://opa:8181/health?bundles

2023-10-12 16:09:59,288 DEBG 'uvicorn-0' stderr output:
[2023-10-12 16:09:59,288] ERROR django.request: Internal Server Error: /api/server/health/

2023-10-12 16:09:59,289 DEBG 'uvicorn-0' stdout output:
INFO:     10.215.197.219:0 - "GET /api/server/health/?format=json&org= HTTP/1.0" 500 Internal Server Error

2023-10-12 16:10:02,362 DEBG 'uvicorn-0' stderr output:
[2023-10-12 16:10:02,361] ERROR health-check: unknown error: 500 Server Error: Internal Server Error for url: http://opa:8181/health?bundles
Traceback (most recent call last):
  File "/home/django/cvat/apps/health/backends.py", line 25, in check_status
    response.raise_for_status()
  File "/opt/venv/lib/python3.10/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http://opa:8181/health?bundles

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/venv/lib/python3.10/site-packages/health_check/backends.py", line 30, in run_check
    self.check_status()
  File "/home/django/cvat/apps/health/backends.py", line 27, in check_status
    raise HealthCheckException(str(e))
health_check.exceptions.HealthCheckException: unknown error: 500 Server Error: Internal Server Error for url: http://opa:8181/health?bundles

2023-10-12 16:10:02,388 DEBG 'uvicorn-0' stderr output:
[2023-10-12 16:10:02,388] ERROR django.request: Internal Server Error: /api/server/health/

2023-10-12 16:10:02,388 DEBG 'uvicorn-0' stdout output:
INFO:     10.215.197.219:0 - "GET /api/server/health/?format=json&org= HTTP/1.0" 500 Internal Server Error

2023-10-12 16:10:05,436 DEBG 'uvicorn-0' stderr output:
[2023-10-12 16:10:05,435] ERROR health-check: unknown error: 500 Server Error: Internal Server Error for url: http://opa:8181/health?bundles
Traceback (most recent call last):
  File "/home/django/cvat/apps/health/backends.py", line 25, in check_status
    response.raise_for_status()
  File "/opt/venv/lib/python3.10/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http://opa:8181/health?bundles

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/venv/lib/python3.10/site-packages/health_check/backends.py", line 30, in run_check
    self.check_status()
  File "/home/django/cvat/apps/health/backends.py", line 27, in check_status
    raise HealthCheckException(str(e))
health_check.exceptions.HealthCheckException: unknown error: 500 Server Error: Internal Server Error for url: http://opa:8181/health?bundles

2023-10-12 16:10:05,465 DEBG 'uvicorn-0' stderr output:
[2023-10-12 16:10:05,464] ERROR django.request: Internal Server Error: /api/server/health/

2023-10-12 16:10:05,465 DEBG 'uvicorn-0' stdout output:
INFO:     10.215.197.219:0 - "GET /api/server/health/?format=json&org= HTTP/1.0" 500 Internal Server Error

2023-10-12 16:10:08,545 DEBG 'uvicorn-0' stderr output:
[2023-10-12 16:10:08,545] ERROR health-check: unknown error: 500 Server Error: Internal Server Error for url: http://opa:8181/health?bundles
Traceback (most recent call last):
  File "/home/django/cvat/apps/health/backends.py", line 25, in check_status
    response.raise_for_status()
  File "/opt/venv/lib/python3.10/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http://opa:8181/health?bundles

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/venv/lib/python3.10/site-packages/health_check/backends.py", line 30, in run_check
    self.check_status()
  File "/home/django/cvat/apps/health/backends.py", line 27, in check_status
    raise HealthCheckException(str(e))
health_check.exceptions.HealthCheckException: unknown error: 500 Server Error: Internal Server Error for url: http://opa:8181/health?bundles

2023-10-12 16:10:08,576 DEBG 'uvicorn-0' stderr output:
[2023-10-12 16:10:08,576] ERROR django.request: Internal Server Error: /api/server/health/

2023-10-12 16:10:08,577 DEBG 'uvicorn-0' stdout output:
INFO:     10.215.197.219:0 - "GET /api/server/health/?format=json&org= HTTP/1.0" 500 Internal Server Error

cvat_opa logs:

{"level":"error","msg":"Bundle load failed: server replied with Internal Server Error","name":"cvat","plugin":"bundle","time":"2023-10-12T16:14:00Z"}
{"level":"error","msg":"Bundle load failed: server replied with Internal Server Error","name":"cvat","plugin":"bundle","time":"2023-10-12T16:14:00Z"}
{"level":"error","msg":"Bundle load failed: server replied with Internal Server Error","name":"cvat","plugin":"bundle","time":"2023-10-12T16:14:00Z"}
{"level":"error","msg":"Bundle load failed: server replied with Internal Server Error","name":"cvat","plugin":"bundle","time":"2023-10-12T16:14:01Z"}
{"level":"error","msg":"Bundle load failed: server replied with Internal Server Error","name":"cvat","plugin":"bundle","time":"2023-10-12T16:14:01Z"}
{"level":"error","msg":"Bundle load failed: server replied with Internal Server Error","name":"cvat","plugin":"bundle","time":"2023-10-12T16:14:02Z"}
{"level":"error","msg":"Bundle load failed: server replied with Internal Server Error","name":"cvat","plugin":"bundle","time":"2023-10-12T16:14:03Z"}
{"level":"error","msg":"Bundle load failed: server replied with Internal Server Error","name":"cvat","plugin":"bundle","time":"2023-10-12T16:14:04Z"}
{"level":"error","msg":"Bundle load failed: server replied with Internal Server Error","name":"cvat","plugin":"bundle","time":"2023-10-12T16:14:07Z"}
{"level":"error","msg":"Bundle load failed: server replied with Internal Server Error","name":"cvat","plugin":"bundle","time":"2023-10-12T16:14:11Z"}
{"level":"error","msg":"Bundle load failed: server replied with Internal Server Error","name":"cvat","plugin":"bundle","time":"2023-10-12T16:14:17Z"}
stevezkw1998 commented 11 months ago

On my side, docker exec -t cvat_server python manage.py health_check just keep stucking with out any response, not even raise exception

yunxuo commented 8 months ago

same! !!

silence-de commented 7 months ago

same error! any updates please? in my case, i am behind a corporate proxy settings, i think i have set them correctly in docker-compose.yml file. but still encounter this error

cloudinwind commented 6 months ago

same error!

the output of docker exec -t cvat_server python manage.py health_check command:

yzh@user:~$ sudo docker exec -t cvat_server python manage.py health_check
[2024-05-10 13:36:28,552] ERROR health-check: unknown error: 500 Server Error: Internal Server Error for url: http://opa:8181/health?bundles
Traceback (most recent call last):
  File "/home/django/cvat/apps/health/backends.py", line 22, in check_status
    response.raise_for_status()
  File "/opt/venv/lib/python3.10/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http://opa:8181/health?bundles

During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/opt/venv/lib/python3.10/site-packages/health_check/backends.py", line 30, in run_check
    self.check_status()
  File "/home/django/cvat/apps/health/backends.py", line 24, in check_status
    raise HealthCheckException(str(e))
health_check.exceptions.HealthCheckException: unknown error: 500 Server Error: Internal Server Error for url: http://opa:8181/health?bundles
[2024-05-10 13:36:28,647] ERROR health-check: unexpected result: Connection Error
Traceback (most recent call last):
  File "/opt/venv/lib/python3.10/site-packages/redis/connection.py", line 698, in connect
    sock = self.retry.call_with_retry(
  File "/opt/venv/lib/python3.10/site-packages/redis/retry.py", line 46, in call_with_retry
    return do()
  File "/opt/venv/lib/python3.10/site-packages/redis/connection.py", line 699, in <lambda>
    lambda: self._connect(), lambda error: self.disconnect(error)
  File "/opt/venv/lib/python3.10/site-packages/redis/connection.py", line 987, in _connect
    raise err
  File "/opt/venv/lib/python3.10/site-packages/redis/connection.py", line 975, in _connect
    sock.connect(socket_address)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/venv/lib/python3.10/site-packages/health_check/cache/backends.py", line 35, in check_status
    cache.set(self.cache_key, "itworks")
  File "/opt/venv/lib/python3.10/site-packages/django/core/cache/backends/redis.py", line 191, in set
    self._cache.set(key, value, self.get_backend_timeout(timeout))
  File "/opt/venv/lib/python3.10/site-packages/django/core/cache/backends/redis.py", line 108, in set
    client.set(key, value, ex=timeout)
  File "/opt/venv/lib/python3.10/site-packages/redis/commands/core.py", line 2302, in set
    return self.execute_command("SET", *pieces, **options)
  File "/opt/venv/lib/python3.10/site-packages/redis/client.py", line 1255, in execute_command
    conn = self.connection or pool.get_connection(command_name, **options)
  File "/opt/venv/lib/python3.10/site-packages/redis/connection.py", line 1442, in get_connection
    connection.connect()
  File "/opt/venv/lib/python3.10/site-packages/redis/connection.py", line 704, in connect
    raise ConnectionError(self._error_message(e))
redis.exceptions.ConnectionError: Error 111 connecting to localhost:6666. Connection refused.
Cache backend: default   ... working
Cache backend: media     ... unexpected result: Connection Error
DatabaseBackend          ... working
DiskUsage                ... working
MemoryUsage              ... working
OPAHealthCheck           ... unknown error: 500 Server Error: Internal Server Error for url: http://opa:8181/health?bundles

and the docker logs cvat_server :


2024-05-10 13:31:57,972 DEBG 'uvicorn-0' stderr output:
[2024-05-10 13:31:57,971] ERROR health-check: unknown error: 500 Server Error: Internal Server Error for url: http://opa:8181/health?bundles
Traceback (most recent call last):
  File "/home/django/cvat/apps/health/backends.py", line 22, in check_status
    response.raise_for_status()
  File "/opt/venv/lib/python3.10/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http://opa:8181/health?bundles

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/venv/lib/python3.10/site-packages/health_check/backends.py", line 30, in run_check
    self.check_status()
  File "/home/django/cvat/apps/health/backends.py", line 24, in check_status
    raise HealthCheckException(str(e))
health_check.exceptions.HealthCheckException: unknown error: 500 Server Error: Internal Server Error for url: http://opa:8181/health?bundles

2024-05-10 13:31:57,980 DEBG 'uvicorn-0' stderr output:
[2024-05-10 13:31:57,980] ERROR django.request: Internal Server Error: /api/server/health/

2024-05-10 13:31:57,981 DEBG 'uvicorn-0' stdout output:
INFO:     192.168.7.172:0 - "GET /api/server/health/?format=json&org= HTTP/1.0" 500 Internal Server Error

and the logs docker logs cvat_opa:

yzh@user:~$ sudo docker logs cvat_opa
{"level":"error","msg":"Bundle load failed: request failed: Get \"http://localhost:8080/api/auth/rules\": dial tcp [::1]:8080: connect: connection refused","name":"cvat","plugin":"bundle","time":"2024-05-10T13:30:09Z"}
{"level":"error","msg":"Bundle load failed: request failed: Get \"http://localhost:8080/api/auth/rules\": dial tcp [::1]:8080: connect: connection refused","name":"cvat","plugin":"bundle","time":"2024-05-10T13:30:09Z"}
{"level":"error","msg":"Bundle load failed: request failed: Get \"http://localhost:8080/api/auth/rules\": dial tcp [::1]:8080: connect: connection refused","name":"cvat","plugin":"bundle","time":"2024-05-10T13:30:09Z"}

How to solve the issue?

tsune10696 commented 1 month ago

@cloudinwind @azhavoro

How to solve the issue?

↑ Has this problem been resolved? If it has been resolved, I would like to know how it was resolved.