search

cvat-ai / cvat

Annotate better with CVAT, the industry-leading data engine for machine learning. Used and trusted by teams at any scale, for data of any scale.
https://cvat.ai
MIT License
12.44k stars 2.99k forks source link

Error with Helm Chart | PostgreSql connection #6872

Closed hadilou closed 12 months ago

hadilou commented 1 year ago

My actions before raising this issue

Steps to Reproduce (for bugs)

  1. I followed the documentation on how to install with helm https://opencv.github.io/cvat/docs/administration/advanced/k8s_deployment_with_helm/
  2. Ran python manage.py health_check inside cvat-server

Expected Behaviour

I was expecting the health check to complete without errors instead got postgresql wrong password error.

Current Behaviour

[2023-09-17 13:40:09,576] ERROR health-check: unavailable: Database error
Traceback (most recent call last):
  File "/opt/venv/lib/python3.10/site-packages/django/db/backends/base/base.py", line 289, in ensure_connection
    self.connect()
  File "/opt/venv/lib/python3.10/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/opt/venv/lib/python3.10/site-packages/django/db/backends/base/base.py", line 270, in connect
    self.connection = self.get_new_connection(conn_params)
  File "/opt/venv/lib/python3.10/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/opt/venv/lib/python3.10/site-packages/django/db/backends/postgresql/base.py", line 275, in get_new_connection
    connection = self.Database.connect(**conn_params)
  File "/opt/venv/lib/python3.10/site-packages/psycopg2/__init__.py", line 122, in connect
    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: connection to server at "cvat-release-postgresql" (10.99.1.87), port 5432 failed: FATAL:  password authentication failed for user "cvat"

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/venv/lib/python3.10/site-packages/health_check/db/backends.py", line 12, in check_status
    obj = TestModel.objects.create(title="test")
  File "/opt/venv/lib/python3.10/site-packages/django/db/models/manager.py", line 87, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/opt/venv/lib/python3.10/site-packages/django/db/models/query.py", line 658, in create
    obj.save(force_insert=True, using=self.db)
  File "/opt/venv/lib/python3.10/site-packages/django/db/models/base.py", line 814, in save
    self.save_base(
  File "/opt/venv/lib/python3.10/site-packages/django/db/models/base.py", line 877, in save_base
    updated = self._save_table(
  File "/opt/venv/lib/python3.10/site-packages/django/db/models/base.py", line 1020, in _save_table
    results = self._do_insert(
  File "/opt/venv/lib/python3.10/site-packages/django/db/models/base.py", line 1061, in _do_insert
    return manager._insert(
  File "/opt/venv/lib/python3.10/site-packages/django/db/models/manager.py", line 87, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/opt/venv/lib/python3.10/site-packages/django/db/models/query.py", line 1805, in _insert
    return query.get_compiler(using=using).execute_sql(returning_fields)
  File "/opt/venv/lib/python3.10/site-packages/django/db/models/sql/compiler.py", line 1820, in execute_sql
    with self.connection.cursor() as cursor:
  File "/opt/venv/lib/python3.10/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/opt/venv/lib/python3.10/site-packages/django/db/backends/base/base.py", line 330, in cursor
    return self._cursor()
  File "/opt/venv/lib/python3.10/site-packages/django/db/backends/base/base.py", line 306, in _cursor
    self.ensure_connection()
  File "/opt/venv/lib/python3.10/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/opt/venv/lib/python3.10/site-packages/django/db/backends/base/base.py", line 288, in ensure_connection
    with self.wrap_database_errors:
  File "/opt/venv/lib/python3.10/site-packages/django/db/utils.py", line 91, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/opt/venv/lib/python3.10/site-packages/django/db/backends/base/base.py", line 289, in ensure_connection
    self.connect()
  File "/opt/venv/lib/python3.10/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/opt/venv/lib/python3.10/site-packages/django/db/backends/base/base.py", line 270, in connect
    self.connection = self.get_new_connection(conn_params)
  File "/opt/venv/lib/python3.10/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/opt/venv/lib/python3.10/site-packages/django/db/backends/postgresql/base.py", line 275, in get_new_connection
    connection = self.Database.connect(**conn_params)
  File "/opt/venv/lib/python3.10/site-packages/psycopg2/__init__.py", line 122, in connect
    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: connection to server at "cvat-release-postgresql" (10.99.1.87), port 5432 failed: FATAL:  password authentication failed for user "cvat"
During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/venv/lib/python3.10/site-packages/health_check/backends.py", line 30, in run_check
    self.check_status()
  File "/opt/venv/lib/python3.10/site-packages/health_check/db/backends.py", line 19, in check_status
    raise ServiceUnavailable("Database error")
health_check.exceptions.ServiceUnavailable: unavailable: Database error
[2023-09-17 13:40:09,577] ERROR health-check: unknown error: 500 Server Error: Internal Server Error for url: http://opa:8181/health?bundles
Traceback (most recent call last):
  File "/home/django/cvat/apps/health/backends.py", line 25, in check_status
    response.raise_for_status()
  File "/opt/venv/lib/python3.10/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http://opa:8181/health?bundles

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/venv/lib/python3.10/site-packages/health_check/backends.py", line 30, in run_check
    self.check_status()
  File "/home/django/cvat/apps/health/backends.py", line 27, in check_status
    raise HealthCheckException(str(e))
health_check.exceptions.HealthCheckException: unknown error: 500 Server Error: Internal Server Error for url: http://opa:8181/health?bundles
[2023-09-17 13:40:09,780] ERROR health-check: unavailable: Database is not ready
Traceback (most recent call last):
  File "/opt/venv/lib/python3.10/site-packages/django/db/backends/base/base.py", line 289, in ensure_connection
    self.connect()
  File "/opt/venv/lib/python3.10/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/opt/venv/lib/python3.10/site-packages/django/db/backends/base/base.py", line 270, in connect
    self.connection = self.get_new_connection(conn_params)
  File "/opt/venv/lib/python3.10/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/opt/venv/lib/python3.10/site-packages/django/db/backends/postgresql/base.py", line 275, in get_new_connection
    connection = self.Database.connect(**conn_params)
  File "/opt/venv/lib/python3.10/site-packages/psycopg2/__init__.py", line 122, in connect
    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: connection to server at "cvat-release-postgresql" (10.99.1.87), port 5432 failed: FATAL:  password authentication failed for user "cvat"

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/venv/lib/python3.10/site-packages/health_check/contrib/migrations/backends.py", line 20, in check_status
    executor = MigrationExecutor(connections[db_alias])
  File "/opt/venv/lib/python3.10/site-packages/django/db/migrations/executor.py", line 18, in __init__
    self.loader = MigrationLoader(self.connection)
  File "/opt/venv/lib/python3.10/site-packages/django/db/migrations/loader.py", line 58, in __init__
    self.build_graph()
  File "/opt/venv/lib/python3.10/site-packages/django/db/migrations/loader.py", line 235, in build_graph
    self.applied_migrations = recorder.applied_migrations()
  File "/opt/venv/lib/python3.10/site-packages/django/db/migrations/recorder.py", line 81, in applied_migrations
    if self.has_table():
  File "/opt/venv/lib/python3.10/site-packages/django/db/migrations/recorder.py", line 57, in has_table
    with self.connection.cursor() as cursor:
  File "/opt/venv/lib/python3.10/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/opt/venv/lib/python3.10/site-packages/django/db/backends/base/base.py", line 330, in cursor
    return self._cursor()
  File "/opt/venv/lib/python3.10/site-packages/django/db/backends/base/base.py", line 306, in _cursor
    self.ensure_connection()
  File "/opt/venv/lib/python3.10/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/opt/venv/lib/python3.10/site-packages/django/db/backends/base/base.py", line 288, in ensure_connection
    with self.wrap_database_errors:
  File "/opt/venv/lib/python3.10/site-packages/django/db/utils.py", line 91, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/opt/venv/lib/python3.10/site-packages/django/db/backends/base/base.py", line 289, in ensure_connection
    self.connect()
  File "/opt/venv/lib/python3.10/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/opt/venv/lib/python3.10/site-packages/django/db/backends/base/base.py", line 270, in connect
    self.connection = self.get_new_connection(conn_params)
  File "/opt/venv/lib/python3.10/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/opt/venv/lib/python3.10/site-packages/django/db/backends/postgresql/base.py", line 275, in get_new_connection
    connection = self.Database.connect(**conn_params)
  File "/opt/venv/lib/python3.10/site-packages/psycopg2/__init__.py", line 122, in connect
    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: connection to server at "cvat-release-postgresql" (10.99.1.87), port 5432 failed: FATAL:  password authentication failed for user "cvat"

Cache backend: default   ... working
Cache backend: media     ... working
DatabaseBackend          ... unavailable: Database error
DiskUsage                ... working
MemoryUsage              ... working
MigrationsHealthCheck    ... unavailable: Database is not ready
OPAHealthCheck           ... unknown error: 500 Server Error: Internal Server Error for url: http://opa:8181/health?bundles

I saw the same error in the logs of the cvat-server pod.

Possible Solution

Context

I am using minikube on local like in the guide above, I made no override to values.yml file. My guess is that the error is related to the version of postgresql needing a different way of authentication? The helm chart needs to be fixed.

azhavoro commented 1 year ago

Which version are you using?

hadilou commented 1 year ago

Here is the output of git log -1 

git log -1
commit 657e361227e8c2072b74af6821f12909a158a174 (HEAD -> develop, origin/develop, origin/HEAD)
Author: Boris Sekachev <boris.sekachev@yandex.ru>
Date:   Fri Sep 15 16:56:21 2023 +0300

    [Refactoring] Removed custom exif rotation method (#6835)
hadilou commented 1 year ago

Sorry to ping but is there any luck with this? @azhavoro

azhavoro commented 1 year ago

Please provide step by step commands to reproduce the issue