issue with waldur-queue and waldur-db

tyagisanjeev1211 commented 1 week ago

I have recently build solution using composure and getting error for Postgres database is not reachable

2024-10-24 14:33:41,196 WARNING django-csp could not be found, not updating Content-Security-Policy. Please make sure CSP is configured. This can be done by your reverse proxy, django-csp or a custom CSP handler via SAML_CSP_HANDLER. See https://djangosaml2.readthedocs.io/contents/security.html#content-security-policy for more information. This warning can be disabled by settingSAML_CSP_HANDLER=''in your settings. Checking Waldur MasterMind services... Database PostgreSQL connection [**ERROR**] Task runners (Celery workers) [OK] Queue and cache server (Redis) connection [OK] and File "/usr/local/lib/python3.11/site-packages/kombu/connection.py", line 476, in _reraise_as_library_errors raise ConnectionError(str(exc)) from exc kombu.exceptions.OperationalError: Error -2 connecting to **_waldur-queue:6379_**. Name or service not known.

and while creating user File "/usr/local/lib/python3.11/site-packages/psycopg2/__init__.py", line 122, in connect conn = _connect(dsn, connection_factory=connection_factory, **kwasync) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ django.db.utils.OperationalError: **_could not translate host name "waldur-db" to address_**: Name or service not known

livenson commented 1 week ago

It seems that the PSQL container is not up. Can you please check the logs of the container?

tyagisanjeev1211 commented 1 week ago

DB container is working fine (showing healthy as well). however waldur-queue is not sable and showing exiting frequently.

livenson commented 1 week ago

ok, then what is the error in waldur-queue?

tyagisanjeev1211 commented 1 week ago

it is working now.

tyagisanjeev1211 commented 2 days ago

I am facing same issue again, Please find logs of queue container and minstermind-worker container as below:

Logs of waldur-queue

redis 11:14:08.48 INFO  ==>
redis 11:14:08.48 INFO  ==> Welcome to the Bitnami redis container
redis 11:14:08.48 INFO  ==> Subscribe to project updates by watching https://github.com/bitnami/containers
redis 11:14:08.48 INFO  ==> Submit issues and feature requests at https://github.com/bitnami/containers/issues
redis 11:14:08.48 INFO  ==> Upgrade to Tanzu Application Catalog for production environments to access custom-configured and pre-packaged software components. Gain enhanced features, including Software Bill of Materials (SBOM), CVE scan result reports, and VEX documents. To learn more, visit https://bitnami.com/enterprise
redis 11:14:08.49 INFO  ==>
redis 11:14:08.49 INFO  ==> ** Starting Redis setup **
redis 11:14:08.51 INFO  ==> Initializing Redis
redis 11:14:08.52 INFO  ==> Setting Redis config file
redis 11:14:08.55 INFO  ==> ** Redis setup finished! **

redis 11:14:08.56 INFO  ==> ** Starting Redis **
1:C 30 Oct 2024 11:14:08.577 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
1:C 30 Oct 2024 11:14:08.577 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:C 30 Oct 2024 11:14:08.577 * Redis version=7.4.1, bits=64, commit=00000000, modified=1, pid=1, just started
1:C 30 Oct 2024 11:14:08.577 * Configuration loaded
1:M 30 Oct 2024 11:14:08.578 * monotonic clock: POSIX clock_gettime
1:M 30 Oct 2024 11:14:08.578 * Running mode=standalone, port=6379.
1:M 30 Oct 2024 11:14:08.579 * Server initialized
1:M 30 Oct 2024 11:14:08.580 * Creating AOF base file appendonly.aof.1.base.rdb on server start
1:M 30 Oct 2024 11:14:08.582 * Creating AOF incr file appendonly.aof.1.incr.aof on server start
1:M 30 Oct 2024 11:14:08.582 * Ready to accept connections tcp

Logs from waldur-mastermind-worker

[2024-10-30 13:30:47,222: ERROR/MainProcess] consumer: Cannot connect to redis://:**@waldur-queue:6379/1: Error -2 connecting to waldur-queue:6379. Name or service not known..
Trying again in 32.00 seconds... (16/100)

[2024-10-30 13:31:19,312: ERROR/MainProcess] consumer: Cannot connect to redis://:**@waldur-queue:6379/1: Error -2 connecting to waldur-queue:6379. Name or service not known..
Trying again in 32.00 seconds... (16/100)

_CONTAINER ID  IMAGE                                       COMMAND               CREATED      STATUS                     PORTS                                     NAMES_
d0ae859c5b38  docker.io/library/postgres:12               postgres              2 hours ago  Up 2 hours (healthy)                                                 waldur-db
b7ddf2a1a7e8  docker.io/bitnami/redis:latest              /opt/bitnami/scri...  2 hours ago  Up 2 hours                                                           waldur-queue
6b5656d44eed  docker.io/library/postgres:10               postgres              2 hours ago  Up 2 hours (healthy)                                                 keycloak-db
095029fccf06  docker.io/umputun/docker-logger:latest      /srv/docker-logge...  2 hours ago  Exited (0) 1 second ago                                              logger
68e8573346b2  docker.io/opennode/waldur-mastermind:7.0.6  init-whitelabelin...  2 hours ago  Exited (1) 2 hours ago                                               waldur-mastermind-whitelabeling-init
097cf9e26fcf  quay.io/keycloak/keycloak:25.0              start --import-re...  2 hours ago  Up 5 seconds                                                         keycloak
beba9b7a03b4  docker.io/opennode/waldur-mastermind:7.0.6  initdb                2 hours ago  Exited (1) 2 hours ago                                               waldur-mastermind-db-migration
**9a71cbacb3d4  docker.io/opennode/waldur-mastermind:7.0.6  worker                2 hours ago  Up 29 minutes (unhealthy)                                            waldur-mastermind-worker**
d147eb28c2fe  docker.io/opennode/waldur-mastermind:7.0.6  beat                  2 hours ago  Up 33 minutes                                                        waldur-mastermind-beat
eba18fc6e19b  docker.io/opennode/waldur-mastermind:7.0.6  mastermind            2 hours ago  Up 2 hours                                                           waldur-mastermind-api
f43a764ad194  docker.io/opennode/waldur-homeport:7.0.6    /entrypoint.sh        2 hours ago  Up 2 hours (healthy)                                                 waldur-homeport
ea15ac314b8f  docker.io/library/caddy:2                   caddy run --confi...  2 hours ago  Up 2 hours                 0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp  caddy
exit code: 0

livenson commented 2 days ago

Can you please show logs of initdb container? docker logs initdb

tyagisanjeev1211 commented 2 days ago

logs for waldur-mastermind-db-migration

INFO: Welcome to Waldur Mastermind!
INFO: Creating user waldur 984:984
INFO: Create logging directory
'/etc/waldur/id_rsa' -> '/var/lib/waldur/id_rsa'
INFO: Spawning initdb
INFO: Initializing database...
/usr/local/lib/python3.11/site-packages/glanceclient/common/http.py:171: SyntaxWarning: "is not" with a literal. Did you mean "!="?
  if kwargs.get('cacert', None) is not '':
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/django/db/backends/base/base.py", line 289, in ensure_connection
    self.connect()
  File "/usr/local/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/backends/base/base.py", line 270, in connect
    self.connection = self.get_new_connection(conn_params)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/backends/postgresql/base.py", line 275, in get_new_connection
    connection = self.Database.connect(**conn_params)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/psycopg2/__init__.py", line 122, in connect
    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
psycopg2.OperationalError: could not translate host name "waldur-db" to address: Name or service not known

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/bin/waldur", line 6, in <module>
    sys.exit(main())
             ^^^^^^
  File "/usr/src/waldur/src/waldur_core/server/manage.py", line 10, in main
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.11/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.11/site-packages/django/core/management/__init__.py", line 436, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.11/site-packages/django/core/management/base.py", line 412, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.11/site-packages/django/core/management/base.py", line 458, in execute
    output = self.handle(*args, **options)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/core/management/base.py", line 106, in wrapper
    res = handle_func(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/core/management/commands/migrate.py", line 117, in handle
    executor = MigrationExecutor(connection, self.migration_progress_callback)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/migrations/executor.py", line 18, in __init__
    self.loader = MigrationLoader(self.connection)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/migrations/loader.py", line 58, in __init__
    self.build_graph()
  File "/usr/local/lib/python3.11/site-packages/django/db/migrations/loader.py", line 235, in build_graph
    self.applied_migrations = recorder.applied_migrations()
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/migrations/recorder.py", line 81, in applied_migrations
    if self.has_table():
       ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/migrations/recorder.py", line 57, in has_table
    with self.connection.cursor() as cursor:
         ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/backends/base/base.py", line 330, in cursor
    return self._cursor()
           ^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/backends/base/base.py", line 306, in _cursor
    self.ensure_connection()
  File "/usr/local/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/backends/base/base.py", line 288, in ensure_connection
    with self.wrap_database_errors:
  File "/usr/local/lib/python3.11/site-packages/django/db/utils.py", line 91, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/usr/local/lib/python3.11/site-packages/django/db/backends/base/base.py", line 289, in ensure_connection
    self.connect()
  File "/usr/local/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/backends/base/base.py", line 270, in connect
    self.connection = self.get_new_connection(conn_params)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/backends/postgresql/base.py", line 275, in get_new_connection
    connection = self.Database.connect(**conn_params)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/psycopg2/__init__.py", line 122, in connect
    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
django.db.utils.OperationalError: could not translate host name "waldur-db" to address: Name or service not known

and containers:

CONTAINER ID  IMAGE                                       COMMAND               CREATED             STATUS                         PORTS                                     NAMES
44e3d34871fe  docker.io/library/postgres:12               postgres              2 minutes ago       Up 2 minutes (healthy)                                                   waldur-db
29b58286b43c  docker.io/bitnami/redis:latest              /opt/bitnami/scri...  2 minutes ago       Up 2 minutes                                                             waldur-queue
22095297b90a  docker.io/library/postgres:10               postgres              2 minutes ago       Up 2 minutes (healthy)                                                   keycloak-db
731f5812c85f  docker.io/umputun/docker-logger:latest      /srv/docker-logge...  2 minutes ago       Up Less than a second                                                    logger
64b039788205  docker.io/opennode/waldur-mastermind:7.0.7  init-whitelabelin...  2 minutes ago       Exited (1) 28 seconds ago                                                waldur-mastermind-whitelabeling-init
c7b307f93ccc  quay.io/keycloak/keycloak:25.0              start --import-re...  2 minutes ago       Up 4 seconds                                                             keycloak
d2b2148c42a0  docker.io/opennode/waldur-mastermind:7.0.7  initdb                2 minutes ago       Exited (1) About a minute ago                                            waldur-mastermind-db-migration
ced48f8f1dcd  docker.io/opennode/waldur-mastermind:7.0.7  worker                2 minutes ago       Up 2 minutes (unhealthy)                                                 waldur-mastermind-worker
9abe31c23be7  docker.io/opennode/waldur-mastermind:7.0.7  beat                  2 minutes ago       Up 2 minutes                                                             waldur-mastermind-beat
8bb17520642b  docker.io/opennode/waldur-mastermind:7.0.7  mastermind            2 minutes ago       Up 2 minutes                                                             waldur-mastermind-api
1ba66d745cf8  docker.io/opennode/waldur-homeport:7.0.7    /entrypoint.sh        About a minute ago  Up About a minute (healthy)                                              waldur-homeport
2086dbef45c2  docker.io/library/caddy:2                   caddy run --confi...  About a minute ago  Up About a minute              0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp  caddy

livenson commented 1 day ago

Symptoms are very similar to the issue described in https://github.com/docker/compose/issues/9513 . Basically for some reason service names are not resolvable. Could you please consider upgrading your docker version?

waldur / waldur-docker-compose