search

jdelic / django-dbconn-retry

Patches Django to reconnect on a failed database connection once before failing. Helping with running Django ORM through HAProxy, for example.
BSD 3-Clause "New" or "Revised" License
76 stars 25 forks source link

django.db.utils.OperationalError when trying to apply migrations while db is still launching #4

Closed malinoff closed 6 years ago

malinoff commented 6 years ago

Hi, thanks for the project. I'm trying to use it instead of wait-for-it script with docker-compose.

runserver doesn't fail because of an unavailable db - and that's right! However, when trying to apply migrations, the command fails:

migrator_1  | Reconnecting to the database didn't help could not connect to server: Connection refused
migrator_1  |   Is the server running on host "db" (172.22.0.4) and accepting
migrator_1  |   TCP/IP connections on port 5432?
migrator_1  |
migrator_1  | Traceback (most recent call last):
migrator_1  |   File "/usr/local/lib/python3.6/site-packages/django_dbconn_retry/__init__.py", line 53, in ensure_connection_with_retries
migrator_1  |     self.connect()
migrator_1  |   File "/usr/local/lib/python3.6/site-packages/django/db/backends/base/base.py", line 194, in connect
migrator_1  |     self.connection = self.get_new_connection(conn_params)
migrator_1  |   File "/usr/local/lib/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
migrator_1  |     connection = Database.connect(**conn_params)
migrator_1  |   File "/usr/local/lib/python3.6/site-packages/psycopg2/__init__.py", line 130, in connect
migrator_1  |     conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
migrator_1  | psycopg2.OperationalError: could not connect to server: Connection refused
migrator_1  |   Is the server running on host "db" (172.22.0.4) and accepting
migrator_1  |   TCP/IP connections on port 5432?
migrator_1  |
migrator_1  |
migrator_1  | During handling of the above exception, another exception occurred:
migrator_1  |
migrator_1  | Traceback (most recent call last):
migrator_1  |   File "/usr/local/lib/python3.6/site-packages/django_dbconn_retry/__init__.py", line 53, in ensure_connection_with_retries
migrator_1  |     self.connect()
migrator_1  |   File "/usr/local/lib/python3.6/site-packages/django/db/backends/base/base.py", line 194, in connect
migrator_1  |     self.connection = self.get_new_connection(conn_params)
migrator_1  |   File "/usr/local/lib/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
migrator_1  |     connection = Database.connect(**conn_params)
migrator_1  |   File "/usr/local/lib/python3.6/site-packages/psycopg2/__init__.py", line 130, in connect
migrator_1  |     conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
migrator_1  | psycopg2.OperationalError: could not connect to server: Connection refused
migrator_1  |   Is the server running on host "db" (172.22.0.4) and accepting
migrator_1  |   TCP/IP connections on port 5432?
migrator_1  |
migrator_1  |
migrator_1  | The above exception was the direct cause of the following exception:
migrator_1  |
migrator_1  | Traceback (most recent call last):
migrator_1  |   File "./manage.py", line 15, in <module>
migrator_1  |     execute_from_command_line(sys.argv)
migrator_1  |   File "/usr/local/lib/python3.6/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
migrator_1  |     utility.execute()
migrator_1  |   File "/usr/local/lib/python3.6/site-packages/django/core/management/__init__.py", line 375, in execute
migrator_1  |     self.fetch_command(subcommand).run_from_argv(self.argv)
migrator_1  |   File "/usr/local/lib/python3.6/site-packages/django/core/management/base.py", line 316, in run_from_argv
migrator_1  |     self.execute(*args, **cmd_options)
migrator_1  |   File "/usr/local/lib/python3.6/site-packages/django/core/management/base.py", line 353, in execute
migrator_1  |     output = self.handle(*args, **options)
migrator_1  |   File "/usr/local/lib/python3.6/site-packages/django/core/management/base.py", line 83, in wrapped
migrator_1  |     res = handle_func(*args, **kwargs)
migrator_1  |   File "/usr/local/lib/python3.6/site-packages/django/core/management/commands/migrate.py", line 82, in handle
migrator_1  |     executor = MigrationExecutor(connection, self.migration_progress_callback)
migrator_1  |   File "/usr/local/lib/python3.6/site-packages/django/db/migrations/executor.py", line 18, in __init__
migrator_1  |     self.loader = MigrationLoader(self.connection)
migrator_1  |   File "/usr/local/lib/python3.6/site-packages/django/db/migrations/loader.py", line 49, in __init__
migrator_1  |     self.build_graph()
migrator_1  |   File "/usr/local/lib/python3.6/site-packages/django/db/migrations/loader.py", line 212, in build_graph
migrator_1  |     self.applied_migrations = recorder.applied_migrations()
migrator_1  |   File "/usr/local/lib/python3.6/site-packages/django/db/migrations/recorder.py", line 61, in applied_migrations
migrator_1  |     if self.has_table():
migrator_1  |   File "/usr/local/lib/python3.6/site-packages/django/db/migrations/recorder.py", line 44, in has_table
migrator_1  |     return self.Migration._meta.db_table in self.connection.introspection.table_names(self.connection.cursor())
migrator_1  |   File "/usr/local/lib/python3.6/site-packages/django/db/backends/base/base.py", line 255, in cursor
migrator_1  |     return self._cursor()
migrator_1  |   File "/usr/local/lib/python3.6/site-packages/django/db/backends/base/base.py", line 232, in _cursor
migrator_1  |     self.ensure_connection()
migrator_1  |   File "/usr/local/lib/python3.6/site-packages/django_dbconn_retry/__init__.py", line 71, in ensure_connection_with_retries
migrator_1  |     self.ensure_connection()
migrator_1  |   File "/usr/local/lib/python3.6/site-packages/django_dbconn_retry/__init__.py", line 81, in ensure_connection_with_retries
migrator_1  |     del self._in_connecting
migrator_1  |   File "/usr/local/lib/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
migrator_1  |     raise dj_exc_value.with_traceback(traceback) from exc_value
migrator_1  |   File "/usr/local/lib/python3.6/site-packages/django_dbconn_retry/__init__.py", line 53, in ensure_connection_with_retries
migrator_1  |     self.connect()
migrator_1  |   File "/usr/local/lib/python3.6/site-packages/django/db/backends/base/base.py", line 194, in connect
migrator_1  |     self.connection = self.get_new_connection(conn_params)
migrator_1  |   File "/usr/local/lib/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
migrator_1  |     connection = Database.connect(**conn_params)
migrator_1  |   File "/usr/local/lib/python3.6/site-packages/psycopg2/__init__.py", line 130, in connect
migrator_1  |     conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
migrator_1  | django.db.utils.OperationalError: could not connect to server: Connection refused
migrator_1  |   Is the server running on host "db" (172.22.0.4) and accepting
migrator_1  |   TCP/IP connections on port 5432?

Here is the relevant docker-compose.yml part:

  migrator:
    build: ./backend
    command: ./manage.py migrate
    working_dir: /app
    volumes:
    - ./backend/:/app
    env_file:
    - .env
  db:
    image: postgres:10-alpine
    ports:
      - 5432:5432
jdelic commented 6 years ago

From a plain reading of your stacktrace it seems to me that the PostgreSQL server is unavailable. Please note that django-dbconn-retry will only retry connecting once. The purpose of this project is not to wait for an unavailable database server, but instead allow other tools to, for example, refresh database credentials without interrupting the current request. I use django-dbconn-retry to grab new database credentials from a Hashicorp Vault when my project can't log into the database using the credentials it already knows.

You could patch ensure_connection_with_retries() so that it retries more than once. However, you probably want to introduce some time between retries and this project has problems with threading as it is, so you're on your own there.

tl;dr: the line Reconnecting to the database didn't help could not connect to server: Connection refused points to django-dbconn-retry trying to connect to your PostgreSQL twice and failing both times.

Feel free to reopen this if I misunderstood your issue. Thanks.