search

canonical / maas-ansible-playbook

An Ansible playbook for installing and configuring MAAS
Apache License 2.0
48 stars 37 forks source link

Migrate MAAS database fails on 2/3 hosts #58

Closed samuelallan72 closed 1 year ago

samuelallan72 commented 1 year ago 
TASK [maas_region_controller : Migrate MAAS database] **************************
fatal: [192.168.122.67]: FAILED! => {"changed": false, "cmd": ["maas", "migrate"], "delta": "0:00:03.611762", "end": "2022-12-11 21:41:51.493348", "msg": "non-zero return code", "rc": 1, "start": "2022-12-11 21:41:47.881586", "stderr": "Traceback (most recent call last):\n  File \"/snap/maas/23947/usr/lib/python3/dist-packages/django/db/backends/utils.py\", line 82, in _execute\n    return self.cursor.execute(sql)\npsycopg2.errors.DeadlockDetected: deadlock detected\nDETAIL:  Process 16642 waits for AccessExclusiveLock on object 19776 of class 2620 of database 16385; blocked by process 16641.\nProcess 16641 waits for AccessExclusiveLock on relation 16739 of database 16385; blocked by process 16642.\nHINT:  See server log for query details.\n\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n  File \"/snap/maas/23947/bin/maas-region\", line 8, in <module>\n    sys.exit(run())\n  File \"/snap/maas/23947/lib/python3.8/site-packages/maasserver/region_script.py\", line 77, in run\n    run_django(is_snap, is_devenv)\n  File \"/snap/maas/23947/lib/python3.8/site-packages/maasserver/region_script.py\", line 66, in run_django\n    management.execute_from_command_line()\n  File \"/snap/maas/23947/usr/lib/python3/dist-packages/django/core/management/__init__.py\", line 381, in execute_from_command_line\n    utility.execute()\n  File \"/snap/maas/23947/usr/lib/python3/dist-packages/django/core/management/__init__.py\", line 375, in execute\n    self.fetch_command(subcommand).run_from_argv(self.argv)\n  File \"/snap/maas/23947/usr/lib/python3/dist-packages/django/core/management/base.py\", line 323, in run_from_argv\n    self.execute(*args, **cmd_options)\n  File \"/snap/maas/23947/usr/lib/python3/dist-packages/django/core/management/base.py\", line 364, in execute\n    output = self.handle(*args, **options)\n  File \"/snap/maas/23947/lib/python3.8/site-packages/maasserver/management/commands/dbupgrade.py\", line 122, in handle\n    self._drop_all_triggers(database)\n  File \"/snap/maas/23947/lib/python3.8/site-packages/maasserver/management/commands/dbupgrade.py\", line 77, in _drop_all_triggers\n    cursor.execute(\n  File \"/snap/maas/23947/usr/lib/python3/dist-packages/django/db/backends/utils.py\", line 67, in execute\n    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)\n  File \"/snap/maas/23947/usr/lib/python3/dist-packages/django/db/backends/utils.py\", line 76, in _execute_with_wrappers\n    return executor(sql, params, many, context)\n  File \"/snap/maas/23947/usr/lib/python3/dist-packages/django/db/backends/utils.py\", line 84, in _execute\n    return self.cursor.execute(sql, params)\n  File \"/snap/maas/23947/usr/lib/python3/dist-packages/django/db/utils.py\", line 89, in __exit__\n    raise dj_exc_value.with_traceback(traceback) from exc_value\n  File \"/snap/maas/23947/usr/lib/python3/dist-packages/django/db/backends/utils.py\", line 82, in _execute\n    return self.cursor.execute(sql)\ndjango.db.utils.OperationalError: deadlock detected\nDETAIL:  Process 16642 waits for AccessExclusiveLock on object 19776 of class 2620 of database 16385; blocked by process 16641.\nProcess 16641 waits for AccessExclusiveLock on relation 16739 of database 16385; blocked by process 16642.\nHINT:  See server log for query details.\n\nTraceback (most recent call last):\n  File \"/snap/maas/23947/bin/maas\", line 8, in <module>\n    sys.exit(main())\n  File \"/snap/maas/23947/lib/python3.8/site-packages/maascli/__init__.py\", line 39, in main\n    options.execute(options)\n  File \"/snap/maas/23947/lib/python3.8/site-packages/maascli/snap.py\", line 448, in __call__\n    raise exc\n  File \"/snap/maas/23947/lib/python3.8/site-packages/maascli/snap.py\", line 445, in __call__\n    self.handle(options)\n  File \"/snap/maas/23947/lib/python3.8/site-packages/maascli/snap.py\", line 958, in handle\n    sys.exit(migrate_db())\n  File \"/snap/maas/23947/lib/python3.8/site-packages/maascli/snap.py\", line 353, in migrate_db\n    subprocess.check_call(\n  File \"/usr/lib/python3.8/subprocess.py\", line 364, in check_call\n    raise CalledProcessError(retcode, cmd)\nsubprocess.CalledProcessError: Command '['/snap/maas/23947/bin/maas-region', 'dbupgrade']' returned non-zero exit status 1.", "stderr_lines": ["Traceback (most recent call last):", "  File \"/snap/maas/23947/usr/lib/python3/dist-packages/django/db/backends/utils.py\", line 82, in _execute", "    return self.cursor.execute(sql)", "psycopg2.errors.DeadlockDetected: deadlock detected", "DETAIL:  Process 16642 waits for AccessExclusiveLock on object 19776 of class 2620 of database 16385; blocked by process 16641.", "Process 16641 waits for AccessExclusiveLock on relation 16739 of database 16385; blocked by process 16642.", "HINT:  See server log for query details.", "", "", "The above exception was the direct cause of the following exception:", "", "Traceback (most recent call last):", "  File \"/snap/maas/23947/bin/maas-region\", line 8, in <module>", "    sys.exit(run())", "  File \"/snap/maas/23947/lib/python3.8/site-packages/maasserver/region_script.py\", line 77, in run", "    run_django(is_snap, is_devenv)", "  File \"/snap/maas/23947/lib/python3.8/site-packages/maasserver/region_script.py\", line 66, in run_django", "    management.execute_from_command_line()", "  File \"/snap/maas/23947/usr/lib/python3/dist-packages/django/core/management/__init__.py\", line 381, in execute_from_command_line", "    utility.execute()", "  File \"/snap/maas/23947/usr/lib/python3/dist-packages/django/core/management/__init__.py\", line 375, in execute", "    self.fetch_command(subcommand).run_from_argv(self.argv)", "  File \"/snap/maas/23947/usr/lib/python3/dist-packages/django/core/management/base.py\", line 323, in run_from_argv", "    self.execute(*args, **cmd_options)", "  File \"/snap/maas/23947/usr/lib/python3/dist-packages/django/core/management/base.py\", line 364, in execute", "    output = self.handle(*args, **options)", "  File \"/snap/maas/23947/lib/python3.8/site-packages/maasserver/management/commands/dbupgrade.py\", line 122, in handle", "    self._drop_all_triggers(database)", "  File \"/snap/maas/23947/lib/python3.8/site-packages/maasserver/management/commands/dbupgrade.py\", line 77, in _drop_all_triggers", "    cursor.execute(", "  File \"/snap/maas/23947/usr/lib/python3/dist-packages/django/db/backends/utils.py\", line 67, in execute", "    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)", "  File \"/snap/maas/23947/usr/lib/python3/dist-packages/django/db/backends/utils.py\", line 76, in _execute_with_wrappers", "    return executor(sql, params, many, context)", "  File \"/snap/maas/23947/usr/lib/python3/dist-packages/django/db/backends/utils.py\", line 84, in _execute", "    return self.cursor.execute(sql, params)", "  File \"/snap/maas/23947/usr/lib/python3/dist-packages/django/db/utils.py\", line 89, in __exit__", "    raise dj_exc_value.with_traceback(traceback) from exc_value", "  File \"/snap/maas/23947/usr/lib/python3/dist-packages/django/db/backends/utils.py\", line 82, in _execute", "    return self.cursor.execute(sql)", "django.db.utils.OperationalError: deadlock detected", "DETAIL:  Process 16642 waits for AccessExclusiveLock on object 19776 of class 2620 of database 16385; blocked by process 16641.", "Process 16641 waits for AccessExclusiveLock on relation 16739 of database 16385; blocked by process 16642.", "HINT:  See server log for query details.", "", "Traceback (most recent call last):", "  File \"/snap/maas/23947/bin/maas\", line 8, in <module>", "    sys.exit(main())", "  File \"/snap/maas/23947/lib/python3.8/site-packages/maascli/__init__.py\", line 39, in main", "    options.execute(options)", "  File \"/snap/maas/23947/lib/python3.8/site-packages/maascli/snap.py\", line 448, in __call__", "    raise exc", "  File \"/snap/maas/23947/lib/python3.8/site-packages/maascli/snap.py\", line 445, in __call__", "    self.handle(options)", "  File \"/snap/maas/23947/lib/python3.8/site-packages/maascli/snap.py\", line 958, in handle", "    sys.exit(migrate_db())", "  File \"/snap/maas/23947/lib/python3.8/site-packages/maascli/snap.py\", line 353, in migrate_db", "    subprocess.check_call(", "  File \"/usr/lib/python3.8/subprocess.py\", line 364, in check_call", "    raise CalledProcessError(retcode, cmd)", "subprocess.CalledProcessError: Command '['/snap/maas/23947/bin/maas-region', 'dbupgrade']' returned non-zero exit status 1."], "stdout": "", "stdout_lines": []}
fatal: [192.168.122.181]: FAILED! => {"changed": false, "cmd": ["maas", "migrate"], "delta": "0:00:06.705494", "end": "2022-12-11 21:41:54.568847", "msg": "non-zero return code", "rc": 1, "start": "2022-12-11 21:41:47.863353", "stderr": "Traceback (most recent call last):\n  File \"/snap/maas/23947/usr/lib/python3/dist-packages/django/db/backends/utils.py\", line 82, in _execute\n    return self.cursor.execute(sql)\npsycopg2.errors.DeadlockDetected: deadlock detected\nDETAIL:  Process 16643 waits for AccessExclusiveLock on object 19745 of class 2620 of database 16385; blocked by process 16641.\nProcess 16641 waits for AccessExclusiveLock on relation 16845 of database 16385; blocked by process 16643.\nHINT:  See server log for query details.\n\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n  File \"/snap/maas/23947/bin/maas-region\", line 8, in <module>\n    sys.exit(run())\n  File \"/snap/maas/23947/lib/python3.8/site-packages/maasserver/region_script.py\", line 77, in run\n    run_django(is_snap, is_devenv)\n  File \"/snap/maas/23947/lib/python3.8/site-packages/maasserver/region_script.py\", line 66, in run_django\n    management.execute_from_command_line()\n  File \"/snap/maas/23947/usr/lib/python3/dist-packages/django/core/management/__init__.py\", line 381, in execute_from_command_line\n    utility.execute()\n  File \"/snap/maas/23947/usr/lib/python3/dist-packages/django/core/management/__init__.py\", line 375, in execute\n    self.fetch_command(subcommand).run_from_argv(self.argv)\n  File \"/snap/maas/23947/usr/lib/python3/dist-packages/django/core/management/base.py\", line 323, in run_from_argv\n    self.execute(*args, **cmd_options)\n  File \"/snap/maas/23947/usr/lib/python3/dist-packages/django/core/management/base.py\", line 364, in execute\n    output = self.handle(*args, **options)\n  File \"/snap/maas/23947/lib/python3.8/site-packages/maasserver/management/commands/dbupgrade.py\", line 122, in handle\n    self._drop_all_triggers(database)\n  File \"/snap/maas/23947/lib/python3.8/site-packages/maasserver/management/commands/dbupgrade.py\", line 77, in _drop_all_triggers\n    cursor.execute(\n  File \"/snap/maas/23947/usr/lib/python3/dist-packages/django/db/backends/utils.py\", line 67, in execute\n    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)\n  File \"/snap/maas/23947/usr/lib/python3/dist-packages/django/db/backends/utils.py\", line 76, in _execute_with_wrappers\n    return executor(sql, params, many, context)\n  File \"/snap/maas/23947/usr/lib/python3/dist-packages/django/db/backends/utils.py\", line 84, in _execute\n    return self.cursor.execute(sql, params)\n  File \"/snap/maas/23947/usr/lib/python3/dist-packages/django/db/utils.py\", line 89, in __exit__\n    raise dj_exc_value.with_traceback(traceback) from exc_value\n  File \"/snap/maas/23947/usr/lib/python3/dist-packages/django/db/backends/utils.py\", line 82, in _execute\n    return self.cursor.execute(sql)\ndjango.db.utils.OperationalError: deadlock detected\nDETAIL:  Process 16643 waits for AccessExclusiveLock on object 19745 of class 2620 of database 16385; blocked by process 16641.\nProcess 16641 waits for AccessExclusiveLock on relation 16845 of database 16385; blocked by process 16643.\nHINT:  See server log for query details.\n\nTraceback (most recent call last):\n  File \"/snap/maas/23947/bin/maas\", line 8, in <module>\n    sys.exit(main())\n  File \"/snap/maas/23947/lib/python3.8/site-packages/maascli/__init__.py\", line 39, in main\n    options.execute(options)\n  File \"/snap/maas/23947/lib/python3.8/site-packages/maascli/snap.py\", line 448, in __call__\n    raise exc\n  File \"/snap/maas/23947/lib/python3.8/site-packages/maascli/snap.py\", line 445, in __call__\n    self.handle(options)\n  File \"/snap/maas/23947/lib/python3.8/site-packages/maascli/snap.py\", line 958, in handle\n    sys.exit(migrate_db())\n  File \"/snap/maas/23947/lib/python3.8/site-packages/maascli/snap.py\", line 353, in migrate_db\n    subprocess.check_call(\n  File \"/usr/lib/python3.8/subprocess.py\", line 364, in check_call\n    raise CalledProcessError(retcode, cmd)\nsubprocess.CalledProcessError: Command '['/snap/maas/23947/bin/maas-region', 'dbupgrade']' returned non-zero exit status 1.", "stderr_lines": ["Traceback (most recent call last):", "  File \"/snap/maas/23947/usr/lib/python3/dist-packages/django/db/backends/utils.py\", line 82, in _execute", "    return self.cursor.execute(sql)", "psycopg2.errors.DeadlockDetected: deadlock detected", "DETAIL:  Process 16643 waits for AccessExclusiveLock on object 19745 of class 2620 of database 16385; blocked by process 16641.", "Process 16641 waits for AccessExclusiveLock on relation 16845 of database 16385; blocked by process 16643.", "HINT:  See server log for query details.", "", "", "The above exception was the direct cause of the following exception:", "", "Traceback (most recent call last):", "  File \"/snap/maas/23947/bin/maas-region\", line 8, in <module>", "    sys.exit(run())", "  File \"/snap/maas/23947/lib/python3.8/site-packages/maasserver/region_script.py\", line 77, in run", "    run_django(is_snap, is_devenv)", "  File \"/snap/maas/23947/lib/python3.8/site-packages/maasserver/region_script.py\", line 66, in run_django", "    management.execute_from_command_line()", "  File \"/snap/maas/23947/usr/lib/python3/dist-packages/django/core/management/__init__.py\", line 381, in execute_from_command_line", "    utility.execute()", "  File \"/snap/maas/23947/usr/lib/python3/dist-packages/django/core/management/__init__.py\", line 375, in execute", "    self.fetch_command(subcommand).run_from_argv(self.argv)", "  File \"/snap/maas/23947/usr/lib/python3/dist-packages/django/core/management/base.py\", line 323, in run_from_argv", "    self.execute(*args, **cmd_options)", "  File \"/snap/maas/23947/usr/lib/python3/dist-packages/django/core/management/base.py\", line 364, in execute", "    output = self.handle(*args, **options)", "  File \"/snap/maas/23947/lib/python3.8/site-packages/maasserver/management/commands/dbupgrade.py\", line 122, in handle", "    self._drop_all_triggers(database)", "  File \"/snap/maas/23947/lib/python3.8/site-packages/maasserver/management/commands/dbupgrade.py\", line 77, in _drop_all_triggers", "    cursor.execute(", "  File \"/snap/maas/23947/usr/lib/python3/dist-packages/django/db/backends/utils.py\", line 67, in execute", "    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)", "  File \"/snap/maas/23947/usr/lib/python3/dist-packages/django/db/backends/utils.py\", line 76, in _execute_with_wrappers", "    return executor(sql, params, many, context)", "  File \"/snap/maas/23947/usr/lib/python3/dist-packages/django/db/backends/utils.py\", line 84, in _execute", "    return self.cursor.execute(sql, params)", "  File \"/snap/maas/23947/usr/lib/python3/dist-packages/django/db/utils.py\", line 89, in __exit__", "    raise dj_exc_value.with_traceback(traceback) from exc_value", "  File \"/snap/maas/23947/usr/lib/python3/dist-packages/django/db/backends/utils.py\", line 82, in _execute", "    return self.cursor.execute(sql)", "django.db.utils.OperationalError: deadlock detected", "DETAIL:  Process 16643 waits for AccessExclusiveLock on object 19745 of class 2620 of database 16385; blocked by process 16641.", "Process 16641 waits for AccessExclusiveLock on relation 16845 of database 16385; blocked by process 16643.", "HINT:  See server log for query details.", "", "Traceback (most recent call last):", "  File \"/snap/maas/23947/bin/maas\", line 8, in <module>", "    sys.exit(main())", "  File \"/snap/maas/23947/lib/python3.8/site-packages/maascli/__init__.py\", line 39, in main", "    options.execute(options)", "  File \"/snap/maas/23947/lib/python3.8/site-packages/maascli/snap.py\", line 448, in __call__", "    raise exc", "  File \"/snap/maas/23947/lib/python3.8/site-packages/maascli/snap.py\", line 445, in __call__", "    self.handle(options)", "  File \"/snap/maas/23947/lib/python3.8/site-packages/maascli/snap.py\", line 958, in handle", "    sys.exit(migrate_db())", "  File \"/snap/maas/23947/lib/python3.8/site-packages/maascli/snap.py\", line 353, in migrate_db", "    subprocess.check_call(", "  File \"/usr/lib/python3.8/subprocess.py\", line 364, in check_call", "    raise CalledProcessError(retcode, cmd)", "subprocess.CalledProcessError: Command '['/snap/maas/23947/bin/maas-region', 'dbupgrade']' returned non-zero exit status 1."], "stdout": "", "stdout_lines": []}
ok: [192.168.122.239]

This is an environment with 3 focal hosts.

Hosts file:

[maas1]
192.168.122.239

[maas2]
192.168.122.181

[maas3]
192.168.122.67

[all:children]
maas1
maas2
maas3

[all:vars]
ansible_user = ubuntu

[maas_postgres_primary:children]
maas1

[maas_postgres_secondary:children]
maas2
maas3

# DO NOT MODIFY THIS GROUP
[maas_postgres:children]
maas_postgres_primary
maas_postgres_secondary

[maas_region_controller:children]
maas1
maas2
maas3

[maas_rack_controller:children]
maas1
maas2
maas3

[maas_proxy:children]
maas1
maas2
maas3

vars:

maas_version: "3.2"
maas_postgres_password: mypassword
maas_installation_type: snap
maas_url: http://192.168.122.239:5240/MAAS

I'm guessing the issue is that the init/migrate is being run on all three hosts, when it should only be run once on a single host. Or at least, not in parallel to avoid race conditions.

samuelallan72 commented 1 year ago 
diff --git a/roles/maas_region_controller/tasks/install_maas.yaml b/roles/maas_region_controller/tasks/install_maas.yaml
index 8286b5b..8ac2c08 100644
--- a/roles/maas_region_controller/tasks/install_maas.yaml
+++ b/roles/maas_region_controller/tasks/install_maas.yaml
@@ -42,6 +42,7 @@
 - name: Migrate MAAS database
   ansible.builtin.command: "{{ 'maas' if maas_installation_type | lower == 'snap' else 'maas-region' }} migrate"
   changed_when: false
+  run_once: true

 # MAAS region controller only needs to be initialized in this case if rbac or candid are in use, otherwise the reigond.conf write handles init
 - name: Initialise MAAS Controller - Deb
diff --git a/roles/maas_region_controller/tasks/update_maas.yaml b/roles/maas_region_controller/tasks/update_maas.yaml
index 3e12c0d..975daeb 100644
--- a/roles/maas_region_controller/tasks/update_maas.yaml
+++ b/roles/maas_region_controller/tasks/update_maas.yaml
@@ -18,6 +18,7 @@
 - name: Migrate MAAS database
   ansible.builtin.command: "{{ 'maas' if maas_installation_type | lower == 'snap' else 'maas-region' }} migrate"
   changed_when: false
+  run_once: true

 - name: Refresh MAAS API
   ansible.builtin.command: "{{ 'maas' if maas_installation_type | lower == 'snap' else 'maas-region' }} refresh"

This appears to fix it, but unsure if there are situations where the database migrations must be performed on each host.

ElineMaaikedeWeerd commented 1 year ago

82 implements the run_once: true fixes, closing this issue now