search

redpanda-data / deployment-automation

Cluster configuration best practices
https://redpanda.com
Apache License 2.0
64 stars 46 forks source link

Upgrading cluster by re-applying `provision-cluster.yml` only works the 2nd time #225

Open vuldin opened 3 months ago

vuldin commented 3 months ago

I have the following hosts.ini file that I deploy:

> cat hosts.ini
[redpanda]
node-a ansible_user=root ansible_become=True private_ip=${IP_A} id=0
node-b ansible_user=root ansible_become=True private_ip=${IP_B} id=1
node-c ansible_user=root ansible_become=True private_ip=${IP_C} id=2
> ansible-playbook --private-key ~/.ssh/id_rsa -v ansible/provision-cluster.yml -i hosts.ini -e redpanda_version=24.1.7-1

And when I do a rolling upgrade, I always have to run the following command twice:

ansible-playbook --private-key ~/.ssh/id_rsa -v ansible/provision-cluster.yml -i hosts.ini -e redpanda_version=24.1.8-1

It never completes the upgrade on the first attempt. There are no errors, it just doesn't seem to complete the task the first time (always the second time). After discussions it looks like package_result was removed (thanks @tmgstevens ).

tmgstevens commented 3 months ago

fixed by https://github.com/redpanda-data/redpanda-ansible-collection/pull/100

vuldin commented 3 months ago

Re-opening this issue, as I am not able to verify the fix works. Attached is most of the output from:

ansible-playbook --private-key ~/.ssh/id_rsa -v ansible/provision-cluster.yml -i hosts.ini -e redpanda_version=24.1.8-1 -vvv

ansible-upgrade.log

vuldin commented 3 months ago

The broker seems to have been upgraded according to this output:

root@node-a:~# rpk version
Version:     v24.1.8
Git ref:     f7ac182695
Build date:  2024-06-15T00:26:52Z
OS/Arch:     linux/amd64
Go version:  go1.22.2

Redpanda Cluster
  node-0  v24.1.7 - 53e6eab6abe26155d840c7ae960f71153f1b66af
  node-1  v24.1.7 - 53e6eab6abe26155d840c7ae960f71153f1b66af
  node-2  v24.1.7 - 53e6eab6abe26155d840c7ae960f71153f1b66af

But it seems the brokers weren't restarted properly:

512 TASK [redpanda.cluster.redpanda_broker : Establish whether restart required] ****************************************************************
  1 task path: /root/deployment-automation/artifacts/collections/ansible_collections/redpanda/cluster/roles/redpanda_broker/tasks/start-redpanda.yml:264
  2 ok: [node-a] => {
  3     "ansible_facts": {
  4         "restart_required": false
  5     },
  6     "changed": false
  7 }
  8 ok: [node-b] => {
  9     "ansible_facts": {
 10         "restart_required": false
 11     },
 12     "changed": false
 13 }
 14 ok: [node-c] => {
 15     "ansible_facts": {
 16         "restart_required": false
 17     },
 18     "changed": false
 19 }
vuldin commented 2 months ago

Details of Install redpanda from repository task:

TASK [redpanda.cluster.redpanda_broker : Install redpanda from repository] ****************************************************************************************************************************************************************************************************************
task path: /root/deployment-automation/artifacts/collections/ansible_collections/redpanda/cluster/roles/redpanda_broker/tasks/install-rp-deb.yml:11
changed: [node-b] => {"cache_update_time": 1720619919, "cache_updated": true, "changed": true, "stderr": "", "stderr_lines": [], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following packages will be upgraded:\n  redpanda\n1 upgraded, 0 newly installed, 0 to remove and 11 not upgraded.\nNeed to get 81.7 MB of archives.\nAfter this operation, 98.3 kB of additional disk space will be used.\nGet:1 https://dl.redpanda.com/public/redpanda/deb/ubuntu jammy/main amd64 redpanda amd64 24.1.8-1 [81.7 MB]\nFetched 81.7 MB in 1s (59.6 MB/s)\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 66245 files and directories currently installed.)\r\nPreparing to unpack .../redpanda_24.1.8-1_amd64.deb ...\r\nUnpacking redpanda (24.1.8-1) over (24.1.7-1) ...\r\nSetting up redpanda (24.1.8-1) ...\r\nredpanda:x:998:999::/var/lib/redpanda:/bin/sh\r\nredpanda:x:999:\r\n\r\nTo get the most out of the fastest queue in the west, enable production mode by\r\nrunning the following:\r\n\r\n    sudo rpk redpanda mode production\r\n\r\nfollowed by:\r\n\r\n    sudo rpk redpanda tune all\r\n    sudo systemctl start redpanda\r\n\r\nThis will autotune your system to give you the best performance from Redpanda.\r\nYou can get more information on the tuning parameters here:\r\nhttps://docs.redpanda.com/docs/introduction/autotune/\r\nNEEDRESTART-VER: 3.5\nNEEDRESTART-KCUR: 6.5.0-1023-gcp\nNEEDRESTART-KEXP: 6.5.0-1023-gcp\nNEEDRESTART-KSTA: 1\nNEEDRESTART-SVC: redpanda.service\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following packages will be upgraded:", "  redpanda", "1 upgraded, 0 newly installed, 0 to remove and 11 not upgraded.", "Need to get 81.7 MB of archives.", "After this operation, 98.3 kB of additional disk space will be used.", "Get:1 https://dl.redpanda.com/public/redpanda/deb/ubuntu jammy/main amd64 redpanda amd64 24.1.8-1 [81.7 MB]", "Fetched 81.7 MB in 1s (59.6 MB/s)", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 66245 files and directories currently installed.)", "Preparing to unpack .../redpanda_24.1.8-1_amd64.deb ...", "Unpacking redpanda (24.1.8-1) over (24.1.7-1) ...", "Setting up redpanda (24.1.8-1) ...", "redpanda:x:998:999::/var/lib/redpanda:/bin/sh", "redpanda:x:999:", "", "To get the most out of the fastest queue in the west, enable production mode by", "running the following:", "", "    sudo rpk redpanda mode production", "", "followed by:", "", "    sudo rpk redpanda tune all", "    sudo systemctl start redpanda", "", "This will autotune your system to give you the best performance from Redpanda.", "You can get more information on the tuning parameters here:", "https://docs.redpanda.com/docs/introduction/autotune/", "NEEDRESTART-VER: 3.5", "NEEDRESTART-KCUR: 6.5.0-1023-gcp", "NEEDRESTART-KEXP: 6.5.0-1023-gcp", "NEEDRESTART-KSTA: 1", "NEEDRESTART-SVC: redpanda.service"]}
changed: [node-c] => {"cache_update_time": 1720619919, "cache_updated": true, "changed": true, "stderr": "", "stderr_lines": [], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following packages will be upgraded:\n  redpanda\n1 upgraded, 0 newly installed, 0 to remove and 11 not upgraded.\nNeed to get 81.7 MB of archives.\nAfter this operation, 98.3 kB of additional disk space will be used.\nGet:1 https://dl.redpanda.com/public/redpanda/deb/ubuntu jammy/main amd64 redpanda amd64 24.1.8-1 [81.7 MB]\nFetched 81.7 MB in 1s (78.2 MB/s)\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 66245 files and directories currently installed.)\r\nPreparing to unpack .../redpanda_24.1.8-1_amd64.deb ...\r\nUnpacking redpanda (24.1.8-1) over (24.1.7-1) ...\r\nSetting up redpanda (24.1.8-1) ...\r\nredpanda:x:998:999::/var/lib/redpanda:/bin/sh\r\nredpanda:x:999:\r\n\r\nTo get the most out of the fastest queue in the west, enable production mode by\r\nrunning the following:\r\n\r\n    sudo rpk redpanda mode production\r\n\r\nfollowed by:\r\n\r\n    sudo rpk redpanda tune all\r\n    sudo systemctl start redpanda\r\n\r\nThis will autotune your system to give you the best performance from Redpanda.\r\nYou can get more information on the tuning parameters here:\r\nhttps://docs.redpanda.com/docs/introduction/autotune/\r\nNEEDRESTART-VER: 3.5\nNEEDRESTART-KCUR: 6.5.0-1023-gcp\nNEEDRESTART-KEXP: 6.5.0-1023-gcp\nNEEDRESTART-KSTA: 1\nNEEDRESTART-SVC: redpanda.service\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following packages will be upgraded:", "  redpanda", "1 upgraded, 0 newly installed, 0 to remove and 11 not upgraded.", "Need to get 81.7 MB of archives.", "After this operation, 98.3 kB of additional disk space will be used.", "Get:1 https://dl.redpanda.com/public/redpanda/deb/ubuntu jammy/main amd64 redpanda amd64 24.1.8-1 [81.7 MB]", "Fetched 81.7 MB in 1s (78.2 MB/s)", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 66245 files and directories currently installed.)", "Preparing to unpack .../redpanda_24.1.8-1_amd64.deb ...", "Unpacking redpanda (24.1.8-1) over (24.1.7-1) ...", "Setting up redpanda (24.1.8-1) ...", "redpanda:x:998:999::/var/lib/redpanda:/bin/sh", "redpanda:x:999:", "", "To get the most out of the fastest queue in the west, enable production mode by", "running the following:", "", "    sudo rpk redpanda mode production", "", "followed by:", "", "    sudo rpk redpanda tune all", "    sudo systemctl start redpanda", "", "This will autotune your system to give you the best performance from Redpanda.", "You can get more information on the tuning parameters here:", "https://docs.redpanda.com/docs/introduction/autotune/", "NEEDRESTART-VER: 3.5", "NEEDRESTART-KCUR: 6.5.0-1023-gcp", "NEEDRESTART-KEXP: 6.5.0-1023-gcp", "NEEDRESTART-KSTA: 1", "NEEDRESTART-SVC: redpanda.service"]}
changed: [node-a] => {"cache_update_time": 1720619919, "cache_updated": true, "changed": true, "stderr": "", "stderr_lines": [], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following packages will be upgraded:\n  redpanda\n1 upgraded, 0 newly installed, 0 to remove and 11 not upgraded.\nNeed to get 81.7 MB of archives.\nAfter this operation, 98.3 kB of additional disk space will be used.\nGet:1 https://dl.redpanda.com/public/redpanda/deb/ubuntu jammy/main amd64 redpanda amd64 24.1.8-1 [81.7 MB]\nFetched 81.7 MB in 1s (61.8 MB/s)\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 66245 files and directories currently installed.)\r\nPreparing to unpack .../redpanda_24.1.8-1_amd64.deb ...\r\nUnpacking redpanda (24.1.8-1) over (24.1.7-1) ...\r\nSetting up redpanda (24.1.8-1) ...\r\nredpanda:x:998:999::/var/lib/redpanda:/bin/sh\r\nredpanda:x:999:\r\n\r\nTo get the most out of the fastest queue in the west, enable production mode by\r\nrunning the following:\r\n\r\n    sudo rpk redpanda mode production\r\n\r\nfollowed by:\r\n\r\n    sudo rpk redpanda tune all\r\n    sudo systemctl start redpanda\r\n\r\nThis will autotune your system to give you the best performance from Redpanda.\r\nYou can get more information on the tuning parameters here:\r\nhttps://docs.redpanda.com/docs/introduction/autotune/\r\nNEEDRESTART-VER: 3.5\nNEEDRESTART-KCUR: 6.5.0-1023-gcp\nNEEDRESTART-KEXP: 6.5.0-1023-gcp\nNEEDRESTART-KSTA: 1\nNEEDRESTART-SVC: redpanda.service\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following packages will be upgraded:", "  redpanda", "1 upgraded, 0 newly installed, 0 to remove and 11 not upgraded.", "Need to get 81.7 MB of archives.", "After this operation, 98.3 kB of additional disk space will be used.", "Get:1 https://dl.redpanda.com/public/redpanda/deb/ubuntu jammy/main amd64 redpanda amd64 24.1.8-1 [81.7 MB]", "Fetched 81.7 MB in 1s (61.8 MB/s)", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 66245 files and directories currently installed.)", "Preparing to unpack .../redpanda_24.1.8-1_amd64.deb ...", "Unpacking redpanda (24.1.8-1) over (24.1.7-1) ...", "Setting up redpanda (24.1.8-1) ...", "redpanda:x:998:999::/var/lib/redpanda:/bin/sh", "redpanda:x:999:", "", "To get the most out of the fastest queue in the west, enable production mode by", "running the following:", "", "    sudo rpk redpanda mode production", "", "followed by:", "", "    sudo rpk redpanda tune all", "    sudo systemctl start redpanda", "", "This will autotune your system to give you the best performance from Redpanda.", "You can get more information on the tuning parameters here:", "https://docs.redpanda.com/docs/introduction/autotune/", "NEEDRESTART-VER: 3.5", "NEEDRESTART-KCUR: 6.5.0-1023-gcp", "NEEDRESTART-KEXP: 6.5.0-1023-gcp", "NEEDRESTART-KSTA: 1", "NEEDRESTART-SVC: redpanda.service"]}

The lines for each broker do include 1 upgraded, which is what we look for here.