SODA installation fails on a Ubuntu 18.04 VM on VirtualBox

[amitdn2005]

Issue/Feature Description: Attempted SODA installation using "INSTALLATION GUIDE USING ANSIBLE" (https://docs.sodafoundation.io/soda-gettingstarted/installation-using-ansible/) SODA installation failed while "install keystone with the docker" step. Error message : ""ERROR: keystone didn't come up. Aborting...""

Why this issue to fixed / feature is needed(give scenarios or use cases): This is a standard installation. It should work on a supported Environment (Ubuntu 18.04 Virtualbox VM).

How to reproduce, in case of a bug:

Run this command to install soda with default configuration

ansible-playbook site.yml -i local.hosts -vvvv

Other Notes / Environment Information: (Please give the env information, log link or any useful information for this issue)

Environment:

Ubuntu 18.04 VM on VirtualBox. Network adaptor: NAT, Host-only.

Error log:

TASK [auth-installer : install keystone with the docker] ***************************************************************************************************************************************************
task path: /home/amit/code/soda/installer/ansible/roles/auth-installer/tasks/main.yml:33
Using module file /usr/lib/python2.7/dist-packages/ansible/modules/commands/command.py
<localhost> ESTABLISH LOCAL CONNECTION FOR USER: root
<localhost> EXEC /bin/sh -c 'echo ~ && sleep 0'
<localhost> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /root/.ansible/tmp/ansible-tmp-1633014420.22-255049496116268 `" && echo ansible-tmp-1633014420.22-255049496116268="` echo /root/.ansible/tmp/ansible-tmp-1633014420.22-255049496116268 `" ) && sleep 0'
<localhost> PUT /root/.ansible/tmp/ansible-local-27961mCD52/tmp7uIZPM TO /root/.ansible/tmp/ansible-tmp-1633014420.22-255049496116268/command.py
<localhost> EXEC /bin/sh -c 'chmod u+x /root/.ansible/tmp/ansible-tmp-1633014420.22-255049496116268/ /root/.ansible/tmp/ansible-tmp-1633014420.22-255049496116268/command.py && sleep 0'
<localhost> EXEC /bin/sh -c '/usr/bin/python /root/.ansible/tmp/ansible-tmp-1633014420.22-255049496116268/command.py && sleep 0'
<localhost> EXEC /bin/sh -c 'rm -f -r /root/.ansible/tmp/ansible-tmp-1633014420.22-255049496116268/ > /dev/null 2>&1 && sleep 0'
failed: [localhost] (item=bash ./script/keystone.sh install docker) => {
    "changed": true, 
    "cmd": "bash ./script/keystone.sh install docker", 
    "delta": "0:03:59.887593", 
    "end": "2021-09-30 20:41:00.247563", 
    "invocation": {
        "module_args": {
            "_raw_params": "bash ./script/keystone.sh install docker", 
            "_uses_shell": true, 
            "chdir": null, 
            "creates": null, 
            "executable": null, 
            "removes": null, 
            "stdin": null, 
            "warn": true
        }
    }, 
    "item": "bash ./script/keystone.sh install docker", 
    "msg": "non-zero return code", 
    "rc": 1, 
    "start": "2021-09-30 20:37:00.359970", 
    "stderr": "", 
    "stderr_lines": [], 
    "stdout": "Starting install keystone...\nlatest: Pulling from opensdsio/opensds-authchecker\n34667c7e4631: Pulling fs layer\nd18d76a881a4: Pulling fs layer\n119c7358fbfc: Pulling fs layer\n2aaf13f3eff0: Pulling fs layer\n5824c5a940ae: Pulling fs layer\n7ebdbe4554b7: Pulling fs layer\n5c860faddb96: Pulling fs layer\n57ff77583ca8: Pulling fs layer\n29146555bfc7: Pulling fs layer\n164a9401817c: Pulling fs layer\n7ebdbe4554b7: Waiting\n5c860faddb96: Waiting\n57ff77583ca8: Waiting\n29146555bfc7: Waiting\n164a9401817c: Waiting\n2aaf13f3eff0: Waiting\n5824c5a940ae: Waiting\nd18d76a881a4: Verifying Checksum\nd18d76a881a4: Download complete\n119c7358fbfc: Verifying Checksum\n119c7358fbfc: Download complete\n2aaf13f3eff0: Verifying Checksum\n2aaf13f3eff0: Download complete\n5824c5a940ae: Verifying Checksum\n5824c5a940ae: Download complete\n7ebdbe4554b7: Verifying Checksum\n7ebdbe4554b7: Download complete\n57ff77583ca8: Verifying Checksum\n57ff77583ca8: Download complete\n34667c7e4631: Verifying Checksum\n34667c7e4631: Download complete\n34667c7e4631: Pull complete\nd18d76a881a4: Pull complete\n119c7358fbfc: Pull complete\n2aaf13f3eff0: Pull complete\n5824c5a940ae: Pull complete\n7ebdbe4554b7: Pull complete\n164a9401817c: Verifying Checksum\n164a9401817c: Download complete\n5c860faddb96: Verifying Checksum\n5c860faddb96: Download complete\n5c860faddb96: Pull complete\n57ff77583ca8: Pull complete\n29146555bfc7: Download complete\n29146555bfc7: Pull complete\n164a9401817c: Pull complete\nDigest: sha256:342f5cd850e65fcd9d5ad0b435e3398f7ce1980bb7a08d3fd0e34ce72972de16\nStatus: Downloaded newer image for opensdsio/opensds-authchecker:latest\n92a1b8f596e6ba72e23f2c008f1285bd95da12d06ed0bbbd22dc70989a557c52\nERROR: keystone didn't come up. Aborting...", 
    "stdout_lines": [
        "Starting install keystone...", 
        "latest: Pulling from opensdsio/opensds-authchecker", 
        "34667c7e4631: Pulling fs layer", 
        "d18d76a881a4: Pulling fs layer", 
        "119c7358fbfc: Pulling fs layer", 
        "2aaf13f3eff0: Pulling fs layer", 
        "5824c5a940ae: Pulling fs layer", 
        "7ebdbe4554b7: Pulling fs layer", 
        "5c860faddb96: Pulling fs layer", 
        "57ff77583ca8: Pulling fs layer", 
        "29146555bfc7: Pulling fs layer", 
        "164a9401817c: Pulling fs layer", 
        "7ebdbe4554b7: Waiting", 
        "5c860faddb96: Waiting", 
        "57ff77583ca8: Waiting", 
        "29146555bfc7: Waiting", 
        "164a9401817c: Waiting", 
        "2aaf13f3eff0: Waiting", 
        "5824c5a940ae: Waiting", 
        "d18d76a881a4: Verifying Checksum", 
        "d18d76a881a4: Download complete", 
        "119c7358fbfc: Verifying Checksum", 
        "119c7358fbfc: Download complete", 
        "2aaf13f3eff0: Verifying Checksum", 
        "2aaf13f3eff0: Download complete", 
        "5824c5a940ae: Verifying Checksum", 
        "5824c5a940ae: Download complete", 
        "7ebdbe4554b7: Verifying Checksum", 
        "7ebdbe4554b7: Download complete", 
        "57ff77583ca8: Verifying Checksum", 
        "57ff77583ca8: Download complete", 
        "34667c7e4631: Verifying Checksum", 
        "34667c7e4631: Download complete", 
        "34667c7e4631: Pull complete", 
        "d18d76a881a4: Pull complete", 
        "119c7358fbfc: Pull complete", 
        "2aaf13f3eff0: Pull complete", 
        "5824c5a940ae: Pull complete", 
        "7ebdbe4554b7: Pull complete", 
        "164a9401817c: Verifying Checksum", 
        "164a9401817c: Download complete", 
        "5c860faddb96: Verifying Checksum", 
        "5c860faddb96: Download complete", 
        "5c860faddb96: Pull complete", 
        "57ff77583ca8: Pull complete", 
        "29146555bfc7: Download complete", 
        "29146555bfc7: Pull complete", 
        "164a9401817c: Pull complete", 
        "Digest: sha256:342f5cd850e65fcd9d5ad0b435e3398f7ce1980bb7a08d3fd0e34ce72972de16", 
        "Status: Downloaded newer image for opensdsio/opensds-authchecker:latest", 
        "92a1b8f596e6ba72e23f2c008f1285bd95da12d06ed0bbbd22dc70989a557c52", 
        "ERROR: keystone didn't come up. Aborting..."
    ]
}
    to retry, use: --limit @/home/amit/code/soda/installer/ansible/site.retry

PLAY RECAP *************************************************************************************************************************************************************************************************
localhost                  : ok=24   changed=11   unreachable=0    failed=1   

root@amit-VirtualBox:/home/amit/code/soda/installer/ansible# 

[amitdn2005]

Root cause

The root cause was that the keystone was taking time to come up and our installer installer/ansible/script/keystone.sh has a method called wait_for_keystone which ran for 12 counts and then threw an error.

wait_for_keystone () {
    local count=0
    local interval=${1:-10}
    local times=${2:-12}    <<<<< 

Increased the retry count in installer/ansible/script/keystone.sh to 40, assuming keystone was taking time to come up.

In this case keystone came up after around 28 retries.

This worked and the keystone came up later than usual.