[patch] Tablespaces for Manage DB are not created.

[sachbalag]

This fix enhances the error handling when setupdb,sh fails and exits with a non-zero. Even after adding "pipefail" to the "oc exec" command pipeline, we noticed that on some clusters the RC was still returning a zero when setupdb,sh failed. Therefore completely relying on the RC to drive the retry operation is not feasible.

This fix parses the stdout and evaluates an echo string from setupdb.sh using regex_search to determine if a retry is needed.

We tested the role on a local ansible-devops with playbook. The goal for the test was to make sure that the retry operation was triggered when setupdb.sh echoed a failure message as well as a success message. Both the "happy" and "unhappy" paths were tested. Here are the outputs from the test ...

"Happy Path" indicates a successful assertion ...

TASK [ibm.mas_devops.suite_db2_setup_for_manage : Run setupdb to prepare Db2 for use with Maximo TPAE] *************************************************
changed: [localhost]
TASK [ibm.mas_devops.suite_db2_setup_for_manage : Print setupdb.sh return code] ************************************************************************
ok: [localhost] => {
    "msg": [
        "SetupDB return code (rc) .......................... 0",
        "SetupDB Registered Status  ........................ {'changed': True, 'stdout': 'asdsdffsdfjlskasdfl\\nasdsdffsdfjlskasdfl\\nasdsdffsdfjlskasdfl\\nasdsdffsdfjlskasdfl\\nDBstatus-Instance is available\\nasdsdffsdfjlskasdfl\\nasdsdffsdfjlskasdfl\\nasdsdffsdfjlskasdfl', 'stderr': '', 'rc': 0, 'cmd': \"set -o pipefail\\noc exec -n default test-575f5b5d67-5f2st -- su -lc '/tmp/setupdb.sh | tee /tmp/setupdb.log' \\n\", 'start': '2024-04-10 12:54:58.829820', 'end': '2024-04-10 12:54:59.214246', 'delta': '0:00:00.384426', 'msg': '', 'stdout_lines': ['asdsdffsdfjlskasdfl', 'asdsdffsdfjlskasdfl', 'asdsdffsdfjlskasdfl', 'asdsdffsdfjlskasdfl', 'DBstatus-Instance is available', 'asdsdffsdfjlskasdfl', 'asdsdffsdfjlskasdfl', 'asdsdffsdfjlskasdfl'], 'stderr_lines': [], 'failed': False, 'attempts': 1}"
    ]
}
TASK [ibm.mas_devops.suite_db2_setup_for_manage : Assert that manage tablespace or bufferpools are created] ********************************************
ok: [localhost] => {
    "changed": false,
    "msg": "Manage DB setup ran successfully"
}

"Unhappy path" Shows it retrying ...

TASK [ibm.mas_devops.suite_db2_setup_for_manage : Run setupdb to prepare Db2 for use with Maximo TPAE] *************************************************
FAILED - RETRYING: [localhost]: Run setupdb to prepare Db2 for use with Maximo TPAE (3 retries left).
FAILED - RETRYING: [localhost]: Run setupdb to prepare Db2 for use with Maximo TPAE (2 retries left).
FAILED - RETRYING: [localhost]: Run setupdb to prepare Db2 for use with Maximo TPAE (1 retries left).
fatal: [localhost]: FAILED! => {"attempts": 3, "changed": true, "cmd": "set -o pipefail\noc exec -n default test-575f5b5d67-m2l97 -- su -lc '/tmp/setupdb.sh | tee /tmp/setupdb.log' \n", "delta": "0:00:00.329285", "end": "2024-04-10 12:57:46.885567", "msg": "", "rc": 0, "start": "2024-04-10 12:57:46.556282", "stderr": "", "stderr_lines": [], "stdout": "asdsdffsdfjlskasdfl\nasdsdffsdfjlskasdfl\nasdsdffsdfjlskasdfl\nasdsdffsdfjlskasdfl\nasdsdffsdfjlskasdfl\nasdsdffsdfjlskasdfl\nasdsdffsdfjlskasdfl", "stdout_lines": ["asdsdffsdfjlskasdfl", "asdsdffsdfjlskasdfl", "asdsdffsdfjlskasdfl", "asdsdffsdfjlskasdfl", "asdsdffsdfjlskasdfl", "asdsdffsdfjlskasdfl", "asdsdffsdfjlskasdfl"]}
...ignoring
TASK [ibm.mas_devops.suite_db2_setup_for_manage : Print setupdb.sh return code] ************************************************************************
ok: [localhost] => {
    "msg": [
        "SetupDB return code (rc) .......................... 0",
        "SetupDB Registered Status  ........................ {'changed': True, 'stdout': 'asdsdffsdfjlskasdfl\\nasdsdffsdfjlskasdfl\\nasdsdffsdfjlskasdfl\\nasdsdffsdfjlskasdfl\\nasdsdffsdfjlskasdfl\\nasdsdffsdfjlskasdfl\\nasdsdffsdfjlskasdfl', 'stderr': '', 'rc': 0, 'cmd': \"set -o pipefail\\noc exec -n default test-575f5b5d67-m2l97 -- su -lc '/tmp/setupdb.sh | tee /tmp/setupdb.log' \\n\", 'start': '2024-04-10 12:57:46.556282', 'end': '2024-04-10 12:57:46.885567', 'delta': '0:00:00.329285', 'msg': '', 'stdout_lines': ['asdsdffsdfjlskasdfl', 'asdsdffsdfjlskasdfl', 'asdsdffsdfjlskasdfl', 'asdsdffsdfjlskasdfl', 'asdsdffsdfjlskasdfl', 'asdsdffsdfjlskasdfl', 'asdsdffsdfjlskasdfl'], 'stderr_lines': [], 'failed': True, 'attempts': 3}"
    ]
}
TASK [ibm.mas_devops.suite_db2_setup_for_manage : Assert that manage tablespace or bufferpools are created] ********************************************
fatal: [localhost]: FAILED! => {
    "assertion": "prepare_cmds_status.stdout | regex_search('DBstatus-Instance is available', multiline=True)",
    "changed": false,
    "evaluated_to": false,
    "msg": "setupdb.sh failed for Manage Maxinst -  tablespace or bufferpools may not have been created"
}

https://jsw.ibm.com/browse/MASISMIG-53188