Node needs valid IP before joining cluster

SUMMARY

I have been using the express role to add nodes to an existing cluster. Everything works fine, except sometimes I get the following error:

TASK [k8s-cluster-attach : Attaching to Cluster : UUID = 41f44e62-7af5-4323-8c37-bd02726f9836] ***
Friday 22 January 2021  14:21:01 +0000 (0:00:00.308)       0:05:08.418 ********
fatal: [example.hostname.com]: FAILED! => {"access_control_allow_credentials": "true", "changed": false, "connection": "close", "content": "{\"error\":{\"message\":\"Error: node e147f337-e56c-461e-acad-d3b03eee93e5 needs valid IP before joining cluster null\",\"code\":400}}", "content_length": "126", "content_type": "application/json; charset=utf-8", "date": "Fri, 22 Jan 2021 14:21:03 GMT", "elapsed": 1, "json": {"error": {"code": 400, "message": "Error: node e147f337-e56c-461e-acad-d3b03eee93e5 needs valid IP before joining cluster null"}}, "msg": "Status code was 400 and not [200]: HTTP Error 400: Bad Request", "redirected": false, "server": "nginx/1.14.1", "status": 400, "url": "https://example.platform9.net/qbert/v1/clusters/41f44e62-7af5-4323-8c37-bd02726f9836/attach"}
...ignoring

TASK [k8s-cluster-attach : debug] **********************************************
Friday 22 January 2021  14:21:03 +0000 (0:00:02.120)       0:05:10.539 ********
ok: [example.hostname.com] => {
    "uri_result": {
        "access_control_allow_credentials": "true",
        "changed": false,
        "connection": "close",
        "content": "{\"error\":{\"message\":\"Error: node e147f337-e56c-461e-acad-d3b03eee93e5 needs valid IP before joining cluster null\",\"code\":400}}",
        "content_length": "126",
        "content_type": "application/json; charset=utf-8",
        "date": "Fri, 22 Jan 2021 14:21:03 GMT",
        "elapsed": 1,
        "failed": true,
        "json": {
            "error": {
                "code": 400,
                "message": "Error: node e147f337-e56c-461e-acad-d3b03eee93e5 needs valid IP before joining cluster null"
            }
        },
        "msg": "Status code was 400 and not [200]: HTTP Error 400: Bad Request",
        "redirected": false,
        "server": "nginx/1.14.1",
        "status": 400,
        "url": "https://example.platform9.net/qbert/v1/clusters/41f44e62-7af5-4323-8c37-bd02726f9836/attach"
    }
}

So node needs valid IP? Not sure what that means. When I run the playbook again it attaches the node to the cluster with no issue. Sometimes the playbook works from the first time as well. Is there a certain check that needs to be added/is skipped?

ISSUE TYPE

Bug Report

COMPONENT NAME

Express role

ANSIBLE VERSION

ansible 2.9.16
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/opt/app-root/src/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /opt/app-root/lib/python3.6/site-packages/ansible
  executable location = /opt/app-root/bin/ansible
  python version = 3.6.9 (default, Nov 11 2019, 11:24:16) [GCC 4.8.5 20150623 (Red Hat 4.8.5-39)]

CONFIGURATION

ANSIBLE_NOCOWS(/etc/ansible/ansible.cfg) = True
ANSIBLE_PIPELINING(/etc/ansible/ansible.cfg) = True
COMMAND_WARNINGS(/etc/ansible/ansible.cfg) = False
DEFAULT_CALLBACK_WHITELIST(/etc/ansible/ansible.cfg) = ['profile_roles']
DEFAULT_HOST_LIST(/etc/ansible/ansible.cfg) = ['/etc/ansible/inventory/hosts']
DEFAULT_ROLES_PATH(/etc/ansible/ansible.cfg) = ['/opt/pf9/express/roles']
DEPRECATION_WARNINGS(/etc/ansible/ansible.cfg) = False
HOST_KEY_CHECKING(/etc/ansible/ansible.cfg) = False
RETRY_FILES_ENABLED(/etc/ansible/ansible.cfg) = False

OS / ENVIRONMENT

NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

EXPECTED RESULTS

Attach node successfully

TASK [k8s-cluster-attach : Attaching to Cluster : UUID = 41f44e62-7af5-4323-8c37-bd02726f9836] ***
Friday 22 January 2021  14:32:43 +0000 (0:00:00.155)       0:04:23.126 ********
ok: [example.hostname.com] => {"access_control_allow_credentials": "true", "access_control_expose_headers": "X-Subject-Token", "changed": false, "connection": "close", "content_length": "2", "content_type": "text/plain; charset=utf-8", "cookies": {}, "cookies_string": "", "date": "Fri, 22 Jan 2021 14:32:45 GMT", "elapsed": 1, "msg": "OK (2 bytes)", "redirected": false, "server": "nginx/1.14.1", "status": 200, "url": "https://example.platform9.net/qbert/v1/clusters/41f44e62-7af5-4323-8c37-bd02726f9836/attach"}

TASK [k8s-cluster-attach : debug] **********************************************
Friday 22 January 2021  14:32:45 +0000 (0:00:02.157)       0:04:25.283 ********
ok: [example.hostname.com] => {
    "uri_result": {
        "access_control_allow_credentials": "true",
        "access_control_expose_headers": "X-Subject-Token",
        "changed": false,
        "connection": "close",
        "content_length": "2",
        "content_type": "text/plain; charset=utf-8",
        "cookies": {},
        "cookies_string": "",
        "date": "Fri, 22 Jan 2021 14:32:45 GMT",
        "elapsed": 1,
        "failed": false,
        "msg": "OK (2 bytes)",
        "redirected": false,
        "server": "nginx/1.14.1",
        "status": 200,
        "url": "https://example.platform9.net/qbert/v1/clusters/41f44e62-7af5-4323-8c37-bd02726f9836/attach"
    }
}

ACTUAL RESULTS

TASK [k8s-cluster-attach : Attaching to Cluster : UUID = 41f44e62-7af5-4323-8c37-bd02726f9836] ***
Friday 22 January 2021  14:21:01 +0000 (0:00:00.308)       0:05:08.418 ********
fatal: [example.hostname.com]: FAILED! => {"access_control_allow_credentials": "true", "changed": false, "connection": "close", "content": "{\"error\":{\"message\":\"Error: node e147f337-e56c-461e-acad-d3b03eee93e5 needs valid IP before joining cluster null\",\"code\":400}}", "content_length": "126", "content_type": "application/json; charset=utf-8", "date": "Fri, 22 Jan 2021 14:21:03 GMT", "elapsed": 1, "json": {"error": {"code": 400, "message": "Error: node e147f337-e56c-461e-acad-d3b03eee93e5 needs valid IP before joining cluster null"}}, "msg": "Status code was 400 and not [200]: HTTP Error 400: Bad Request", "redirected": false, "server": "nginx/1.14.1", "status": 400, "url": "https://example.platform9.net/qbert/v1/clusters/41f44e62-7af5-4323-8c37-bd02726f9836/attach"}
...ignoring

TASK [k8s-cluster-attach : debug] **********************************************
Friday 22 January 2021  14:21:03 +0000 (0:00:02.120)       0:05:10.539 ********
ok: [example.hostname.com] => {
    "uri_result": {
        "access_control_allow_credentials": "true",
        "changed": false,
        "connection": "close",
        "content": "{\"error\":{\"message\":\"Error: node e147f337-e56c-461e-acad-d3b03eee93e5 needs valid IP before joining cluster null\",\"code\":400}}",
        "content_length": "126",
        "content_type": "application/json; charset=utf-8",
        "date": "Fri, 22 Jan 2021 14:21:03 GMT",
        "elapsed": 1,
        "failed": true,
        "json": {
            "error": {
                "code": 400,
                "message": "Error: node e147f337-e56c-461e-acad-d3b03eee93e5 needs valid IP before joining cluster null"
            }
        },
        "msg": "Status code was 400 and not [200]: HTTP Error 400: Bad Request",
        "redirected": false,
        "server": "nginx/1.14.1",
        "status": 400,
        "url": "https://example.platform9.net/qbert/v1/clusters/41f44e62-7af5-4323-8c37-bd02726f9836/attach"
    }
}

platform9 / express

Node needs valid IP before joining cluster #271