"Connection failure: The read operation timed out" when run the sample_role_cpm_provision.yml

[lmiaobj]

ansible version: 2.9.0 ibm_zos_zosmf version: 1.0.0

When I run the sample_role_cpm_provision.yml to create the software service, the instance can be provisioned, but I got following error in task log:

TASK [provision_software_service : Provision template] ************************************************************************************************************
fatal: [S0W1 -> localhost]: FAILED! => {"changed": false, "content": "", "elapsed": 31, "msg": "Status code was -1 and not [200]: Connection failure: The read operation timed out", "redirected": false, "status": -1, "url": "https://xx.xx.xx.xx:443/zosmf/provisioning/rest/1.0/psc/CICS55/actions/run"}

[shreeji818]

Will you please run playbook with -vvvv option and see if there is additional information that can help resolving the problem.

[lmiaobj]

Here is the trace log for it.

TASK [provision_software_service : Provision template] **** task path: /Users/liumiao/.ansible/collections/ansible_collections/ibm/ibm_zos_zosmf/roles/provision_software_service/tasks/main.yml:7

ESTABLISH LOCAL CONNECTION FOR USER: liumiao EXEC /bin/sh -c 'echo ~liumiao && sleep 0' EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /Users/liumiao/.ansible/tmp/ansible-tmp-1585101397.0664-180491297716772 `" && echo ansible-tmp-1585101397.0664-180491297716772="` echo /Users/liumiao/.ansible/tmp/ansible-tmp-1585101397.0664-180491297716772 `" ) && sleep 0' Using module file /usr/local/Cellar/ansible/2.9.0/libexec/lib/python3.7/site-packages/ansible/modules/net_tools/basics/uri.py PUT /Users/liumiao/.ansible/tmp/ansible-local-73034ecqknsd9/tmp7r51tyxt TO /Users/liumiao/.ansible/tmp/ansible-tmp-1585101397.0664-180491297716772/AnsiballZ_uri.py EXEC /bin/sh -c 'chmod u+x /Users/liumiao/.ansible/tmp/ansible-tmp-1585101397.0664-180491297716772/ /Users/liumiao/.ansible/tmp/ansible-tmp-1585101397.0664-180491297716772/AnsiballZ_uri.py && sleep 0' EXEC /bin/sh -c '/usr/local/Cellar/ansible/2.9.0/libexec/bin/python3.7 /Users/liumiao/.ansible/tmp/ansible-tmp-1585101397.0664-180491297716772/AnsiballZ_uri.py && sleep 0' EXEC /bin/sh -c 'rm -f -r /Users/liumiao/.ansible/tmp/ansible-tmp-1585101397.0664-180491297716772/ > /dev/null 2>&1 && sleep 0' fatal: [S0W1 -> localhost]: FAILED! => { "changed": false, "content": "", "elapsed": 36, "invocation": { "module_args": { "attributes": null, "backup": null, "body": { "domain-name": "default", "systems-nicknames": [ "S0W1" ], "tenant-name": "default" }, "body_format": "json", "client_cert": null, "client_key": null, "content": null, "creates": null, "delimiter": null, "dest": null, "directory_mode": null, "follow": false, "follow_redirects": "safe", "force": false, "force_basic_auth": true, "group": null, "headers": { "Content-Type": "application/json", "Host": "9.21.106.xx", "Origin": "https://9.21.106.xx" }, "http_agent": "ansible-httpget", "method": "POST", "mode": null, "owner": null, "password": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER", "regexp": null, "remote_src": null, "removes": null, "return_content": true, "selevel": null, "serole": null, "setype": null, "seuser": null, "src": null, "status_code": [ "200" ], "timeout": 30, "unix_socket": null, "unsafe_writes": null, "url": "https://9.21.106.xx:443/zosmf/provisioning/rest/1.0/psc/CICS55/actions/run", "url_password": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER", "url_username": "ibmuser", "use_proxy": true, "user": "ibmuser", "validate_certs": false } }, "msg": "Status code was -1 and not [200]: Connection failure: The read operation timed out", "redirected": false, "status": -1, "url": "https://9.21.106.xx:443/zosmf/provisioning/rest/1.0/psc/CICS55/actions/run" }

[viane]

The -1 status code means your ansible is not able to communicate with the 9.x.x.x zosmf (network issue). Are you able to ping that zosmf locally? Or if u have somehow restricted ansible network outbound...

I suggest do a simple GET to ur zosmf home page by using a simple playbook to test the network connectivity.

[shreeji818]

Looks like there is a network connection issue between local machine and target z/OSMF system. If your system is behind firewall you may need VPN. Were you able to launch z/OSMF UI from your workstation browser https://9.21.106.xx:443/zosmf/ ?

[lmiaobj]

@viane I test the network using that simple playbook, it works well. And I also can copy file from my workstation to z/OS system using ansible. @shreeji818 Yes, I can launch https://9.21.106.xx:443/zosmf/ from my workstation browser.

[shreeji818]

@lmiaobj Will you please send me email on hiren@us.ibm.com?

[shreeji818]

@lmiaobj Are you still experiencing problem? Can we close this issue?

[marija-perkovic]

Hi, I have the same issue, although the workflow is triggered and the resource provisioned ... but it seems Ansible doesn't wait on the response .... Is there a way to change timeout or ... (I tried, but nothing works). Thank you!

[shreeji818]

@marija-perkovic I believe you are getting into a failure like below bur provisioning continues and is successful, right? Default socket time out is 30 seconds but it may be possible that on slow network, you may not receiving response back in 30 seconds.

TASK [ibm.ibm_zos_zosmf.zmf_cpm_provision_software_service : Provision template] ***** fatal: [ClientCtr]: FAILED! => {"changed": false, "content": "", "elapsed": 1, "msg": "Status code was -1 and not [200]: Connection failure: The read operation timed out", "redirected": false, "status": -1, "url": "https://ex240n01.pbm.ihost.com:443/zosmf/provisioning/rest/1.0/psc/WLPTemplate/actions/run"}

You can change add a timeout parameter in the ../ansible_collections/ibm/ibm_zos_zosmf/roles/zmf_cpm_provision_software_service/tasks/main.yml. Following is an example of timeout specification that waits for response for up to 5 minutes.

• name: Provision template uri: timeout: 300 url: "https://{{ zmf_host }}:{{ zmf_port }}/zosmf/provisioning/rest/1.0/psc/{{ cpm_template_name }}/actions/run" return_content: yes .. .. ..

[viane]

Since the lab env wasn't cover long request-response, I suggest that if you can configure your ansible to give enough wait time on ALL requests by doing the following:

1. Modify the ansible.cfg under ibm_zos_zosmf/playbooks and add connect_timeout = xxx command_timeout = xxx in seconds, for explanation reference this example
2. set both values to be the number of seconds you want to wait. (keep in mind that command_timeout >= connect_timeout)
3. Profit

@shreeji818 since this change will improve the stability in general, should we add to the source?

[marija-perkovic]

@shreeji818 - yes, I was even able to find the place finally, and it worked ... I am working with zPDT, so it is a bit slower, 60 seconds was enough, but more doesn't hurt.

@viane - I tried all different timeouts in ansible.cfg, but not those two.

So, I think either to externalize timeout variable or to document/suggest those defaults for ansible.cfg woud be good.

Thanks!

[viane]

@marija-perkovic we can do this since it's an easy fix to add, but before that could you confirm that the change in the cfg file solves the issue? Also, we have to come up with a safe number for the timeout so that the user actually has the network connectivity issue will not wait excessive time.

[shreeji818]

@viane The timeout value is maximum time socket connectivity would wait for. If socket connection established earlier, playbook would run with out the wait. So, I am suggesting to set timeout=300 (5 minutes) which is good amount of time to wait for connectivity in case of zPDT type of slow network environment.

[tehdisko]

Hi, I have the similar issue. I do a long operation on a network device, having output:

    "changed": false,
    "module_stderr": "Traceback (most recent call last):\n  File \"/home/user/.ansible/tmp/ansible-local-25969j2q87ymj/ansible-tmp-1650986164.1702852-203734164353076/AnsiballZ_cp_mgmt_group.py\", line 102, in <module>\n    _ansiballz_main()\n  File \"/home/user/.ansible/tmp/ansible-local-25969j2q87ymj/ansible-tmp-1650986164.1702852-203734164353076/AnsiballZ_cp_mgmt_group.py\", line 94, in _ansiballz_main\n    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\n  File \"/home/user/.ansible/tmp/ansible-local-25969j2q87ymj/ansible-tmp-1650986164.1702852-203734164353076/AnsiballZ_cp_mgmt_group.py\", line 40, in invoke_module\n    runpy.run_module(mod_name='ansible_collections.check_point.mgmt.plugins.modules.cp_mgmt_group', init_globals=None, run_name='__main__', alter_sys=True)\n  File \"/usr/lib/python3.8/runpy.py\", line 207, in run_module\n    return _run_module_code(code, init_globals, run_name, mod_spec)\n  File \"/usr/lib/python3.8/runpy.py\", line 97, in _run_module_code\n    _run_code(code, mod_globals, init_globals,\n  File \"/usr/lib/python3.8/runpy.py\", line 87, in _run_code\n    exec(code, run_globals)\n  File \"/tmp/ansible_check_point.mgmt.cp_mgmt_group_payload_6zczmhaz/ansible_check_point.mgmt.cp_mgmt_group_payload.zip/ansible_collections/check_point/mgmt/plugins/modules/cp_mgmt_group.py\", line 140, in <module>\n  File \"/tmp/ansible_check_point.mgmt.cp_mgmt_group_payload_6zczmhaz/ansible_check_point.mgmt.cp_mgmt_group_payload.zip/ansible_collections/check_point/mgmt/plugins/modules/cp_mgmt_group.py\", line 135, in main\n  File \"/tmp/ansible_check_point.mgmt.cp_mgmt_group_payload_6zczmhaz/ansible_check_point.mgmt.cp_mgmt_group_payload.zip/ansible_collections/check_point/mgmt/plugins/module_utils/checkpoint.py\", line 317, in api_call\n  File \"/tmp/ansible_check_point.mgmt.cp_mgmt_group_payload_6zczmhaz/ansible_check_point.mgmt.cp_mgmt_group_payload.zip/ansible_collections/check_point/mgmt/plugins/module_utils/checkpoint.py\", line 71, in send_request\n  File \"/tmp/ansible_check_point.mgmt.cp_mgmt_group_payload_6zczmhaz/ansible_check_point.mgmt.cp_mgmt_group_payload.zip/ansible/module_utils/connection.py\", line 185, in __rpc__\nansible.module_utils.connection.ConnectionError: The read operation timed out\n",
    "module_stdout": "",
    "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error",
    "rc": 1
}

Changing connect_timeout and command_timeout does't help.

[viane]

@shreeji818 do you want me to look over this?

[shreeji818]

@tehdisko Is this issue with Cloud provisioning roles?

By the way ibm_zos_zosmf collection is deprecated and we recommend using ibm_zosmf collection.

https://galaxy.ansible.com/ibm/ibm_zosmf

Please submit your issue in github repo associated with ibm_zosmf collection.

https://github.com/IBM/ibm_zosmf