Deployment - MariaDB container stuck on restarting on the controller node

[cecco16]

`TASK [mariadb : Waiting for MariaDB service to be ready] ***** FAILED - RETRYING: Waiting for MariaDB service to be ready (10 retries left). FAILED - RETRYING: Waiting for MariaDB service to be ready (9 retries left). FAILED - RETRYING: Waiting for MariaDB service to be ready (8 retries left). FAILED - RETRYING: Waiting for MariaDB service to be ready (7 retries left). FAILED - RETRYING: Waiting for MariaDB service to be ready (6 retries left). FAILED - RETRYING: Waiting for MariaDB service to be ready (5 retries left). FAILED - RETRYING: Waiting for MariaDB service to be ready (4 retries left). FAILED - RETRYING: Waiting for MariaDB service to be ready (3 retries left). FAILED - RETRYING: Waiting for MariaDB service to be ready (2 retries left). FAILED - RETRYING: Waiting for MariaDB service to be ready (1 retries left). fatal: [192.168.6.13]: FAILED! => {"attempts": 10, "changed": false, "elapsed": 60, "msg": "Timeout when waiting for search string MariaDB in 192.168.6.13:3306"} to retry, use: --limit @/root/contrail-ansible-deployer/playbooks/install_contrail.retry

PLAY RECAP *** 192.168.6.12 : ok=55 changed=21 unreachable=0 failed=0 192.168.6.13 : ok=118 changed=55 unreachable=0 failed=1 localhost : ok=7 changed=2 unreachable=0 failed=0

[root@controller5 db]# docker logs --tail 50 --follow --timestamps mariadb 2018-05-09T16:16:49.792265592Z INFO:main:Setting permission for /var/lib/mysql/performance_schema/events_waits_summary_by_host_by_event_name.frm 2018-05-09T16:16:49.792271446Z INFO:main:Setting permission for /var/lib/mysql/performance_schema/events_waits_summary_by_user_by_event_name.frm 2018-05-09T16:16:49.792279855Z INFO:main:Setting permission for /var/lib/mysql/performance_schema/events_waits_summary_by_account_by_event_name.frm 2018-05-09T16:16:49.792286029Z INFO:main:Setting permission for /var/lib/mysql/performance_schema/events_waits_summary_by_thread_by_event_name.frm 2018-05-09T16:16:49.792291929Z INFO:main:Setting permission for /var/lib/mysql/performance_schema/events_waits_summary_global_by_event_name.frm 2018-05-09T16:16:49.792297786Z INFO:main:Setting permission for /var/lib/mysql/performance_schema/file_instances.frm 2018-05-09T16:16:49.792303566Z INFO:main:Setting permission for /var/lib/mysql/performance_schema/file_summary_by_event_name.frm 2018-05-09T16:16:49.792322838Z INFO:main:Setting permission for /var/lib/mysql/performance_schema/file_summary_by_instance.frm 2018-05-09T16:16:49.792328799Z INFO:main:Setting permission for /var/lib/mysql/performance_schema/socket_instances.frm 2018-05-09T16:16:49.792334582Z INFO:main:Setting permission for /var/lib/mysql/performance_schema/socket_summary_by_instance.frm 2018-05-09T16:16:49.792340462Z INFO:main:Setting permission for /var/lib/mysql/performance_schema/socket_summary_by_event_name.frm 2018-05-09T16:16:49.792346235Z INFO:main:Setting permission for /var/lib/mysql/performance_schema/host_cache.frm 2018-05-09T16:16:49.792351929Z INFO:main:Setting permission for /var/lib/mysql/performance_schema/mutex_instances.frm 2018-05-09T16:16:49.792357692Z INFO:main:Setting permission for /var/lib/mysql/performance_schema/objects_summary_global_by_type.frm 2018-05-09T16:16:49.792363482Z INFO:main:Setting permission for /var/lib/mysql/performance_schema/performance_timers.frm 2018-05-09T16:16:49.792369255Z INFO:main:Setting permission for /var/lib/mysql/performance_schema/rwlock_instances.frm 2018-05-09T16:16:49.792374965Z INFO:main:Setting permission for /var/lib/mysql/performance_schema/setup_actors.frm 2018-05-09T16:16:49.792380682Z INFO:main:Setting permission for /var/lib/mysql/performance_schema/setup_consumers.frm 2018-05-09T16:16:49.792386412Z INFO:main:Setting permission for /var/lib/mysql/performance_schema/setup_instruments.frm 2018-05-09T16:16:49.792392750Z INFO:main:Setting permission for /var/lib/mysql/performance_schema/setup_objects.frm 2018-05-09T16:16:49.792398764Z INFO:main:Setting permission for /var/lib/mysql/performance_schema/setup_timers.frm 2018-05-09T16:16:49.792404584Z INFO:main:Setting permission for /var/lib/mysql/performance_schema/table_io_waits_summary_by_index_usage.frm 2018-05-09T16:16:49.792410574Z INFO:main:Setting permission for /var/lib/mysql/performance_schema/table_io_waits_summary_by_table.frm 2018-05-09T16:16:49.792417319Z INFO:main:Setting permission for /var/lib/mysql/performance_schema/table_lock_waits_summary_by_table.frm 2018-05-09T16:16:49.792423249Z INFO:main:Setting permission for /var/lib/mysql/performance_schema/threads.frm 2018-05-09T16:16:49.792428935Z INFO:main:Setting permission for /var/lib/mysql/performance_schema/events_stages_current.frm 2018-05-09T16:16:49.792434772Z INFO:main:Setting permission for /var/lib/mysql/performance_schema/events_stages_history.frm 2018-05-09T16:16:49.792440539Z INFO:main:Setting permission for /var/lib/mysql/performance_schema/events_stages_history_long.frm 2018-05-09T16:16:49.792446292Z INFO:main:Setting permission for /var/lib/mysql/performance_schema/events_stages_summary_by_thread_by_event_name.frm 2018-05-09T16:16:49.792452202Z INFO:main:Setting permission for /var/lib/mysql/performance_schema/events_stages_summary_by_host_by_event_name.frm 2018-05-09T16:16:49.792458026Z INFO:main:Setting permission for /var/lib/mysql/performance_schema/events_stages_summary_by_user_by_event_name.frm 2018-05-09T16:16:49.792467704Z INFO:main:Setting permission for /var/lib/mysql/performance_schema/events_stages_summary_by_account_by_event_name.frm 2018-05-09T16:16:49.792473801Z INFO:main:Setting permission for /var/lib/mysql/performance_schema/events_stages_summary_global_by_event_name.frm 2018-05-09T16:16:49.792479714Z INFO:main:Setting permission for /var/lib/mysql/performance_schema/events_statements_current.frm 2018-05-09T16:16:49.792485498Z INFO:main:Setting permission for /var/lib/mysql/performance_schema/events_statements_history.frm 2018-05-09T16:16:49.792491318Z INFO:main:Setting permission for /var/lib/mysql/performance_schema/events_statements_history_long.frm 2018-05-09T16:16:49.792497121Z INFO:main:Setting permission for /var/lib/mysql/performance_schema/events_statements_summary_by_thread_by_event_name.frm 2018-05-09T16:16:49.792503011Z INFO:main:Setting permission for /var/lib/mysql/performance_schema/events_statements_summary_by_host_by_event_name.frm 2018-05-09T16:16:49.792508941Z INFO:main:Setting permission for /var/lib/mysql/performance_schema/events_statements_summary_by_user_by_event_name.frm 2018-05-09T16:16:49.792514852Z INFO:main:Setting permission for /var/lib/mysql/performance_schema/events_statements_summary_by_account_by_event_name.frm 2018-05-09T16:16:49.792520782Z INFO:main:Setting permission for /var/lib/mysql/performance_schema/events_statements_summary_global_by_event_name.frm 2018-05-09T16:16:49.792527837Z INFO:main:Setting permission for /var/lib/mysql/performance_schema/hosts.frm 2018-05-09T16:16:49.792533703Z INFO:main:Setting permission for /var/lib/mysql/performance_schema/users.frm 2018-05-09T16:16:49.792539477Z INFO:main:Setting permission for /var/lib/mysql/performance_schema/accounts.frm 2018-05-09T16:16:49.792545177Z INFO:main:Setting permission for /var/lib/mysql/performance_schema/events_statements_summary_by_digest.frm 2018-05-09T16:16:49.792551033Z INFO:main:Setting permission for /var/lib/mysql/performance_schema/session_connect_attrs.frm 2018-05-09T16:16:49.792556920Z INFO:main:Setting permission for /var/lib/mysql/performance_schema/session_account_connect_attrs.frm 2018-05-09T16:16:49.808016993Z Running command: '/usr/bin/mysqld_safe' 2018-05-09T16:16:49.978048481Z 180509 16:16:49 mysqld_safe Logging to '/var/log/kolla/mariadb/mariadb.log'. 2018-05-09T16:16:50.028770833Z 180509 16:16:50 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql/ `

[fclaudiopalmeira]

Hello My Friend, this happens due to the newer docker version incompatibility with the playbooks, the solution is to execute the following command, before attempting to deploy the containers? pip install docker==2.7.0

Cheers m8.

[cecco16]

Hi there,

thanks for tor the tip. Tried to install the right version right before running the playbooks/install_contrail.yml

[root@controller5-2 ~]# pip install docker==2.7.0 Collecting docker==2.7.0 Downloading https://files.pythonhosted.org/packages/6b/01/74d61c7de9fb99cebbbfab62ee5c88a9b23e8b1c25a57eded5fc22e87bd2/docker-2.7.0-py2.py3-none-any.whl (119kB) 100% |████████████████████████████████| 122kB 3.7MB/s Requirement already satisfied (use --upgrade to upgrade): backports.ssl-match-hostname>=3.5; python_version < "3.5" in /usr/lib/python2.7/site-packages (from docker==2.7.0) Requirement already satisfied (use --upgrade to upgrade): docker-pycreds>=0.2.1 in /usr/lib/python2.7/site-packages (from docker==2.7.0) Collecting requests!=2.18.0,>=2.14.2 (from docker==2.7.0) Downloading https://files.pythonhosted.org/packages/49/df/50aa1999ab9bde74656c2919d9c0c085fd2b3775fd3eca826012bef76d8c/requests-2.18.4-py2.py3-none-any.whl (88kB) 100% |████████████████████████████████| 92kB 3.6MB/s Requirement already satisfied (use --upgrade to upgrade): ipaddress>=1.0.16; python_version < "3.3" in /usr/lib/python2.7/site-packages (from docker==2.7.0) Requirement already satisfied (use --upgrade to upgrade): websocket-client>=0.32.0 in /usr/lib/python2.7/site-packages (from docker==2.7.0) Requirement already satisfied (use --upgrade to upgrade): six>=1.4.0 in /usr/lib/python2.7/site-packages (from docker==2.7.0) Collecting certifi>=2017.4.17 (from requests!=2.18.0,>=2.14.2->docker==2.7.0) Downloading https://files.pythonhosted.org/packages/7c/e6/92ad559b7192d846975fc916b65f667c7b8c3a32bea7372340bfe9a15fa5/certifi-2018.4.16-py2.py3-none-any.whl (150kB) 100% |████████████████████████████████| 153kB 3.1MB/s Collecting chardet<3.1.0,>=3.0.2 (from requests!=2.18.0,>=2.14.2->docker==2.7.0) Downloading https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl (133kB) 100% |████████████████████████████████| 143kB 3.4MB/s Collecting idna<2.7,>=2.5 (from requests!=2.18.0,>=2.14.2->docker==2.7.0) Downloading https://files.pythonhosted.org/packages/27/cc/6dd9a3869f15c2edfab863b992838277279ce92663d334df9ecf5106f5c6/idna-2.6-py2.py3-none-any.whl (56kB) 100% |████████████████████████████████| 61kB 4.6MB/s Collecting urllib3<1.23,>=1.21.1 (from requests!=2.18.0,>=2.14.2->docker==2.7.0) Downloading https://files.pythonhosted.org/packages/63/cb/6965947c13a94236f6d4b8223e21beb4d576dc72e8130bd7880f600839b8/urllib3-1.22-py2.py3-none-any.whl (132kB) 100% |████████████████████████████████| 133kB 4.0MB/s Installing collected packages: certifi, chardet, idna, urllib3, requests, docker Found existing installation: chardet 2.2.1 Uninstalling chardet-2.2.1: Successfully uninstalled chardet-2.2.1 Found existing installation: urllib3 1.10.2 Uninstalling urllib3-1.10.2: Successfully uninstalled urllib3-1.10.2 Found existing installation: requests 2.11.1 Uninstalling requests-2.11.1: Successfully uninstalled requests-2.11.1 Successfully installed certifi-2018.4.16 chardet-3.0.4 docker-2.7.0 idna-2.6 requests-2.18.4 urllib3-1.22 You are using pip version 8.1.2, however version 10.0.1 is available. You should consider upgrading via the 'pip install --upgrade pip' command.

but not luck, it fails at the first container

TASK [common : Starting fluentd container] 
**********************************************************************************************************************************************
fatal: [192.168.6.13]: FAILED! => {"changed": true, "msg": "'Traceback (most recent call last):\\n  File \"/tmp/ansible_MAbP41/ansible_module_kolla_docker.py\", line 801, in main\\n    result = bool(getattr(dw, module.params.get(\\'action\\'))())\\n  File \"/tmp/ansible_MAbP41/ansible_module_kolla_docker.py\", line 610, in start_container\\n    self.pull_image()\\n  File \"/tmp/ansible_MAbP41/ansible_module_kolla_docker.py\", line 452, in pull_image\\n    repository=image, tag=tag, stream=True\\n  File \"/usr/lib/python2.7/site-packages/docker/api/image.py\", line 381, in pull\\n    header = auth.get_config_header(self, registry)\\nAttributeError: \\'module\\' object has no attribute \\'get_config_header\\'\\n'"}
        to retry, use: --limit @/root/contrail-ansible-deployer/playbooks/install_contrail.retry

PLAY RECAP *******************************************************************************************************************************************************************************
192.168.6.12               : ok=54   changed=10   unreachable=0    failed=0
192.168.6.13               : ok=73   changed=14   unreachable=0    failed=1
localhost                  : ok=7    changed=2    unreachable=0    failed=0

[fclaudiopalmeira]

have you checked if the tag for the contrail version that you are using is the correct one at docker hub?what repository are you using?

[cecco16]

For the time being I am using the latest version of opencontrailnightly.

[fclaudiopalmeira]

could you post your instances.yml contents?

[cecco16]

Sure, enclosed the file I am using now.

By the way the behavior has been changeling:

1. 2 days ago: some container would start, but MariaDB on the controller node;
2. 1 days ago: manually installed docker on the controller VM using pip install docker==2.7.0, no container started (fluentd first to fail);
3. 1 day ago: cloned the ansible deployment tool to try from a green field, but the contrail installation playbook fails on the epel installation not being able to reach the repos. The centos repo for KVM (CentOS-7 - QEMU EV) is failing while running configure_instances.yml
4. 1 day ago: set to skip the CentOS-7 - QEMU EV on the VMs and we are back to square 2

TASK [common : Starting fluentd container] *********************************************************************************************************************************************** fatal: [192.168.6.13]: FAILED! => {"changed": true, "msg": "'Traceback (most recent call last):\\n File \"/tmp/ansible_hankWL/ansible_module_kolla_docker.py\", line 801, in main\\n result = bool(getattr(dw, module.params.get(\\'action\\'))())\\n File \"/tmp/ansible_hankWL/ansible_module_kolla_docker.py\", line 610, in start_container\\n self.pull_image()\\n File \"/tmp/ansible_hankWL/ansible_module_kolla_docker.py\", line 452, in pull_image\\n repository=image, tag=tag, stream=True\\n File \"/usr/lib/python2.7/site-packages/docker/api/image.py\", line 381, in pull\\n header = auth.get_config_header(self, registry)\\nAttributeError: \\'module\\' object has no attribute \\'get_config_header\\'\\n'"} to retry, use: --limit @/root/contrail-ansible-deployer/playbooks/install_contrail.retry

instances.yaml.txt

[cecco16]

The issue persist and mariadb keeps on restarting, only basic options on the instances.yaml are set and I also tried to deploy on a single host.

Looking at the log files of the mariadb container on the controller the following error comes up

[root@controller5 _data]# cat wsrep_recovery.wTFxME
180516 12:19:11 [Note] InnoDB: Using mutexes to ref count buffer pool pages
180516 12:19:11 [Note] InnoDB: The InnoDB memory heap is disabled
180516 12:19:11 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
180516 12:19:11 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
180516 12:19:11 [Note] InnoDB: Compressed tables use zlib 1.2.7
180516 12:19:11 [Note] InnoDB: Using Linux native AIO
180516 12:19:11 [Note] InnoDB: Using CPU crc32 instructions
180516 12:19:11 [Note] InnoDB: Initializing buffer pool, size = 4.5G
180516 12:19:11 [Note] InnoDB: Completed initialization of buffer pool
180516 12:19:11 [Note] InnoDB: Highest supported file format is Barracuda.
180516 12:19:11 [Note] InnoDB: 128 rollback segment(s) are active.
180516 12:19:11 [Note] InnoDB: Waiting for purge to start
180516 12:19:11 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.39-83.1 started; log sequence number 1617205
180516 12:19:11 [Warning] InnoDB: Skipping buffer pool dump/restore during wsrep recovery.
180516 12:19:11 [Note] Plugin 'FEEDBACK' is disabled.
180516 12:19:11 [Note] Server socket created on IP: '192.168.6.13'.
180516 12:19:11 [ERROR] Can't start server: listen() on TCP/IP port: Address already in use
180516 12:19:11 [ERROR] listen() on TCP/IP failed with error 98
180516 12:19:11 [ERROR] Aborting

180516 12:19:11 [Note] InnoDB: FTS optimize thread exiting.
180516 12:19:11 [Note] InnoDB: Starting shutdown...
180516 12:19:12 [Note] InnoDB: Waiting for page_cleaner to finish flushing of buffer pool
180516 12:19:14 [Note] InnoDB: Shutdown completed; log sequence number 1617215
180516 12:19:14 [Note] /usr/sbin/mysqld: Shutdown complete

listing the ports looks like haproxy is on that port, not sure if that is right considering I am not trying to deploy in HA, but just with a single controller node

[root@controller5 mysql]# ss -anp | grep 3306 tcp LISTEN 0 128 192.168.6.13:3306 *:* users:(("haproxy",pid=23029,fd=24))

[dmontagner]

I hit this issue before and I worked around it by changing the database port in the kolla group_vars/all:

[root@contrail1 contrail-ansible-deployer]# cat ../contrail-kolla-ansible/ansible/group_vars/all.yml | grep database_port
database_port: "3306"
mariadb_port: "{{ database_port }}"

However, if you are using the latest contrail and kolla deployers, and assuming you have installed the dependencies highlighted, you should not hit this problem.

[ashutosh-mishra]

We also hit with the same issue where MariaDB stuck at restarting. Tried above-mentioned steps, but didn't help, we need proper cleanup also. Mentioning steps which resolved the issue(In case someone needs) :

1. Stop all running containers
2. Remove all containers
3. Stop docker service
4. Upgrade/Downgrade docker to the specific version which specified above pip install docker==2.7.0
5. Cleanup /var/lib/docker (In case upgrade or downgrade Docker, existing Docker has to be erased and /var/lib/docker has to be removed. Otherwise, leftovers in /var/lib/docker (probably metadata) will cause issues.)
6. Start the docker service.

With above-mentioned steps, not observed any issue with MariaDB, and OpenStack installation succeded although still not able to deploy properly.

[Sofioni]

Hi, I've tried the above 6 steps, downgraded docker version to 2.7.0, but maria db service is still not getting started. Container is in restarting state all the time. Someone got it working?

[nicoaws]

same here

[ptech2]

I am having this issue as well. Is there a fix?

[dmontagner]

In many cases, this issue is related to the way you configured the different interfaces of the servers and which API endpoint IP is being used (internal vs external).

Check which IP is MariaDB exposing in its port and check what is the IP of the API calls trying to reach MariaDB.

On 14 Mar 2019, at 08:21, ptech2 notifications@github.com wrote:

I am having this issue as well. Is there a fix?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[cecco16]

Are you guys on version 4 or 5? I did not have the any of this with Contrail 5.

[edgartellez]

Same issue, I have followed all the steps mention above and nothin, do you have found some effective link or answer.

Thanks in advance !!

I am following the udemy Openstack Installation and deployment as maybe some of you !! :D

[AdilKhurshid]

I am facing this issue can anyone pl help me

ASK [mariadb : Cleaning up facts] ************************************************************
ok: [localhost]

TASK [mariadb : include_tasks] ****************************************************************
included: /usr/local/share/kolla-ansible/ansible/roles/mariadb/tasks/check.yml for localhost

TASK [mariadb : Waiting for MariaDB service to be ready through VIP] **************************
FAILED - RETRYING: Waiting for MariaDB service to be ready through VIP (6 retries left).
FAILED - RETRYING: Waiting for MariaDB service to be ready through VIP (5 retries left).
FAILED - RETRYING: Waiting for MariaDB service to be ready through VIP (4 retries left).
FAILED - RETRYING: Waiting for MariaDB service to be ready through VIP (3 retries left).
FAILED - RETRYING: Waiting for MariaDB service to be ready through VIP (2 retries left).
FAILED - RETRYING: Waiting for MariaDB service to be ready through VIP (1 retries left).
fatal: [localhost]: FAILED! => {"attempts": 6, "changed": false, "cmd": ["docker", "exec", "mariadb", "mysql", "-h", "10.10.10.254", "-P", "3306", "-u", "haproxy", "-e", "show databases;"], "delta": "0:00:03.345724", "end": "2019-07-30 11:27:33.625923", "msg": "non-zero return code", "rc": 1, "start": "2019-07-30 11:27:30.280199", "stderr": "ERROR 2003 (HY000): Can't connect to MySQL server on '10.10.10.254' (113 \"No route to host\")", "stderr_lines": ["ERROR 2003 (HY000): Can't connect to MySQL server on '10.10.10.254' (113 \"No route to host\")"], "stdout": "", "stdout_lines": []}
    to retry, use: --limit @/usr/local/share/kolla-ansible/ansible/site.retry

PLAY RECAP ************************************************************************************
localhost                  : ok=54   changed=0    unreachable=0    failed=1   

Command failed ansible-playbook -i ./all-in-one -e @/etc/kolla/globals.yml -e @/etc/kolla/passwords.yml -e CONFIG_DIR=/etc/kolla  -e kolla_action=deploy /usr/local/share/kolla-ansible/ansible/site.yml

[loverto]

you can try this command kolla-ansible -i multinode maridb_recover

[doubando]

I'm having the same issue, even everything working,

• issue resolved after updating DNS with the domain name on the controller nodes

[lianzhang132]

I'm having the same issue, even everything working,

• issue resolved after updating DNS with the domain name on the controller nodes

Can you elaborate on your solution steps?

[fangaofeng]

kolla-ansible , get this error

ESTABLISH SSH CONNECTION FOR USER: None SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/c95568e57f control01 '/bin/sh -c '"'"'echo ~ && sleep 0'"'"'' (0, b'/root\n', b'OpenSSH_8.2p1 Ubuntu-4ubuntu0.1, OpenSSL 1.1.1f 31 Mar 2020\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files\r\ndebug1: /etc/ssh/ssh_config line 21: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 84661\r\ndebug3: mux_client_request_session: session request sent\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n') ESTABLISH SSH CONNECTION FOR USER: None SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/c95568e57f control01 '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo /root/.ansible/tmp `"&& mkdir "` echo /root/.ansible/tmp/ansible-tmp-1614865621.5215487-87891-224217466937257 `" && echo ansible-tmp-1614865621.5215487-87891-224217466937257="` echo /root/.ansible/tmp/ansible-tmp-1614865621.5215487-87891-224217466937257 `" ) && sleep 0'"'"'' (0, b'ansible-tmp-1614865621.5215487-87891-224217466937257=/root/.ansible/tmp/ansible-tmp-1614865621.5215487-87891-224217466937257\n', b'OpenSSH_8.2p1 Ubuntu-4ubuntu0.1, OpenSSL 1.1.1f 31 Mar 2020\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files\r\ndebug1: /etc/ssh/ssh_config line 21: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 84661\r\ndebug3: mux_client_request_session: session request sent\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n') Using module file /home/kolla/venv/lib/python3.8/site-packages/ansible/modules/utilities/logic/wait_for.py PUT /root/.ansible/tmp/ansible-local-87070fb2kda5p/tmp84779mwu TO /root/.ansible/tmp/ansible-tmp-1614865621.5215487-87891-224217466937257/AnsiballZ_wait_for.py SSH: EXEC sftp -b - -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/c95568e57f '[control01]' (0, b'sftp> put /root/.ansible/tmp/ansible-local-87070fb2kda5p/tmp84779mwu /root/.ansible/tmp/ansible-tmp-1614865621.5215487-87891-224217466937257/AnsiballZ_wait_for.py\n', b'OpenSSH_8.2p1 Ubuntu-4ubuntu0.1, OpenSSL 1.1.1f 31 Mar 2020\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files\r\ndebug1: /etc/ssh/ssh_config line 21: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 84661\r\ndebug3: mux_client_request_session: session request sent\r\ndebug2: Remote version: 3\r\ndebug2: Server supports extension "posix-rename@openssh.com" revision 1\r\ndebug2: Server supports extension "statvfs@openssh.com" revision 2\r\ndebug2: Server supports extension "fstatvfs@openssh.com" revision 2\r\ndebug2: Server supports extension "hardlink@openssh.com" revision 1\r\ndebug2: Server supports extension "fsync@openssh.com" revision 1\r\ndebug2: Server supports extension "lsetstat@openssh.com" revision 1\r\ndebug3: Sent message fd 3 T:16 I:1\r\ndebug3: SSH_FXP_REALPATH . -> /root size 0\r\ndebug3: Looking up /root/.ansible/tmp/ansible-local-87070fb2kda5p/tmp84779mwu\r\ndebug3: Sent message fd 3 T:17 I:2\r\ndebug3: Received stat reply T:101 I:2\r\ndebug1: Couldn\'t stat remote file: No such file or directory\r\ndebug3: Sent message SSH2_FXP_OPEN I:3 P:/root/.ansible/tmp/ansible-tmp-1614865621.5215487-87891-224217466937257/AnsiballZ_wait_for.py\r\ndebug3: Sent message SSH2_FXP_WRITE I:4 O:0 S:32768\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 4 32768 bytes at 0\r\ndebug3: Sent message SSH2_FXP_WRITE I:5 O:32768 S:32768\r\ndebug3: Sent message SSH2_FXP_WRITE I:6 O:65536 S:32768\r\ndebug3: Sent message SSH2_FXP_WRITE I:7 O:98304 S:23918\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 5 32768 bytes at 32768\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 6 32768 bytes at 65536\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 7 23918 bytes at 98304\r\ndebug3: Sent message SSH2_FXP_CLOSE I:4\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n') ESTABLISH SSH CONNECTION FOR USER: None SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/c95568e57f control01 '/bin/sh -c '"'"'chmod u+x /root/.ansible/tmp/ansible-tmp-1614865621.5215487-87891-224217466937257/ /root/.ansible/tmp/ansible-tmp-1614865621.5215487-87891-224217466937257/AnsiballZ_wait_for.py && sleep 0'"'"'' (0, b'', b'OpenSSH_8.2p1 Ubuntu-4ubuntu0.1, OpenSSL 1.1.1f 31 Mar 2020\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files\r\ndebug1: /etc/ssh/ssh_config line 21: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 84661\r\ndebug3: mux_client_request_session: session request sent\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n') ESTABLISH SSH CONNECTION FOR USER: None SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/c95568e57f -tt control01 '/bin/sh -c '"'"'/usr/bin/python3 /root/.ansible/tmp/ansible-tmp-1614865621.5215487-87891-224217466937257/AnsiballZ_wait_for.py && sleep 0'"'"'' (1, b'\r\n{"msg": "Timeout when waiting for search string MariaDB in 172.16.158.157:3306", "elapsed": 60, "failed": true, "invocation": {"module_args": {"host": "172.16.158.157", "port": 3306, "connect_timeout": 1, "timeout": 60, "search_regex": "MariaDB", "delay": 0, "active_connection_states": ["ESTABLISHED", "FIN_WAIT1", "FIN_WAIT2", "SYN_RECV", "SYN_SENT", "TIME_WAIT"], "state": "started", "sleep": 1, "path": null, "exclude_hosts": null, "msg": null}}}\r\n', b'OpenSSH_8.2p1 Ubuntu-4ubuntu0.1, OpenSSL 1.1.1f 31 Mar 2020\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files\r\ndebug1: /etc/ssh/ssh_config line 21: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 84661\r\ndebug3: mux_client_request_session: session request sent\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 1\r\nShared connection to control01 closed.\r\n') Failed to connect to the host via ssh: OpenSSH_8.2p1 Ubuntu-4ubuntu0.1, OpenSSL 1.1.1f 31 Mar 2020 debug1: Reading configuration data /etc/ssh/ssh_config debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files debug1: /etc/ssh/ssh_config line 21: Applying options for * debug1: auto-mux: Trying existing master debug2: fd 3 setting O_NONBLOCK debug2: mux_client_hello_exchange: master version 4 debug3: mux_client_forwards: request forwardings: 0 local, 0 remote debug3: mux_client_request_session: entering debug3: mux_client_request_alive: entering debug3: mux_client_request_alive: done pid = 84661 debug3: mux_client_request_session: session request sent debug3: mux_client_read_packet: read header failed: Broken pipe debug2: Received exit status from master 1 Shared connection to control01 closed. ESTABLISH SSH CONNECTION FOR USER: None SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/c95568e57f control01 '/bin/sh -c '"'"'rm -f -r /root/.ansible/tmp/ansible-tmp-1614865621.5215487-87891-224217466937257/ > /dev/null 2>&1 && sleep 0'"'"'' (0, b'', b'OpenSSH_8.2p1 Ubuntu-4ubuntu0.1, OpenSSL 1.1.1f 31 Mar 2020\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files\r\ndebug1: /etc/ssh/ssh_config line 21: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 84661\r\ndebug3: mux_client_request_session: session request sent\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n') fatal: [control01]: FAILED! => { "attempts": 10, "changed": false, "elapsed": 60, "invocation": { "module_args": { "active_connection_states": [ "ESTABLISHED", "FIN_WAIT1", "FIN_WAIT2", "SYN_RECV", "SYN_SENT", "TIME_WAIT" ], "connect_timeout": 1, "delay": 0, "exclude_hosts": null, "host": "172.16.158.157", "msg": null, "path": null, "port": 3306, "search_regex": "MariaDB", "sleep": 1, "state": "started", "timeout": 60 } }, "msg": "Timeout when waiting for search string MariaDB in 172.16.158.157:3306" }

[volehuy1998]

Anyone fix it?

[Abdiooa]

does anyone fix it?