Open cecco16 opened 6 years ago
fclaudiopalmeira
commented
6 years ago Hello My Friend, this happens due to the newer docker version incompatibility with the playbooks, the solution is to execute the following command, before attempting to deploy the containers? pip install docker==2.7.0
Cheers m8.
cecco16
commented
6 years ago Hi there,
thanks for tor the tip. Tried to install the right version right before running the playbooks/install_contrail.yml
[root@controller5-2 ~]# pip install docker==2.7.0 Collecting docker==2.7.0 Downloading https://files.pythonhosted.org/packages/6b/01/74d61c7de9fb99cebbbfab62ee5c88a9b23e8b1c25a57eded5fc22e87bd2/docker-2.7.0-py2.py3-none-any.whl (119kB) 100% |████████████████████████████████| 122kB 3.7MB/s Requirement already satisfied (use --upgrade to upgrade): backports.ssl-match-hostname>=3.5; python_version < "3.5" in /usr/lib/python2.7/site-packages (from docker==2.7.0) Requirement already satisfied (use --upgrade to upgrade): docker-pycreds>=0.2.1 in /usr/lib/python2.7/site-packages (from docker==2.7.0) Collecting requests!=2.18.0,>=2.14.2 (from docker==2.7.0) Downloading https://files.pythonhosted.org/packages/49/df/50aa1999ab9bde74656c2919d9c0c085fd2b3775fd3eca826012bef76d8c/requests-2.18.4-py2.py3-none-any.whl (88kB) 100% |████████████████████████████████| 92kB 3.6MB/s Requirement already satisfied (use --upgrade to upgrade): ipaddress>=1.0.16; python_version < "3.3" in /usr/lib/python2.7/site-packages (from docker==2.7.0) Requirement already satisfied (use --upgrade to upgrade): websocket-client>=0.32.0 in /usr/lib/python2.7/site-packages (from docker==2.7.0) Requirement already satisfied (use --upgrade to upgrade): six>=1.4.0 in /usr/lib/python2.7/site-packages (from docker==2.7.0) Collecting certifi>=2017.4.17 (from requests!=2.18.0,>=2.14.2->docker==2.7.0) Downloading https://files.pythonhosted.org/packages/7c/e6/92ad559b7192d846975fc916b65f667c7b8c3a32bea7372340bfe9a15fa5/certifi-2018.4.16-py2.py3-none-any.whl (150kB) 100% |████████████████████████████████| 153kB 3.1MB/s Collecting chardet<3.1.0,>=3.0.2 (from requests!=2.18.0,>=2.14.2->docker==2.7.0) Downloading https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl (133kB) 100% |████████████████████████████████| 143kB 3.4MB/s Collecting idna<2.7,>=2.5 (from requests!=2.18.0,>=2.14.2->docker==2.7.0) Downloading https://files.pythonhosted.org/packages/27/cc/6dd9a3869f15c2edfab863b992838277279ce92663d334df9ecf5106f5c6/idna-2.6-py2.py3-none-any.whl (56kB) 100% |████████████████████████████████| 61kB 4.6MB/s Collecting urllib3<1.23,>=1.21.1 (from requests!=2.18.0,>=2.14.2->docker==2.7.0) Downloading https://files.pythonhosted.org/packages/63/cb/6965947c13a94236f6d4b8223e21beb4d576dc72e8130bd7880f600839b8/urllib3-1.22-py2.py3-none-any.whl (132kB) 100% |████████████████████████████████| 133kB 4.0MB/s Installing collected packages: certifi, chardet, idna, urllib3, requests, docker Found existing installation: chardet 2.2.1 Uninstalling chardet-2.2.1: Successfully uninstalled chardet-2.2.1 Found existing installation: urllib3 1.10.2 Uninstalling urllib3-1.10.2: Successfully uninstalled urllib3-1.10.2 Found existing installation: requests 2.11.1 Uninstalling requests-2.11.1: Successfully uninstalled requests-2.11.1 Successfully installed certifi-2018.4.16 chardet-3.0.4 docker-2.7.0 idna-2.6 requests-2.18.4 urllib3-1.22 You are using pip version 8.1.2, however version 10.0.1 is available. You should consider upgrading via the 'pip install --upgrade pip' command.
but not luck, it fails at the first container
TASK [common : Starting fluentd container]
**********************************************************************************************************************************************
fatal: [192.168.6.13]: FAILED! => {"changed": true, "msg": "'Traceback (most recent call last):\\n File \"/tmp/ansible_MAbP41/ansible_module_kolla_docker.py\", line 801, in main\\n result = bool(getattr(dw, module.params.get(\\'action\\'))())\\n File \"/tmp/ansible_MAbP41/ansible_module_kolla_docker.py\", line 610, in start_container\\n self.pull_image()\\n File \"/tmp/ansible_MAbP41/ansible_module_kolla_docker.py\", line 452, in pull_image\\n repository=image, tag=tag, stream=True\\n File \"/usr/lib/python2.7/site-packages/docker/api/image.py\", line 381, in pull\\n header = auth.get_config_header(self, registry)\\nAttributeError: \\'module\\' object has no attribute \\'get_config_header\\'\\n'"}
to retry, use: --limit @/root/contrail-ansible-deployer/playbooks/install_contrail.retry
PLAY RECAP *******************************************************************************************************************************************************************************
192.168.6.12 : ok=54 changed=10 unreachable=0 failed=0
192.168.6.13 : ok=73 changed=14 unreachable=0 failed=1
localhost : ok=7 changed=2 unreachable=0 failed=0have you checked if the tag for the contrail version that you are using is the correct one at docker hub?what repository are you using?
cecco16
commented
6 years ago For the time being I am using the latest version of opencontrailnightly.
fclaudiopalmeira
commented
6 years ago could you post your instances.yml contents?
cecco16
commented
6 years ago Sure, enclosed the file I am using now.
By the way the behavior has been changeling:
TASK [common : Starting fluentd container] *********************************************************************************************************************************************** fatal: [192.168.6.13]: FAILED! => {"changed": true, "msg": "'Traceback (most recent call last):\\n File \"/tmp/ansible_hankWL/ansible_module_kolla_docker.py\", line 801, in main\\n result = bool(getattr(dw, module.params.get(\\'action\\'))())\\n File \"/tmp/ansible_hankWL/ansible_module_kolla_docker.py\", line 610, in start_container\\n self.pull_image()\\n File \"/tmp/ansible_hankWL/ansible_module_kolla_docker.py\", line 452, in pull_image\\n repository=image, tag=tag, stream=True\\n File \"/usr/lib/python2.7/site-packages/docker/api/image.py\", line 381, in pull\\n header = auth.get_config_header(self, registry)\\nAttributeError: \\'module\\' object has no attribute \\'get_config_header\\'\\n'"} to retry, use: --limit @/root/contrail-ansible-deployer/playbooks/install_contrail.retry
cecco16
commented
6 years ago The issue persist and mariadb keeps on restarting, only basic options on the instances.yaml are set and I also tried to deploy on a single host.
Looking at the log files of the mariadb container on the controller the following error comes up
[root@controller5 _data]# cat wsrep_recovery.wTFxME
180516 12:19:11 [Note] InnoDB: Using mutexes to ref count buffer pool pages
180516 12:19:11 [Note] InnoDB: The InnoDB memory heap is disabled
180516 12:19:11 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
180516 12:19:11 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
180516 12:19:11 [Note] InnoDB: Compressed tables use zlib 1.2.7
180516 12:19:11 [Note] InnoDB: Using Linux native AIO
180516 12:19:11 [Note] InnoDB: Using CPU crc32 instructions
180516 12:19:11 [Note] InnoDB: Initializing buffer pool, size = 4.5G
180516 12:19:11 [Note] InnoDB: Completed initialization of buffer pool
180516 12:19:11 [Note] InnoDB: Highest supported file format is Barracuda.
180516 12:19:11 [Note] InnoDB: 128 rollback segment(s) are active.
180516 12:19:11 [Note] InnoDB: Waiting for purge to start
180516 12:19:11 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.39-83.1 started; log sequence number 1617205
180516 12:19:11 [Warning] InnoDB: Skipping buffer pool dump/restore during wsrep recovery.
180516 12:19:11 [Note] Plugin 'FEEDBACK' is disabled.
180516 12:19:11 [Note] Server socket created on IP: '192.168.6.13'.
180516 12:19:11 [ERROR] Can't start server: listen() on TCP/IP port: Address already in use
180516 12:19:11 [ERROR] listen() on TCP/IP failed with error 98
180516 12:19:11 [ERROR] Aborting
180516 12:19:11 [Note] InnoDB: FTS optimize thread exiting.
180516 12:19:11 [Note] InnoDB: Starting shutdown...
180516 12:19:12 [Note] InnoDB: Waiting for page_cleaner to finish flushing of buffer pool
180516 12:19:14 [Note] InnoDB: Shutdown completed; log sequence number 1617215
180516 12:19:14 [Note] /usr/sbin/mysqld: Shutdown completelisting the ports looks like haproxy is on that port, not sure if that is right considering I am not trying to deploy in HA, but just with a single controller node
[root@controller5 mysql]# ss -anp | grep 3306 tcp LISTEN 0 128 192.168.6.13:3306 *:* users:(("haproxy",pid=23029,fd=24))
dmontagner
commented
6 years ago I hit this issue before and I worked around it by changing the database port in the kolla group_vars/all:
[root@contrail1 contrail-ansible-deployer]# cat ../contrail-kolla-ansible/ansible/group_vars/all.yml | grep database_port
database_port: "3306"
mariadb_port: "{{ database_port }}"However, if you are using the latest contrail and kolla deployers, and assuming you have installed the dependencies highlighted, you should not hit this problem.
ashutosh-mishra
commented
6 years ago We also hit with the same issue where MariaDB stuck at restarting. Tried above-mentioned steps, but didn't help, we need proper cleanup also. Mentioning steps which resolved the issue(In case someone needs) :
With above-mentioned steps, not observed any issue with MariaDB, and OpenStack installation succeded although still not able to deploy properly.
Sofioni
commented
6 years ago Hi, I've tried the above 6 steps, downgraded docker version to 2.7.0, but maria db service is still not getting started. Container is in restarting state all the time. Someone got it working?
nicoaws
commented
5 years ago same here
ptech2
commented
5 years ago I am having this issue as well. Is there a fix?
dmontagner
commented
5 years ago In many cases, this issue is related to the way you configured the different interfaces of the servers and which API endpoint IP is being used (internal vs external).
Check which IP is MariaDB exposing in its port and check what is the IP of the API calls trying to reach MariaDB.
On 14 Mar 2019, at 08:21, ptech2 notifications@github.com wrote:
I am having this issue as well. Is there a fix?
— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.
cecco16
commented
5 years ago Are you guys on version 4 or 5? I did not have the any of this with Contrail 5.
edgartellez
commented
5 years ago Same issue, I have followed all the steps mention above and nothin, do you have found some effective link or answer.
Thanks in advance !!
I am following the udemy Openstack Installation and deployment as maybe some of you !! :D
AdilKhurshid
commented
5 years ago I am facing this issue can anyone pl help me
ASK [mariadb : Cleaning up facts] ************************************************************
ok: [localhost]
TASK [mariadb : include_tasks] ****************************************************************
included: /usr/local/share/kolla-ansible/ansible/roles/mariadb/tasks/check.yml for localhost
TASK [mariadb : Waiting for MariaDB service to be ready through VIP] **************************
FAILED - RETRYING: Waiting for MariaDB service to be ready through VIP (6 retries left).
FAILED - RETRYING: Waiting for MariaDB service to be ready through VIP (5 retries left).
FAILED - RETRYING: Waiting for MariaDB service to be ready through VIP (4 retries left).
FAILED - RETRYING: Waiting for MariaDB service to be ready through VIP (3 retries left).
FAILED - RETRYING: Waiting for MariaDB service to be ready through VIP (2 retries left).
FAILED - RETRYING: Waiting for MariaDB service to be ready through VIP (1 retries left).
fatal: [localhost]: FAILED! => {"attempts": 6, "changed": false, "cmd": ["docker", "exec", "mariadb", "mysql", "-h", "10.10.10.254", "-P", "3306", "-u", "haproxy", "-e", "show databases;"], "delta": "0:00:03.345724", "end": "2019-07-30 11:27:33.625923", "msg": "non-zero return code", "rc": 1, "start": "2019-07-30 11:27:30.280199", "stderr": "ERROR 2003 (HY000): Can't connect to MySQL server on '10.10.10.254' (113 \"No route to host\")", "stderr_lines": ["ERROR 2003 (HY000): Can't connect to MySQL server on '10.10.10.254' (113 \"No route to host\")"], "stdout": "", "stdout_lines": []}
to retry, use: --limit @/usr/local/share/kolla-ansible/ansible/site.retry
PLAY RECAP ************************************************************************************
localhost : ok=54 changed=0 unreachable=0 failed=1
Command failed ansible-playbook -i ./all-in-one -e @/etc/kolla/globals.yml -e @/etc/kolla/passwords.yml -e CONFIG_DIR=/etc/kolla -e kolla_action=deploy /usr/local/share/kolla-ansible/ansible/site.yml
loverto
commented
5 years ago you can try this command
kolla-ansible -i multinode maridb_recover
doubando
commented
3 years ago I'm having the same issue, even everything working,
lianzhang132
commented
3 years ago I'm having the same issue, even everything working,
- issue resolved after updating DNS with the domain name on the controller nodes
Can you elaborate on your solution steps?
fangaofeng
commented
3 years ago kolla-ansible , get this error
volehuy1998
commented
1 year ago Anyone fix it?
`TASK [mariadb : Waiting for MariaDB service to be ready] ***** FAILED - RETRYING: Waiting for MariaDB service to be ready (10 retries left). FAILED - RETRYING: Waiting for MariaDB service to be ready (9 retries left). FAILED - RETRYING: Waiting for MariaDB service to be ready (8 retries left). FAILED - RETRYING: Waiting for MariaDB service to be ready (7 retries left). FAILED - RETRYING: Waiting for MariaDB service to be ready (6 retries left). FAILED - RETRYING: Waiting for MariaDB service to be ready (5 retries left). FAILED - RETRYING: Waiting for MariaDB service to be ready (4 retries left). FAILED - RETRYING: Waiting for MariaDB service to be ready (3 retries left). FAILED - RETRYING: Waiting for MariaDB service to be ready (2 retries left). FAILED - RETRYING: Waiting for MariaDB service to be ready (1 retries left). fatal: [192.168.6.13]: FAILED! => {"attempts": 10, "changed": false, "elapsed": 60, "msg": "Timeout when waiting for search string MariaDB in 192.168.6.13:3306"} to retry, use: --limit @/root/contrail-ansible-deployer/playbooks/install_contrail.retry
PLAY RECAP *** 192.168.6.12 : ok=55 changed=21 unreachable=0 failed=0 192.168.6.13 : ok=118 changed=55 unreachable=0 failed=1 localhost : ok=7 changed=2 unreachable=0 failed=0
[root@controller5 db]# docker logs --tail 50 --follow --timestamps mariadb 2018-05-09T16:16:49.792265592Z INFO:main:Setting permission for /var/lib/mysql/performance_schema/events_waits_summary_by_host_by_event_name.frm 2018-05-09T16:16:49.792271446Z INFO:main:Setting permission for /var/lib/mysql/performance_schema/events_waits_summary_by_user_by_event_name.frm 2018-05-09T16:16:49.792279855Z INFO:main:Setting permission for /var/lib/mysql/performance_schema/events_waits_summary_by_account_by_event_name.frm 2018-05-09T16:16:49.792286029Z INFO:main:Setting permission for /var/lib/mysql/performance_schema/events_waits_summary_by_thread_by_event_name.frm 2018-05-09T16:16:49.792291929Z INFO:main:Setting permission for /var/lib/mysql/performance_schema/events_waits_summary_global_by_event_name.frm 2018-05-09T16:16:49.792297786Z INFO:main:Setting permission for /var/lib/mysql/performance_schema/file_instances.frm 2018-05-09T16:16:49.792303566Z INFO:main:Setting permission for /var/lib/mysql/performance_schema/file_summary_by_event_name.frm 2018-05-09T16:16:49.792322838Z INFO:main:Setting permission for /var/lib/mysql/performance_schema/file_summary_by_instance.frm 2018-05-09T16:16:49.792328799Z INFO:main:Setting permission for /var/lib/mysql/performance_schema/socket_instances.frm 2018-05-09T16:16:49.792334582Z INFO:main:Setting permission for /var/lib/mysql/performance_schema/socket_summary_by_instance.frm 2018-05-09T16:16:49.792340462Z INFO:main:Setting permission for /var/lib/mysql/performance_schema/socket_summary_by_event_name.frm 2018-05-09T16:16:49.792346235Z INFO:main:Setting permission for /var/lib/mysql/performance_schema/host_cache.frm 2018-05-09T16:16:49.792351929Z INFO:main:Setting permission for /var/lib/mysql/performance_schema/mutex_instances.frm 2018-05-09T16:16:49.792357692Z INFO:main:Setting permission for /var/lib/mysql/performance_schema/objects_summary_global_by_type.frm 2018-05-09T16:16:49.792363482Z INFO:main:Setting permission for /var/lib/mysql/performance_schema/performance_timers.frm 2018-05-09T16:16:49.792369255Z INFO:main:Setting permission for /var/lib/mysql/performance_schema/rwlock_instances.frm 2018-05-09T16:16:49.792374965Z INFO:main:Setting permission for /var/lib/mysql/performance_schema/setup_actors.frm 2018-05-09T16:16:49.792380682Z INFO:main:Setting permission for /var/lib/mysql/performance_schema/setup_consumers.frm 2018-05-09T16:16:49.792386412Z INFO:main:Setting permission for /var/lib/mysql/performance_schema/setup_instruments.frm 2018-05-09T16:16:49.792392750Z INFO:main:Setting permission for /var/lib/mysql/performance_schema/setup_objects.frm 2018-05-09T16:16:49.792398764Z INFO:main:Setting permission for /var/lib/mysql/performance_schema/setup_timers.frm 2018-05-09T16:16:49.792404584Z INFO:main:Setting permission for /var/lib/mysql/performance_schema/table_io_waits_summary_by_index_usage.frm 2018-05-09T16:16:49.792410574Z INFO:main:Setting permission for /var/lib/mysql/performance_schema/table_io_waits_summary_by_table.frm 2018-05-09T16:16:49.792417319Z INFO:main:Setting permission for /var/lib/mysql/performance_schema/table_lock_waits_summary_by_table.frm 2018-05-09T16:16:49.792423249Z INFO:main:Setting permission for /var/lib/mysql/performance_schema/threads.frm 2018-05-09T16:16:49.792428935Z INFO:main:Setting permission for /var/lib/mysql/performance_schema/events_stages_current.frm 2018-05-09T16:16:49.792434772Z INFO:main:Setting permission for /var/lib/mysql/performance_schema/events_stages_history.frm 2018-05-09T16:16:49.792440539Z INFO:main:Setting permission for /var/lib/mysql/performance_schema/events_stages_history_long.frm 2018-05-09T16:16:49.792446292Z INFO:main:Setting permission for /var/lib/mysql/performance_schema/events_stages_summary_by_thread_by_event_name.frm 2018-05-09T16:16:49.792452202Z INFO:main:Setting permission for /var/lib/mysql/performance_schema/events_stages_summary_by_host_by_event_name.frm 2018-05-09T16:16:49.792458026Z INFO:main:Setting permission for /var/lib/mysql/performance_schema/events_stages_summary_by_user_by_event_name.frm 2018-05-09T16:16:49.792467704Z INFO:main:Setting permission for /var/lib/mysql/performance_schema/events_stages_summary_by_account_by_event_name.frm 2018-05-09T16:16:49.792473801Z INFO:main:Setting permission for /var/lib/mysql/performance_schema/events_stages_summary_global_by_event_name.frm 2018-05-09T16:16:49.792479714Z INFO:main:Setting permission for /var/lib/mysql/performance_schema/events_statements_current.frm 2018-05-09T16:16:49.792485498Z INFO:main:Setting permission for /var/lib/mysql/performance_schema/events_statements_history.frm 2018-05-09T16:16:49.792491318Z INFO:main:Setting permission for /var/lib/mysql/performance_schema/events_statements_history_long.frm 2018-05-09T16:16:49.792497121Z INFO:main:Setting permission for /var/lib/mysql/performance_schema/events_statements_summary_by_thread_by_event_name.frm 2018-05-09T16:16:49.792503011Z INFO:main:Setting permission for /var/lib/mysql/performance_schema/events_statements_summary_by_host_by_event_name.frm 2018-05-09T16:16:49.792508941Z INFO:main:Setting permission for /var/lib/mysql/performance_schema/events_statements_summary_by_user_by_event_name.frm 2018-05-09T16:16:49.792514852Z INFO:main:Setting permission for /var/lib/mysql/performance_schema/events_statements_summary_by_account_by_event_name.frm 2018-05-09T16:16:49.792520782Z INFO:main:Setting permission for /var/lib/mysql/performance_schema/events_statements_summary_global_by_event_name.frm 2018-05-09T16:16:49.792527837Z INFO:main:Setting permission for /var/lib/mysql/performance_schema/hosts.frm 2018-05-09T16:16:49.792533703Z INFO:main:Setting permission for /var/lib/mysql/performance_schema/users.frm 2018-05-09T16:16:49.792539477Z INFO:main:Setting permission for /var/lib/mysql/performance_schema/accounts.frm 2018-05-09T16:16:49.792545177Z INFO:main:Setting permission for /var/lib/mysql/performance_schema/events_statements_summary_by_digest.frm 2018-05-09T16:16:49.792551033Z INFO:main:Setting permission for /var/lib/mysql/performance_schema/session_connect_attrs.frm 2018-05-09T16:16:49.792556920Z INFO:main:Setting permission for /var/lib/mysql/performance_schema/session_account_connect_attrs.frm 2018-05-09T16:16:49.808016993Z Running command: '/usr/bin/mysqld_safe' 2018-05-09T16:16:49.978048481Z 180509 16:16:49 mysqld_safe Logging to '/var/log/kolla/mariadb/mariadb.log'. 2018-05-09T16:16:50.028770833Z 180509 16:16:50 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql/ `