ccollicutt / swiftacular

Deploy OpenStack Swift with Ansible and Vagrant
BSD 2-Clause "Simplified" License
47 stars 13 forks source link

Don't restart keystone on every run #23

Closed ccollicutt closed 10 years ago

ccollicutt commented 10 years ago

I don't remember why I'm restarting keystone on every run, but I shouldn't be.

However, right now that occasionally creates an error in the playbook run because keystone isn't listening.

TASK: [authentication | restart debian keystone] ****************************** 
changed: [swift-keystone-01]

TASK: [authentication | restart redhat keystone] ****************************** 
skipping: [swift-keystone-01]

TASK: [authentication | ensure keystone debian keystone is running] *********** 
ok: [swift-keystone-01]

TASK: [authentication | ensure redhat keystone is running] ******************** 
skipping: [swift-keystone-01]

TASK: [authentication | wait for keystone to come back up] ******************** 
ok: [swift-keystone-01]

TASK: [authentication | create keystone identity point] *********************** 
failed: [swift-keystone-01] => {"failed": true}
msg: Traceback (most recent call last):
  File "/home/vagrant/.ansible/tmp/ansible-tmp-1405260545.05-26645671583576/keystone_service", line 287, in main
    check_mode)
  File "/home/vagrant/.ansible/tmp/ansible-tmp-1405260545.05-26645671583576/keystone_service", line 216, in dispatch
    check_mode)
  File "/home/vagrant/.ansible/tmp/ansible-tmp-1405260545.05-26645671583576/keystone_service", line 152, in ensure_service_present
    description=description)
  File "/usr/lib/python2.7/dist-packages/keystoneclient/v2_0/services.py", line 43, in create
    return self._create("/OS-KSADM/services", body, "OS-KSADM:service")
  File "/usr/lib/python2.7/dist-packages/keystoneclient/base.py", line 142, in _create
    return self._post(url, body, response_key, return_raw)
  File "/usr/lib/python2.7/dist-packages/keystoneclient/base.py", line 155, in _post
    resp, body = self.client.post(url, body=body)
  File "/usr/lib/python2.7/dist-packages/keystoneclient/httpclient.py", line 596, in post
    return self._cs_request(url, 'POST', **kwargs)
  File "/usr/lib/python2.7/dist-packages/keystoneclient/httpclient.py", line 582, in _cs_request
    return self.request(url, method, **kwargs)
  File "/usr/lib/python2.7/dist-packages/keystoneclient/httpclient.py", line 564, in request
    resp = super(HTTPClient, self).request(url, method, **kwargs)
  File "/usr/lib/python2.7/dist-packages/keystoneclient/baseclient.py", line 21, in request
    return self.session.request(url, method, **kwargs)
  File "/usr/lib/python2.7/dist-packages/keystoneclient/utils.py", line 318, in inner
    return func(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/keystoneclient/session.py", line 242, in request
    resp = self._send_request(url, method, redirect, **kwargs)
  File "/usr/lib/python2.7/dist-packages/keystoneclient/session.py", line 271, in _send_request
    raise exceptions.ConnectionError(msg)
ConnectionError: Unable to establish connection to https://127.0.0.1:35357/v2.0/OS-KSADM/services

FATAL: all hosts have already failed -- aborting

I think it's because a connection is still open. Note the 192.168.100.50:35357 192.168.100.50:48032 CLOSE_WAIT.

vagrant@swift-keystone-01:~$ netstat -ant
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State      
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:5000            0.0.0.0:*               LISTEN     
tcp        0      0 127.0.0.1:3306          0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:56403           0.0.0.0:*               LISTEN     
tcp        1      0 192.168.100.50:35357    192.168.100.50:48032    CLOSE_WAIT 
tcp        0      0 192.168.100.50:22       192.168.100.1:51909     ESTABLISHED
tcp        0      0 10.0.2.15:22            10.0.2.2:52119          ESTABLISHED
tcp        0      0 192.168.100.50:48032    192.168.100.50:35357    FIN_WAIT2  
tcp6       0      0 :::22                   :::*                    LISTEN     
tcp6       0      0 :::111                  :::*                    LISTEN     
tcp6       0      0 :::37299                :::*                    LISTEN  
ccollicutt commented 10 years ago

Actually it looks like a bug, because there are two restarts, one for debian and one for redhat but the when criteria didn't include "keystone.changed" for the debian restart whereas the redhat restart did.

ccollicutt commented 10 years ago

And it may also be some kind of bug in keystone?

(keystone.openstack.common.versionutils): 2014-07-13 08:18:18,045 WARNING Deprecated: keystone.middleware.core.XmlBodyMiddleware is deprecated as of Icehouse in favor of support for "application/json" only and may be removed in K.
(keystone.common.environment.eventlet_server): 2014-07-13 08:20:54,144 ERROR Server error
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/keystone/common/environment/eventlet_server.py", line 121, in _run
    log=log.WritableLogger(logger), debug=False)
  File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", line 671, in server
    client_socket = sock.accept()
  File "/usr/lib/python2.7/dist-packages/eventlet/green/ssl.py", line 279, in accept
    suppress_ragged_eofs=self.suppress_ragged_eofs)
  File "/usr/lib/python2.7/dist-packages/eventlet/green/ssl.py", line 46, in __init__
    super(GreenSSLSocket, self).__init__(sock.fd, *args, **kw)
  File "/usr/lib/python2.7/ssl.py", line 241, in __init__
    ciphers)
SSLError: [Errno 336265218] _ssl.c:355: error:140B0002:SSL routines:SSL_CTX_use_PrivateKey_file:system lib
ccollicutt commented 10 years ago

Actually it was a bug, typo. The when check was not checking the return code, just if keystone_pem_missing > 0. oof.

- name: remove /etc/keystone/ssl if there is not a keystone.pem file
  file: path=/etc/keystone/ssl state=absent
  when: keystone_pem_missing.rc > 0
  changed_when: keystone_pem_missing.rc > 0