search

openstack-charmers / zaza-openstack-tests

OpenStack Charms Functional Test Library for Zaza
Apache License 2.0
7 stars 77 forks source link

Keystone connection aborted during cinder-backup test. #322

Open ajkavanagh opened 4 years ago

ajkavanagh commented 4 years ago

Essentially, either it was a timeout issue, or the server went away during the connection. Either way, it's an unstable test:

2020-06-05 16:57:27 [INFO] test_410_cinder_vol_create_backup_delete_restore_pool_inspect (zaza.openstack.charm_tests.cinder_backup.tests.CinderBackupTest)
2020-06-05 16:57:27 [INFO] Create, backup, delete, restore a ceph-backed cinder volume.
2020-06-05 16:57:27 [INFO]  ... 
2020-06-05 16:57:29 [INFO] Checking ceph cinder pool original samples...
2020-06-05 16:57:30 [INFO] creating
2020-06-05 16:57:32 [INFO] available
2020-06-05 16:57:32 [INFO] creating
2020-06-05 16:57:33 [INFO] creating
2020-06-05 16:57:35 [INFO] creating
2020-06-05 16:57:39 [INFO] available
2020-06-05 16:57:56 [INFO] restoring
2020-06-05 16:57:57 [INFO] restoring
2020-06-05 16:57:59 [INFO] restoring
2020-06-05 16:58:03 [INFO] available
2020-06-05 16:58:19 [INFO] Checking ceph cinder pool samples after volume create...
2020-06-05 16:58:24 [INFO] ERROR
2020-06-05 16:58:24 [INFO] ======================================================================
2020-06-05 16:58:24 [INFO] ERROR: test_410_cinder_vol_create_backup_delete_restore_pool_inspect (zaza.openstack.charm_tests.cinder_backup.tests.CinderBackupTest)
2020-06-05 16:58:24 [INFO] Create, backup, delete, restore a ceph-backed cinder volume.
2020-06-05 16:58:24 [INFO] ----------------------------------------------------------------------
2020-06-05 16:58:24 [INFO] Traceback (most recent call last):
2020-06-05 16:58:24 [INFO]   File "/tmp/tmp.v6WpcSKxEY/func/lib/python3.5/site-packages/urllib3/connectionpool.py", line 677, in urlopen
2020-06-05 16:58:24 [INFO]     chunked=chunked,
2020-06-05 16:58:24 [INFO]   File "/tmp/tmp.v6WpcSKxEY/func/lib/python3.5/site-packages/urllib3/connectionpool.py", line 426, in _make_request
2020-06-05 16:58:24 [INFO]     six.raise_from(e, None)
2020-06-05 16:58:24 [INFO]   File "<string>", line 3, in raise_from
2020-06-05 16:58:24 [INFO]   File "/tmp/tmp.v6WpcSKxEY/func/lib/python3.5/site-packages/urllib3/connectionpool.py", line 421, in _make_request
2020-06-05 16:58:24 [INFO]     httplib_response = conn.getresponse()
2020-06-05 16:58:24 [INFO]   File "/usr/lib/python3.5/http/client.py", line 1225, in getresponse
2020-06-05 16:58:24 [INFO]     response.begin()
2020-06-05 16:58:24 [INFO]   File "/usr/lib/python3.5/http/client.py", line 307, in begin
2020-06-05 16:58:24 [INFO]     version, status, reason = self._read_status()
2020-06-05 16:58:24 [INFO]   File "/usr/lib/python3.5/http/client.py", line 276, in _read_status
2020-06-05 16:58:24 [INFO]     raise RemoteDisconnected("Remote end closed connection without"
2020-06-05 16:58:24 [INFO] http.client.RemoteDisconnected: Remote end closed connection without response
2020-06-05 16:58:24 [INFO] During handling of the above exception, another exception occurred:
2020-06-05 16:58:24 [INFO] Traceback (most recent call last):
2020-06-05 16:58:24 [INFO]   File "/tmp/tmp.v6WpcSKxEY/func/lib/python3.5/site-packages/requests/adapters.py", line 449, in send
2020-06-05 16:58:24 [INFO]     timeout=timeout
2020-06-05 16:58:24 [INFO]   File "/tmp/tmp.v6WpcSKxEY/func/lib/python3.5/site-packages/urllib3/connectionpool.py", line 725, in urlopen
2020-06-05 16:58:24 [INFO]     method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
2020-06-05 16:58:24 [INFO]   File "/tmp/tmp.v6WpcSKxEY/func/lib/python3.5/site-packages/urllib3/util/retry.py", line 403, in increment
2020-06-05 16:58:24 [INFO]     raise six.reraise(type(error), error, _stacktrace)
2020-06-05 16:58:24 [INFO]   File "/tmp/tmp.v6WpcSKxEY/func/lib/python3.5/site-packages/urllib3/packages/six.py", line 734, in reraise
2020-06-05 16:58:24 [INFO]     raise value.with_traceback(tb)
2020-06-05 16:58:24 [INFO]   File "/tmp/tmp.v6WpcSKxEY/func/lib/python3.5/site-packages/urllib3/connectionpool.py", line 677, in urlopen
2020-06-05 16:58:24 [INFO]     chunked=chunked,
2020-06-05 16:58:24 [INFO]   File "/tmp/tmp.v6WpcSKxEY/func/lib/python3.5/site-packages/urllib3/connectionpool.py", line 426, in _make_request
2020-06-05 16:58:24 [INFO]     six.raise_from(e, None)
2020-06-05 16:58:24 [INFO]   File "<string>", line 3, in raise_from
2020-06-05 16:58:24 [INFO]   File "/tmp/tmp.v6WpcSKxEY/func/lib/python3.5/site-packages/urllib3/connectionpool.py", line 421, in _make_request
2020-06-05 16:58:24 [INFO]     httplib_response = conn.getresponse()
2020-06-05 16:58:24 [INFO]   File "/usr/lib/python3.5/http/client.py", line 1225, in getresponse
2020-06-05 16:58:24 [INFO]     response.begin()
2020-06-05 16:58:24 [INFO]   File "/usr/lib/python3.5/http/client.py", line 307, in begin
2020-06-05 16:58:24 [INFO]     version, status, reason = self._read_status()
2020-06-05 16:58:24 [INFO]   File "/usr/lib/python3.5/http/client.py", line 276, in _read_status
2020-06-05 16:58:24 [INFO]     raise RemoteDisconnected("Remote end closed connection without"
2020-06-05 16:58:24 [INFO] urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',))
2020-06-05 16:58:24 [INFO] During handling of the above exception, another exception occurred:
2020-06-05 16:58:24 [INFO] Traceback (most recent call last):
2020-06-05 16:58:24 [INFO]   File "/tmp/tmp.v6WpcSKxEY/func/lib/python3.5/site-packages/keystoneauth1/session.py", line 1004, in _send_request
2020-06-05 16:58:24 [INFO]     resp = self.session.request(method, url, **kwargs)
2020-06-05 16:58:24 [INFO]   File "/tmp/tmp.v6WpcSKxEY/func/lib/python3.5/site-packages/requests/sessions.py", line 530, in request
2020-06-05 16:58:24 [INFO]     resp = self.send(prep, **send_kwargs)
2020-06-05 16:58:24 [INFO]   File "/tmp/tmp.v6WpcSKxEY/func/lib/python3.5/site-packages/requests/sessions.py", line 643, in send
2020-06-05 16:58:24 [INFO]     r = adapter.send(request, **kwargs)
2020-06-05 16:58:24 [INFO]   File "/tmp/tmp.v6WpcSKxEY/func/lib/python3.5/site-packages/requests/adapters.py", line 498, in send
2020-06-05 16:58:24 [INFO]     raise ConnectionError(err, request=request)
2020-06-05 16:58:24 [INFO] requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',))
2020-06-05 16:58:24 [INFO] During handling of the above exception, another exception occurred:
2020-06-05 16:58:24 [INFO] Traceback (most recent call last):
2020-06-05 16:58:24 [INFO]   File "/tmp/tmp.v6WpcSKxEY/func/lib/python3.5/site-packages/zaza/openstack/charm_tests/cinder_backup/tests.py", line 147, in test_410_cinder_vol_create_backup_delete_restore_pool_inspect
2020-06-05 16:58:24 [INFO]     vols = self.cinder_client.volumes.list()
2020-06-05 16:58:24 [INFO]   File "/tmp/tmp.v6WpcSKxEY/func/lib/python3.5/site-packages/cinderclient/v2/volumes.py", line 300, in list
2020-06-05 16:58:24 [INFO]     return self._list(url, resource_type, limit=limit)
2020-06-05 16:58:24 [INFO]   File "/tmp/tmp.v6WpcSKxEY/func/lib/python3.5/site-packages/cinderclient/base.py", line 80, in _list
2020-06-05 16:58:24 [INFO]     resp, body = self.api.client.get(url)
2020-06-05 16:58:24 [INFO]   File "/tmp/tmp.v6WpcSKxEY/func/lib/python3.5/site-packages/cinderclient/client.py", line 214, in get
2020-06-05 16:58:24 [INFO]     return self._cs_request(url, 'GET', **kwargs)
2020-06-05 16:58:24 [INFO]   File "/tmp/tmp.v6WpcSKxEY/func/lib/python3.5/site-packages/cinderclient/client.py", line 205, in _cs_request
2020-06-05 16:58:24 [INFO]     return self.request(url, method, **kwargs)
2020-06-05 16:58:24 [INFO]   File "/tmp/tmp.v6WpcSKxEY/func/lib/python3.5/site-packages/cinderclient/client.py", line 188, in request
2020-06-05 16:58:24 [INFO]     **kwargs)
2020-06-05 16:58:24 [INFO]   File "/tmp/tmp.v6WpcSKxEY/func/lib/python3.5/site-packages/keystoneauth1/adapter.py", line 545, in request
2020-06-05 16:58:24 [INFO]     resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
2020-06-05 16:58:24 [INFO]   File "/tmp/tmp.v6WpcSKxEY/func/lib/python3.5/site-packages/keystoneauth1/adapter.py", line 248, in request
2020-06-05 16:58:24 [INFO]     return self.session.request(url, method, **kwargs)
2020-06-05 16:58:24 [INFO]   File "/tmp/tmp.v6WpcSKxEY/func/lib/python3.5/site-packages/keystoneauth1/session.py", line 913, in request
2020-06-05 16:58:24 [INFO]     resp = send(**kwargs)
2020-06-05 16:58:24 [INFO]   File "/tmp/tmp.v6WpcSKxEY/func/lib/python3.5/site-packages/keystoneauth1/session.py", line 1020, in _send_request
2020-06-05 16:58:24 [INFO]     raise exceptions.ConnectFailure(msg)
2020-06-05 16:58:24 [INFO] keystoneauth1.exceptions.connection.ConnectFailure: Unable to establish connection to http://172.17.112.4:8776/v2/ffd9b06fb14b493d8c4b4d50ed496ab6/volumes/detail: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',))
lourot commented 4 years ago

Hitting this also in keystone tests: https://review.opendev.org/#/c/729781

Can't reproduce on my bastion after 10+ attempts!