search

openstack-charmers / zaza-openstack-tests

OpenStack Charms Functional Test Library for Zaza
Apache License 2.0
7 stars 77 forks source link

Attaching additional port to instance: novaclient.exceptions.ClientException: Unexpected API Error #768

Open freyes opened 2 years ago

freyes commented 2 years ago

when running a CI job the z-o-t test that adds a second nic to a instance may fail due to undercloud (transient) issues, so it would be appropriate to make the method create_additional_port_for_machines() idempotent and wrapped with a retry decorator.

Issue found at https://openstack-ci-reports.ubuntu.com/artifacts/afd/841979/1/check/ceph-focal-wallaby_nova/afd2b80/

Relevant stacktrace:

2022-05-19 11:00:50.939845 | focal-medium | 2022-05-19 11:00:50 [INFO] AUTH_URL: http://10.245.161.156:5000/v3, api_ver: 3
2022-05-19 11:00:50.940030 | focal-medium | 2022-05-19 11:00:50 [INFO] Using keystone API V3 (or later) for undercloud auth
2022-05-19 11:01:00.810025 | focal-medium | 2022-05-19 11:01:00 [INFO] looking at application: {'name': 'keystone', 'type': {'pkg': 'keystone', 'origin_setting': 'openstack-origin'}}
2022-05-19 11:01:08.272579 | focal-medium | 2022-05-19 11:01:08 [WARNING] Adding second interface for dataport to guest netplan for bionic-queens and later
2022-05-19 11:01:08.272842 | focal-medium | 2022-05-19 11:01:08 [INFO] Configuring network for OpenStack undercloud/provider
2022-05-19 11:01:26.640403 | focal-medium | 2022-05-19 11:01:26 [INFO] Attaching additional port to instance ("f6959d28-95a3-45fa-b96a-089cf4435a9a"), connected to net id: 94a87d6a-6d82-4931-878d-aedcf0e62962
2022-05-19 11:02:35.824977 | focal-medium | 2022-05-19 11:02:35 [ERROR] {'ceph': 'zaza-3ac3c2b5764f'}
2022-05-19 11:02:35.825279 | focal-medium | 2022-05-19 11:02:35 [ERROR] Model ceph (zaza-3ac3c2b5764f)
2022-05-19 11:02:37.419028 | focal-medium | Traceback (most recent call last):
2022-05-19 11:02:37.419319 | focal-medium |   File "/home/ubuntu/src/review.opendev.org/openstack/charm-nova-compute/.tox/func-target/bin/functest-run-suite", line 8, in <module>
2022-05-19 11:02:37.419350 | focal-medium |     sys.exit(main())
2022-05-19 11:02:37.419408 | focal-medium |   File "/home/ubuntu/src/review.opendev.org/openstack/charm-nova-compute/.tox/func-target/lib/python3.8/site-packages/zaza/charm_lifecycle/func_test_runner.py", line 364, in main
2022-05-19 11:02:37.419486 | focal-medium |     func_test_runner(
2022-05-19 11:02:37.419521 | focal-medium |   File "/home/ubuntu/src/review.opendev.org/openstack/charm-nova-compute/.tox/func-target/lib/python3.8/site-packages/zaza/charm_lifecycle/func_test_runner.py", line 277, in func_test_runner
2022-05-19 11:02:37.419536 | focal-medium |     run_env_deployment(env_deployment, keep_model=preserve_model,
2022-05-19 11:02:37.419563 | focal-medium |   File "/home/ubuntu/src/review.opendev.org/openstack/charm-nova-compute/.tox/func-target/lib/python3.8/site-packages/zaza/charm_lifecycle/func_test_runner.py", line 151, in run_env_deployment
2022-05-19 11:02:37.419579 | focal-medium |     configure.configure(
2022-05-19 11:02:37.419591 | focal-medium |   File "/home/ubuntu/src/review.opendev.org/openstack/charm-nova-compute/.tox/func-target/lib/python3.8/site-packages/zaza/charm_lifecycle/configure.py", line 51, in configure
2022-05-19 11:02:37.419604 | focal-medium |     run_configure_list(functions)
2022-05-19 11:02:37.419616 | focal-medium |   File "/home/ubuntu/src/review.opendev.org/openstack/charm-nova-compute/.tox/func-target/lib/python3.8/site-packages/zaza/charm_lifecycle/configure.py", line 37, in run_configure_list
2022-05-19 11:02:37.419627 | focal-medium |     utils.get_class(func)()
2022-05-19 11:02:37.419642 | focal-medium |   File "/home/ubuntu/src/review.opendev.org/openstack/charm-nova-compute/.tox/func-target/lib/python3.8/site-packages/zaza/openstack/charm_tests/neutron/setup.py", line 103, in basic_overcloud_network
2022-05-19 11:02:37.419661 | focal-medium |     network.setup_gateway_ext_port(network_config,
2022-05-19 11:02:37.419673 | focal-medium |   File "/home/ubuntu/src/review.opendev.org/openstack/charm-nova-compute/.tox/func-target/lib/python3.8/site-packages/zaza/openstack/configure/network.py", line 286, in setup_gateway_ext_port
2022-05-19 11:02:37.419685 | focal-medium |     openstack_utils.configure_gateway_ext_port(
2022-05-19 11:02:37.419703 | focal-medium |   File "/home/ubuntu/src/review.opendev.org/openstack/charm-nova-compute/.tox/func-target/lib/python3.8/site-packages/zaza/openstack/utilities/openstack.py", line 1047, in configure_gateway_ext_port
2022-05-19 11:02:37.419715 | focal-medium |     macs = create_additional_port_for_machines(
2022-05-19 11:02:37.419726 | focal-medium |   File "/home/ubuntu/src/review.opendev.org/openstack/charm-nova-compute/.tox/func-target/lib/python3.8/site-packages/zaza/openstack/utilities/openstack.py", line 945, in create_additional_port_for_machines
2022-05-19 11:02:37.419738 | focal-medium |     server.interface_attach(port_id=port['port']['id'],
2022-05-19 11:02:37.419750 | focal-medium |   File "/home/ubuntu/src/review.opendev.org/openstack/charm-nova-compute/.tox/func-target/lib/python3.8/site-packages/novaclient/api_versions.py", line 393, in substitution
2022-05-19 11:02:37.419761 | focal-medium |     return methods[-1].func(obj, *args, **kwargs)
2022-05-19 11:02:37.419773 | focal-medium |   File "/home/ubuntu/src/review.opendev.org/openstack/charm-nova-compute/.tox/func-target/lib/python3.8/site-packages/novaclient/v2/servers.py", line 613, in interface_attach
2022-05-19 11:02:37.419785 | focal-medium |     return self.manager.interface_attach(self, port_id, net_id, fixed_ip)
2022-05-19 11:02:37.419797 | focal-medium |   File "/home/ubuntu/src/review.opendev.org/openstack/charm-nova-compute/.tox/func-target/lib/python3.8/site-packages/novaclient/api_versions.py", line 393, in substitution
2022-05-19 11:02:37.419880 | focal-medium |     return methods[-1].func(obj, *args, **kwargs)
2022-05-19 11:02:37.419898 | focal-medium |   File "/home/ubuntu/src/review.opendev.org/openstack/charm-nova-compute/.tox/func-target/lib/python3.8/site-packages/novaclient/v2/servers.py", line 2065, in interface_attach
2022-05-19 11:02:37.419908 | focal-medium |     return self._create('/servers/%s/os-interface' % base.getid(server),
2022-05-19 11:02:37.419919 | focal-medium |   File "/home/ubuntu/src/review.opendev.org/openstack/charm-nova-compute/.tox/func-target/lib/python3.8/site-packages/novaclient/base.py", line 364, in _create
2022-05-19 11:02:37.419930 | focal-medium |     resp, body = self.api.client.post(url, body=body)
2022-05-19 11:02:37.419941 | focal-medium |   File "/home/ubuntu/src/review.opendev.org/openstack/charm-nova-compute/.tox/func-target/lib/python3.8/site-packages/keystoneauth1/adapter.py", line 401, in post
2022-05-19 11:02:37.419951 | focal-medium |     return self.request(url, 'POST', **kwargs)
2022-05-19 11:02:37.419962 | focal-medium |   File "/home/ubuntu/src/review.opendev.org/openstack/charm-nova-compute/.tox/func-target/lib/python3.8/site-packages/novaclient/client.py", line 78, in request
2022-05-19 11:02:37.419972 | focal-medium |     raise exceptions.from_response(resp, body, url, method)
2022-05-19 11:02:37.419985 | focal-medium | novaclient.exceptions.ClientException: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
2022-05-19 11:02:37.420069 | focal-medium | <class 'oslo_messaging.exceptions.MessagingTimeout'> (HTTP 500) (Request-ID: req-18ba7adb-f06c-46b3-9bd0-4ac06233d99f)
2022-05-19 11:02:37.785417 | focal-medium | ERROR: InvocationError for command /home/ubuntu/src/review.opendev.org/openstack/charm-nova-compute/.tox/func-target/bin/functest-run-suite --keep-model --bundle ceph:focal-wallaby (exited with code 1)
freyes commented 2 years ago

the approach I have in mind is to create the port using a set of tags that can identify the port, so in a subsequent call (a retry) we can look for a port with those tags to know if the port was really created in the background.

Tags:

model_name=$modelname
unit_id=$principal_unit
freyes commented 2 years ago

Another occurrence:

https://openstack-ci-reports.ubuntu.com/artifacts/125/841978/1/check/vault-focal-xena/12583a5/

2022-05-23 15:10:23 [INFO] Attaching additional port to instance ("69e1ec08-d847-4018-b13c-c56bd9804419"), connected to net id: 94a87d6a-6d82-4931-878d-aedcf0e62962
2022-05-23 15:11:53 [ERROR] {'vault': 'zaza-4b8742462d18'}
2022-05-23 15:11:53 [ERROR] Model vault (zaza-4b8742462d18)
Traceback (most recent call last):
  File "/home/ubuntu/src/review.opendev.org/openstack/charm-nova-cloud-controller/.tox/func-target/lib/python3.8/site-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "/home/ubuntu/src/review.opendev.org/openstack/charm-nova-cloud-controller/.tox/func-target/lib/python3.8/site-packages/urllib3/connectionpool.py", line 449, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/home/ubuntu/src/review.opendev.org/openstack/charm-nova-cloud-controller/.tox/func-target/lib/python3.8/site-packages/urllib3/connectionpool.py", line 444, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib/python3.8/http/client.py", line 1348, in getresponse
    response.begin()
  File "/usr/lib/python3.8/http/client.py", line 316, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.8/http/client.py", line 285, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ubuntu/src/review.opendev.org/openstack/charm-nova-cloud-controller/.tox/func-target/lib/python3.8/site-packages/requests/adapters.py", line 440, in send
    resp = conn.urlopen(
  File "/home/ubuntu/src/review.opendev.org/openstack/charm-nova-cloud-controller/.tox/func-target/lib/python3.8/site-packages/urllib3/connectionpool.py", line 785, in urlopen
    retries = retries.increment(
  File "/home/ubuntu/src/review.opendev.org/openstack/charm-nova-cloud-controller/.tox/func-target/lib/python3.8/site-packages/urllib3/util/retry.py", line 550, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/home/ubuntu/src/review.opendev.org/openstack/charm-nova-cloud-controller/.tox/func-target/lib/python3.8/site-packages/urllib3/packages/six.py", line 769, in reraise
    raise value.with_traceback(tb)
  File "/home/ubuntu/src/review.opendev.org/openstack/charm-nova-cloud-controller/.tox/func-target/lib/python3.8/site-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "/home/ubuntu/src/review.opendev.org/openstack/charm-nova-cloud-controller/.tox/func-target/lib/python3.8/site-packages/urllib3/connectionpool.py", line 449, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/home/ubuntu/src/review.opendev.org/openstack/charm-nova-cloud-controller/.tox/func-target/lib/python3.8/site-packages/urllib3/connectionpool.py", line 444, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib/python3.8/http/client.py", line 1348, in getresponse
    response.begin()
  File "/usr/lib/python3.8/http/client.py", line 316, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.8/http/client.py", line 285, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ubuntu/src/review.opendev.org/openstack/charm-nova-cloud-controller/.tox/func-target/lib/python3.8/site-packages/keystoneauth1/session.py", line 1022, in _send_request
    resp = self.session.request(method, url, **kwargs)
  File "/home/ubuntu/src/review.opendev.org/openstack/charm-nova-cloud-controller/.tox/func-target/lib/python3.8/site-packages/requests/sessions.py", line 529, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/ubuntu/src/review.opendev.org/openstack/charm-nova-cloud-controller/.tox/func-target/lib/python3.8/site-packages/requests/sessions.py", line 645, in send
    r = adapter.send(request, **kwargs)
  File "/home/ubuntu/src/review.opendev.org/openstack/charm-nova-cloud-controller/.tox/func-target/lib/python3.8/site-packages/requests/adapters.py", line 501, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ubuntu/src/review.opendev.org/openstack/charm-nova-cloud-controller/.tox/func-target/bin/functest-run-suite", line 8, in <module>
    sys.exit(main())
  File "/home/ubuntu/src/review.opendev.org/openstack/charm-nova-cloud-controller/.tox/func-target/lib/python3.8/site-packages/zaza/charm_lifecycle/func_test_runner.py", line 364, in main
    func_test_runner(
  File "/home/ubuntu/src/review.opendev.org/openstack/charm-nova-cloud-controller/.tox/func-target/lib/python3.8/site-packages/zaza/charm_lifecycle/func_test_runner.py", line 277, in func_test_runner
    run_env_deployment(env_deployment, keep_model=preserve_model,
  File "/home/ubuntu/src/review.opendev.org/openstack/charm-nova-cloud-controller/.tox/func-target/lib/python3.8/site-packages/zaza/charm_lifecycle/func_test_runner.py", line 151, in run_env_deployment
    configure.configure(
  File "/home/ubuntu/src/review.opendev.org/openstack/charm-nova-cloud-controller/.tox/func-target/lib/python3.8/site-packages/zaza/charm_lifecycle/configure.py", line 51, in configure
    run_configure_list(functions)
  File "/home/ubuntu/src/review.opendev.org/openstack/charm-nova-cloud-controller/.tox/func-target/lib/python3.8/site-packages/zaza/charm_lifecycle/configure.py", line 37, in run_configure_list
    utils.get_class(func)()
  File "/home/ubuntu/src/review.opendev.org/openstack/charm-nova-cloud-controller/.tox/func-target/lib/python3.8/site-packages/zaza/openstack/charm_tests/neutron/setup.py", line 103, in basic_overcloud_network
    network.setup_gateway_ext_port(network_config,
  File "/home/ubuntu/src/review.opendev.org/openstack/charm-nova-cloud-controller/.tox/func-target/lib/python3.8/site-packages/zaza/openstack/configure/network.py", line 286, in setup_gateway_ext_port
    openstack_utils.configure_gateway_ext_port(
  File "/home/ubuntu/src/review.opendev.org/openstack/charm-nova-cloud-controller/.tox/func-target/lib/python3.8/site-packages/zaza/openstack/utilities/openstack.py", line 1047, in configure_gateway_ext_port
    macs = create_additional_port_for_machines(
  File "/home/ubuntu/src/review.opendev.org/openstack/charm-nova-cloud-controller/.tox/func-target/lib/python3.8/site-packages/zaza/openstack/utilities/openstack.py", line 944, in create_additional_port_for_machines
    port = neutronclient.create_port(body=body_value)
  File "/home/ubuntu/src/review.opendev.org/openstack/charm-nova-cloud-controller/.tox/func-target/lib/python3.8/site-packages/neutronclient/v2_0/client.py", line 803, in create_port
    return self.post(self.ports_path, body=body)
  File "/home/ubuntu/src/review.opendev.org/openstack/charm-nova-cloud-controller/.tox/func-target/lib/python3.8/site-packages/neutronclient/v2_0/client.py", line 358, in post
    return self.do_request("POST", action, body=body,
  File "/home/ubuntu/src/review.opendev.org/openstack/charm-nova-cloud-controller/.tox/func-target/lib/python3.8/site-packages/neutronclient/v2_0/client.py", line 281, in do_request
    resp, replybody = self.httpclient.do_request(action, method, body=body,
  File "/home/ubuntu/src/review.opendev.org/openstack/charm-nova-cloud-controller/.tox/func-target/lib/python3.8/site-packages/neutronclient/client.py", line 340, in do_request
    return self.request(url, method, **kwargs)
  File "/home/ubuntu/src/review.opendev.org/openstack/charm-nova-cloud-controller/.tox/func-target/lib/python3.8/site-packages/neutronclient/client.py", line 328, in request
    resp = super(SessionClient, self).request(*args, **kwargs)
  File "/home/ubuntu/src/review.opendev.org/openstack/charm-nova-cloud-controller/.tox/func-target/lib/python3.8/site-packages/keystoneauth1/adapter.py", line 257, in request
    return self.session.request(url, method, **kwargs)
  File "/home/ubuntu/src/review.opendev.org/openstack/charm-nova-cloud-controller/.tox/func-target/lib/python3.8/site-packages/keystoneauth1/session.py", line 931, in request
    resp = send(**kwargs)
  File "/home/ubuntu/src/review.opendev.org/openstack/charm-nova-cloud-controller/.tox/func-target/lib/python3.8/site-packages/keystoneauth1/session.py", line 1038, in _send_request
    raise exceptions.ConnectFailure(msg)
keystoneauth1.exceptions.connection.ConnectFailure: Unable to establish connection to http://10.245.161.159:9696/v2.0/ports: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
ERROR: InvocationError for command /home/ubuntu/src/review.opendev.org/openstack/charm-nova-cloud-controller/.tox/func-target/bin/functest-run-suite --keep-model --bundle vault:focal-xena (exited with code 1)
___________________________________ summary ____________________________________
ERROR:   func-target: commands failed