Missing retry/exception handling for Nova overcloud API call

[fnordahl]

2021-05-22 06:25:07.362241 | bionic-medium | 2021-05-22 06:25:07 [ERROR] {'default_alias': 'zaza-21f88ed5ac6e'}
2021-05-22 06:25:07.362486 | bionic-medium | 2021-05-22 06:25:07 [ERROR] Model default_alias (zaza-21f88ed5ac6e)
2021-05-22 06:25:07.987688 | bionic-medium | Traceback (most recent call last):
2021-05-22 06:25:07.987777 | bionic-medium |   File "/home/ubuntu/src/review.opendev.org/openstack/charm-octavia/build/builds/octavia/.tox/func-target/bin/functest-run-suite", line 8, in <module>
2021-05-22 06:25:07.987821 | bionic-medium |     sys.exit(main())
2021-05-22 06:25:07.987854 | bionic-medium |   File "/home/ubuntu/src/review.opendev.org/openstack/charm-octavia/build/builds/octavia/.tox/func-target/lib/python3.6/site-packages/zaza/charm_lifecycle/func_test_runner.py", line 314, in main
2021-05-22 06:25:07.987901 | bionic-medium |     test_directory=args.test_directory)
2021-05-22 06:25:07.987944 | bionic-medium |   File "/home/ubuntu/src/review.opendev.org/openstack/charm-octavia/build/builds/octavia/.tox/func-target/lib/python3.6/site-packages/zaza/charm_lifecycle/func_test_runner.py", line 243, in func_test_runner
2021-05-22 06:25:07.987968 | bionic-medium |     force=force, test_directory=test_directory)
2021-05-22 06:25:07.987985 | bionic-medium |   File "/home/ubuntu/src/review.opendev.org/openstack/charm-octavia/build/builds/octavia/.tox/func-target/lib/python3.6/site-packages/zaza/charm_lifecycle/func_test_runner.py", line 144, in run_env_deployment
2021-05-22 06:25:07.987998 | bionic-medium |     test_directory=test_directory)
2021-05-22 06:25:07.988011 | bionic-medium |   File "/home/ubuntu/src/review.opendev.org/openstack/charm-octavia/build/builds/octavia/.tox/func-target/lib/python3.6/site-packages/zaza/charm_lifecycle/configure.py", line 51, in configure
2021-05-22 06:25:07.988028 | bionic-medium |     run_configure_list(functions)
2021-05-22 06:25:07.988053 | bionic-medium |   File "/home/ubuntu/src/review.opendev.org/openstack/charm-octavia/build/builds/octavia/.tox/func-target/lib/python3.6/site-packages/zaza/charm_lifecycle/configure.py", line 37, in run_configure_list
2021-05-22 06:25:07.988069 | bionic-medium |     utils.get_class(func)()
2021-05-22 06:25:07.988082 | bionic-medium |   File "/home/ubuntu/src/review.opendev.org/openstack/charm-octavia/build/builds/octavia/.tox/func-target/lib/python3.6/site-packages/zaza/openstack/charm_tests/neutron/setup.py", line 99, in basic_overcloud_network
2021-05-22 06:25:07.988106 | bionic-medium |     use_juju_wait=use_juju_wait)
2021-05-22 06:25:07.988120 | bionic-medium |   File "/home/ubuntu/src/review.opendev.org/openstack/charm-octavia/build/builds/octavia/.tox/func-target/lib/python3.6/site-packages/zaza/openstack/configure/network.py", line 237, in setup_gateway_ext_port
2021-05-22 06:25:07.988133 | bionic-medium |     use_juju_wait=use_juju_wait)
2021-05-22 06:25:07.988146 | bionic-medium |   File "/home/ubuntu/src/review.opendev.org/openstack/charm-octavia/build/builds/octavia/.tox/func-target/lib/python3.6/site-packages/zaza/openstack/utilities/openstack.py", line 1043, in configure_gateway_ext_port
2021-05-22 06:25:07.988159 | bionic-medium |     add_dataport_to_netplan)
2021-05-22 06:25:07.988182 | bionic-medium |   File "/home/ubuntu/src/review.opendev.org/openstack/charm-octavia/build/builds/octavia/.tox/func-target/lib/python3.6/site-packages/zaza/openstack/utilities/openstack.py", line 943, in create_additional_port_for_machines
2021-05-22 06:25:07.988202 | bionic-medium |     net_id=None, fixed_ip=None)
2021-05-22 06:25:07.988217 | bionic-medium |   File "/home/ubuntu/src/review.opendev.org/openstack/charm-octavia/build/builds/octavia/.tox/func-target/lib/python3.6/site-packages/novaclient/api_versions.py", line 393, in substitution
2021-05-22 06:25:07.988249 | bionic-medium |     return methods[-1].func(obj, *args, **kwargs)
2021-05-22 06:25:07.988278 | bionic-medium |   File "/home/ubuntu/src/review.opendev.org/openstack/charm-octavia/build/builds/octavia/.tox/func-target/lib/python3.6/site-packages/novaclient/v2/servers.py", line 613, in interface_attach
2021-05-22 06:25:07.988300 | bionic-medium |     return self.manager.interface_attach(self, port_id, net_id, fixed_ip)
2021-05-22 06:25:07.988314 | bionic-medium |   File "/home/ubuntu/src/review.opendev.org/openstack/charm-octavia/build/builds/octavia/.tox/func-target/lib/python3.6/site-packages/novaclient/api_versions.py", line 393, in substitution
2021-05-22 06:25:07.988345 | bionic-medium |     return methods[-1].func(obj, *args, **kwargs)
2021-05-22 06:25:07.988360 | bionic-medium |   File "/home/ubuntu/src/review.opendev.org/openstack/charm-octavia/build/builds/octavia/.tox/func-target/lib/python3.6/site-packages/novaclient/v2/servers.py", line 2067, in interface_attach
2021-05-22 06:25:07.988380 | bionic-medium |     obj_class=NetworkInterface)
2021-05-22 06:25:07.988401 | bionic-medium |   File "/home/ubuntu/src/review.opendev.org/openstack/charm-octavia/build/builds/octavia/.tox/func-target/lib/python3.6/site-packages/novaclient/base.py", line 364, in _create
2021-05-22 06:25:07.988414 | bionic-medium |     resp, body = self.api.client.post(url, body=body)
2021-05-22 06:25:07.988427 | bionic-medium |   File "/home/ubuntu/src/review.opendev.org/openstack/charm-octavia/build/builds/octavia/.tox/func-target/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 401, in post
2021-05-22 06:25:07.988440 | bionic-medium |     return self.request(url, 'POST', **kwargs)
2021-05-22 06:25:07.988452 | bionic-medium |   File "/home/ubuntu/src/review.opendev.org/openstack/charm-octavia/build/builds/octavia/.tox/func-target/lib/python3.6/site-packages/novaclient/client.py", line 78, in request
2021-05-22 06:25:07.988465 | bionic-medium |     raise exceptions.from_response(resp, body, url, method)
2021-05-22 06:25:07.988494 | bionic-medium | novaclient.exceptions.ClientException: Unknown Error (HTTP 500)

[ajkavanagh]

This would be an excellent test/use of the ObjectRetrier class that, er, does retries for an object automatically. Built exactly for this scenario.

In https://github.com/openstack-charmers/zaza-openstack-tests/blob/eae35cce96e44ab9f871bee87d792f01b09244c6/zaza/openstack/configure/network.py#L186

    # Get authenticated clients
    nova_client = ObjectRetrierWraps(
        openstack_utils.get_nova_session_client(keystone_session))
    neutron_client = ObjectRetrierWraps(
        openstack_utils.get_neutron_session_client(keystone_session))

Essentially, any exception generated by an method on nova_client or neutron_client will be retried. If that's too broad we could restrict it to relevant client and connection errors?

What do you think?

[fnordahl]

@ajkavanagh that is an excellent idea, and thank you for the pointer. I'll have a look.