search

vexxhost / atmosphere

Simple & easy private cloud platform featuring VMs, Kubernetes & bare-metal
88 stars 23 forks source link

OVN tempest failure #662

Open okozachenko1203 opened 11 months ago

okozachenko1203 commented 11 months ago

Tempest is failing frequently at the following senarios

setUpClass (tempest.api.compute.servers.test_create_server.ServersTestBootFromVolume)
tearDownClass (tempest.api.compute.servers.test_create_server.ServersTestJSON)

It is failing with this error

        "",
        "tearDownClass (tempest.api.compute.servers.test_create_server.ServersTestJSON)",
        "------------------------------------------------------------------------------",
        "",
        "Captured traceback:",
        "~~~~~~~~~~~~~~~~~~~",
        "    Traceback (most recent call last):",
        "",
        "      File \"/var/lib/openstack/lib/python3.8/site-packages/tempest/test.py\", line 220, in tearDownClass",
        "    raise value.with_traceback(trace)",
        "",
        "      File \"/var/lib/openstack/lib/python3.8/site-packages/tempest/test.py\", line 192, in tearDownClass",
        "    teardown()",
        "",
        "      File \"/var/lib/openstack/lib/python3.8/site-packages/tempest/test.py\", line 731, in clear_credentials",
        "    cls._creds_provider.clear_creds()",
        "",
        "      File \"/var/lib/openstack/lib/python3.8/site-packages/tempest/lib/common/dynamic_creds.py\", line 555, in clear_creds",
        "    self._clear_isolated_net_resources()",
        "",
        "      File \"/var/lib/openstack/lib/python3.8/site-packages/tempest/lib/common/dynamic_creds.py\", line 535, in _clear_isolated_net_resources",
        "    client.remove_router_interface(",
        "",
        "      File \"/var/lib/openstack/lib/python3.8/site-packages/tempest/lib/services/network/routers_client.py\", line 82, in remove_router_interface",
        "    return self.update_resource(uri, kwargs)",
        "",
        "      File \"/var/lib/openstack/lib/python3.8/site-packages/tempest/lib/services/network/base.py\", line 77, in update_resource",
        "    resp, body = self.put(req_uri, req_post_data)",
        "",
        "      File \"/var/lib/openstack/lib/python3.8/site-packages/tempest/lib/common/rest_client.py\", line 363, in put",
        "    return self.request('PUT', url, extra_headers, headers, body, chunked)",
        "",
        "      File \"/var/lib/openstack/lib/python3.8/site-packages/tempest/lib/common/rest_client.py\", line 703, in request",
        "    self._error_checker(resp, resp_body)",
        "",
        "      File \"/var/lib/openstack/lib/python3.8/site-packages/tempest/lib/common/rest_client.py\", line 879, in _error_checker",
        "    raise exceptions.ServerFault(resp_body, resp=resp,",
        "",
        "    tempest.lib.exceptions.ServerFault: Got server fault",
        "Details: Request Failed: internal server error while processing your request.",
        "",

At that time, neutron server has this error log

2023-10-18 13:26:22.420 9 WARNING neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-403a932d-c1c4-44c3-9e45-f977431efc4f - - - - - -] No hosting information found for port b1940853-e207-437d-9669-f6ebb52834de: RuntimeError: No hosting information found for port b1940853-e207-437d-9669-f6ebb52834de
2023-10-18 13:26:27.465 16 WARNING neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-f521dcd4-e9b5-4170-b3ed-ee1b46550209 - - - - - -] No hosting information found for port 4e58f242-d846-4d47-841c-efa509b1ad99: RuntimeError: No hosting information found for port 4e58f242-d846-4d47-841c-efa509b1ad99
2023-10-18 13:26:34.024 16 WARNING neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-f521dcd4-e9b5-4170-b3ed-ee1b46550209 - - - - - -] No hosting information found for port cdd3786b-bf9d-4661-baaf-fea437e11e27: RuntimeError: No hosting information found for port cdd3786b-bf9d-4661-baaf-fea437e11e27
2023-10-18 13:26:41.791 13 ERROR ovsdbapp.backend.ovs_idl.transaction [None req-e61b8321-d1a9-46d0-b7fe-9750a3cd9e7e ab08758283cc4cecaef4a3ab48643074 21d1195fb18546168ee22766c7e97f79 - - default default] Traceback (most recent call last):
2023-10-18 13:26:43.697 13 ERROR neutron.api.v2.resource [None req-e61b8321-d1a9-46d0-b7fe-9750a3cd9e7e ab08758283cc4cecaef4a3ab48643074 21d1195fb18546168ee22766c7e97f79 - - default default] remove_router_interface failed: No details.: ovsdbapp.backend.ovs_idl.idlutils.RowNotFound: Cannot find Logical_Router_Port with name=lrp-6e87b158-1bd2-4f78-ab8f-f216b9dd55be
2023-10-18 13:26:43.697 13 ERROR neutron.api.v2.resource Traceback (most recent call last):
2023-10-18 13:26:43.697 13 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/neutron/api/v2/resource.py", line 98, in resource
2023-10-18 13:26:43.697 13 ERROR neutron.api.v2.resource     result = method(request=request, **args)
2023-10-18 13:26:43.697 13 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/neutron_lib/db/api.py", line 140, in wrapped
2023-10-18 13:26:43.697 13 ERROR neutron.api.v2.resource     with excutils.save_and_reraise_exception():
2023-10-18 13:26:43.697 13 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2023-10-18 13:26:43.697 13 ERROR neutron.api.v2.resource     self.force_reraise()
2023-10-18 13:26:43.697 13 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2023-10-18 13:26:43.697 13 ERROR neutron.api.v2.resource     raise self.value
2023-10-18 13:26:43.697 13 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/neutron_lib/db/api.py", line 138, in wrapped
2023-10-18 13:26:43.697 13 ERROR neutron.api.v2.resource     return f(*args, **kwargs)
2023-10-18 13:26:43.697 13 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/oslo_db/api.py", line 144, in wrapper
2023-10-18 13:26:43.697 13 ERROR neutron.api.v2.resource     with excutils.save_and_reraise_exception() as ectxt:
2023-10-18 13:26:43.697 13 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2023-10-18 13:26:43.697 13 ERROR neutron.api.v2.resource     self.force_reraise()
2023-10-18 13:26:43.697 13 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2023-10-18 13:26:43.697 13 ERROR neutron.api.v2.resource     raise self.value
2023-10-18 13:26:43.697 13 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/oslo_db/api.py", line 142, in wrapper
2023-10-18 13:26:43.697 13 ERROR neutron.api.v2.resource     return f(*args, **kwargs)
2023-10-18 13:26:43.697 13 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/neutron_lib/db/api.py", line 186, in wrapped
2023-10-18 13:26:43.697 13 ERROR neutron.api.v2.resource     with excutils.save_and_reraise_exception():
2023-10-18 13:26:43.697 13 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2023-10-18 13:26:43.697 13 ERROR neutron.api.v2.resource     self.force_reraise()
2023-10-18 13:26:43.697 13 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2023-10-18 13:26:43.697 13 ERROR neutron.api.v2.resource     raise self.value
2023-10-18 13:26:43.697 13 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/neutron_lib/db/api.py", line 184, in wrapped
2023-10-18 13:26:43.697 13 ERROR neutron.api.v2.resource     return f(*dup_args, **dup_kwargs)
2023-10-18 13:26:43.697 13 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/neutron/api/v2/base.py", line 253, in _handle_action
2023-10-18 13:26:43.697 13 ERROR neutron.api.v2.resource     ret_value = getattr(self._plugin, name)(*arg_list, **kwargs)
2023-10-18 13:26:43.697 13 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/neutron/services/ovn_l3/plugin.py", line 260, in remove_router_interface
2023-10-18 13:26:43.697 13 ERROR neutron.api.v2.resource     with excutils.save_and_reraise_exception():
2023-10-18 13:26:43.697 13 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2023-10-18 13:26:43.697 13 ERROR neutron.api.v2.resource     self.force_reraise()
2023-10-18 13:26:43.697 13 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2023-10-18 13:26:43.697 13 ERROR neutron.api.v2.resource     raise self.value
2023-10-18 13:26:43.697 13 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/neutron/services/ovn_l3/plugin.py", line 257, in remove_router_interface
2023-10-18 13:26:43.697 13 ERROR neutron.api.v2.resource     self._ovn_client.delete_router_port(
2023-10-18 13:26:43.697 13 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1904, in delete_router_port
2023-10-18 13:26:43.697 13 ERROR neutron.api.v2.resource     with self._nb_idl.transaction(check_error=True) as txn:
2023-10-18 13:26:43.697 13 ERROR neutron.api.v2.resource   File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
2023-10-18 13:26:43.697 13 ERROR neutron.api.v2.resource     next(self.gen)
2023-10-18 13:26:43.697 13 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction
2023-10-18 13:26:43.697 13 ERROR neutron.api.v2.resource     with super(OvsdbNbOvnIdl, self).transaction(*args, **kwargs) as t:
2023-10-18 13:26:43.697 13 ERROR neutron.api.v2.resource   File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
2023-10-18 13:26:43.697 13 ERROR neutron.api.v2.resource     next(self.gen)
2023-10-18 13:26:43.697 13 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/ovsdbapp/api.py", line 104, in transaction
2023-10-18 13:26:43.697 13 ERROR neutron.api.v2.resource     with self.create_transaction(
2023-10-18 13:26:43.697 13 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/ovsdbapp/api.py", line 61, in __exit__
2023-10-18 13:26:43.697 13 ERROR neutron.api.v2.resource     self.result = self.commit()
2023-10-18 13:26:43.697 13 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit
2023-10-18 13:26:43.697 13 ERROR neutron.api.v2.resource     raise result.ex
2023-10-18 13:26:43.697 13 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 118, in run
2023-10-18 13:26:43.697 13 ERROR neutron.api.v2.resource     txn.results.put(txn.do_commit())
2023-10-18 13:26:43.697 13 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 92, in do_commit
2023-10-18 13:26:43.697 13 ERROR neutron.api.v2.resource     command.run_idl(txn)
2023-10-18 13:26:43.697 13 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 341, in run_idl
2023-10-18 13:26:43.697 13 ERROR neutron.api.v2.resource     entity = self.api.lookup(self.table, self.entity)
2023-10-18 13:26:43.697 13 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 181, in lookup
2023-10-18 13:26:43.697 13 ERROR neutron.api.v2.resource     return self._lookup(table, record)
2023-10-18 13:26:43.697 13 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 228, in _lookup
2023-10-18 13:26:43.697 13 ERROR neutron.api.v2.resource     row = idlutils.row_by_value(self, rl.table, rl.column, record)
2023-10-18 13:26:43.697 13 ERROR neutron.api.v2.resource   File "/var/lib/openstack/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 114, in row_by_value
2023-10-18 13:26:43.697 13 ERROR neutron.api.v2.resource     raise RowNotFound(table=table, col=column, match=match)
2023-10-18 13:26:43.697 13 ERROR neutron.api.v2.resource ovsdbapp.backend.ovs_idl.idlutils.RowNotFound: Cannot find Logical_Router_Port with name=lrp-6e87b158-1bd2-4f78-ab8f-f216b9dd55be
2023-10-18 13:26:43.697 13 ERROR neutron.api.v2.resource
Collapse
mnaser commented 11 months ago

@ricolin @okozachenko1203 does it make sense to maybe bump our Tempest image? It's probably ancient :P

mnaser commented 6 months ago

I opened upstream issue here: https://bugs.launchpad.net/neutron/+bug/2060163

mnaser commented 4 months ago

We might have been able to fix it..

https://github.com/vexxhost/atmosphere/pull/1263 https://github.com/vexxhost/atmosphere/pull/1262 https://github.com/vexxhost/atmosphere/pull/1261

stable/2023.1 being fixed right now but zed passed twice in a row, and main broken pending Senlin removal but looking good.