Load balancer fails to delete when tls_certificate is removed beforehand

[guilhermesteinmuller]

We have a case in which a user deleted a certificate that was set inside a listener, causing the lb being 'stuck' from being deleted.

openstack/octavia-worker-76f48486bc-v4wr2[octavia-worker]: 2023-09-11 18:16:42.131 20 INFO octavia.controller.queue.v2.endpoints [None req-343061ca-86a6-4610-89b6-685d6a97fa76 - 354c319cc0d546628c719043539e427f - - - -] Deleting load balancer 'c5573e76-9577-4c8f-9e16-76ae88d45736'...
openstack/octavia-worker-76f48486bc-v4wr2[octavia-worker]: 2023-09-11 18:16:42.274 20 INFO octavia.certificates.manager.barbican [None req-d7059199-a898-4945-a9b0-17db107fb979 - 354c319cc0d546628c719043539e427f - - - -] Loading certificate secret https://key-manager.openstack.cloud.local/v1/containers/8ec85cf4-7135-4eec-81a1-bd6ca823792b from Barbican.
openstack/octavia-worker-76f48486bc-v4wr2[octavia-worker]: 2023-09-11 18:16:42.275 20 INFO barbicanclient.base [None req-d7059199-a898-4945-a9b0-17db107fb979 - 354c319cc0d546628c719043539e427f - - - -] Calculated Secrets uuid ref: secrets/8ec85cf4-7135-4eec-81a1-bd6ca823792b
openstack/octavia-worker-76f48486bc-v4wr2[octavia-worker]: 2023-09-11 18:16:42.370 20 ERROR barbicanclient.client [None req-d7059199-a898-4945-a9b0-17db107fb979 - 354c319cc0d546628c719043539e427f - - - -] 4xx Client error: Not Found: Secret not found.
openstack/octavia-worker-76f48486bc-v4wr2[octavia-worker]: 2023-09-11 18:16:42.371 20 INFO octavia.certificates.manager.barbican_legacy [None req-d7059199-a898-4945-a9b0-17db107fb979 - 354c319cc0d546628c719043539e427f - - - -] Loading certificate container https://key-manager.openstack.cloud.local/v1/containers/8ec85cf4-7135-4eec-81a1-bd6ca823792b from Barbican.
openstack/octavia-worker-76f48486bc-v4wr2[octavia-worker]: 2023-09-11 18:16:42.372 20 INFO barbicanclient.base [None req-d7059199-a898-4945-a9b0-17db107fb979 - 354c319cc0d546628c719043539e427f - - - -] Calculated Containers uuid ref: containers/8ec85cf4-7135-4eec-81a1-bd6ca823792b
openstack/octavia-worker-76f48486bc-v4wr2[octavia-worker]: 2023-09-11 18:16:42.395 20 ERROR barbicanclient.client [None req-d7059199-a898-4945-a9b0-17db107fb979 - 354c319cc0d546628c719043539e427f - - - -] 4xx Client error: Not Found: Secrets container not found.: barbicanclient.exceptions.HTTPClientError: Not Found: Secret not found.
openstack/octavia-worker-76f48486bc-v4wr2[octavia-worker]: 2023-09-11 18:16:42.396 20 ERROR octavia.certificates.manager.barbican_legacy [None req-d7059199-a898-4945-a9b0-17db107fb979 - 354c319cc0d546628c719043539e427f - - - -] Error getting cert https://key-manager.openstack.cloud.local/v1/containers/8ec85cf4-7135-4eec-81a1-bd6ca823792b: Not Found: Secrets container not found.: barbicanclient.exceptions.HTTPClientError: Not Found: Secrets container not found.
openstack/octavia-worker-76f48486bc-v4wr2[octavia-worker]: 2023-09-11 18:16:42.397 20 WARNING octavia.common.tls_utils.cert_parser [None req-d7059199-a898-4945-a9b0-17db107fb979 - 354c319cc0d546628c719043539e427f - - - -] Unable to retrieve certificate: https://key-manager.openstack.cloud.local/v1/containers/8ec85cf4-7135-4eec-81a1-bd6ca823792b due to Not Found: Secrets container not found..: barbicanclient.exceptions.HTTPClientError: Not Found: Secrets container not found.
openstack/octavia-worker-76f48486bc-v4wr2[octavia-worker]: 2023-09-11 18:16:42.398 20 WARNING octavia.api.drivers.utils [None req-d7059199-a898-4945-a9b0-17db107fb979 - 354c319cc0d546628c719043539e427f - - - -] Unable to retrieve certificate(s) due to Could not retrieve certificate: https://key-manager.openstack.cloud.local/v1/containers/8ec85cf4-7135-4eec-81a1-bd6ca823792b.: octavia.common.exceptions.CertificateRetrievalException: Could not retrieve certificate: https://key-manager.openstack.cloud.local/v1/containers/8ec85cf4-7135-4eec-81a1-bd6ca823792b
openstack/octavia-worker-76f48486bc-v4wr2[octavia-worker]: 2023-09-11 18:16:42.399 20 ERROR oslo_messaging.rpc.server [None req-d7059199-a898-4945-a9b0-17db107fb979 - 354c319cc0d546628c719043539e427f - - - -] Exception during message handling: octavia.common.exceptions.CertificateRetrievalException: Could not retrieve certificate: https://key-manager.openstack.cloud.local/v1/containers/8ec85cf4-7135-4eec-81a1-bd6ca823792b
openstack/octavia-worker-76f48486bc-v4wr2[octavia-worker]: 2023-09-11 18:16:42.399 20 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
openstack/octavia-worker-76f48486bc-v4wr2[octavia-worker]: 2023-09-11 18:16:42.399 20 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/lib/python3.10/site-packages/octavia/certificates/manager/barbican.py", line 115, in get_cert
openstack/octavia-worker-76f48486bc-v4wr2[octavia-worker]: 2023-09-11 18:16:42.399 20 ERROR oslo_messaging.rpc.server     return pkcs12.PKCS12Cert(cert_secret.payload)
openstack/octavia-worker-76f48486bc-v4wr2[octavia-worker]: 2023-09-11 18:16:42.399 20 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/lib/python3.10/site-packages/barbicanclient/v1/secrets.py", line 192, in payload
openstack/octavia-worker-76f48486bc-cz9gs[octavia-worker]: 2023-09-11 18:14:54.208 32 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/lib/python3.10/site-packages/barbicanclient/v1/secrets.py", line 260, in _fetch_payload
openstack/octavia-worker-76f48486bc-cz9gs[octavia-worker]: 2023-09-11 18:14:54.208 32 ERROR oslo_messaging.rpc.server     if not self.payload_content_type and not self.content_types:
openstack/octavia-worker-76f48486bc-cz9gs[octavia-worker]: 2023-09-11 18:14:54.208 32 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/lib/python3.10/site-packages/barbicanclient/v1/secrets.py", line 183, in payload_content_type
openstack/octavia-worker-76f48486bc-cz9gs[octavia-worker]: 2023-09-11 18:14:54.208 32 ERROR oslo_messaging.rpc.server     if not self._payload_content_type and self.content_types:
openstack/octavia-worker-76f48486bc-cz9gs[octavia-worker]: 2023-09-11 18:14:54.208 32 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/lib/python3.10/site-packages/barbicanclient/v1/secrets.py", line 33, in wrapper
openstack/octavia-worker-76f48486bc-cz9gs[octavia-worker]: 2023-09-11 18:14:54.208 32 ERROR oslo_messaging.rpc.server     self._fill_lazy_properties()
openstack/octavia-worker-76f48486bc-cz9gs[octavia-worker]: 2023-09-11 18:14:54.208 32 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/lib/python3.10/site-packages/barbicanclient/v1/secrets.py", line 416, in _fill_lazy_properties
openstack/octavia-worker-76f48486bc-cz9gs[octavia-worker]: 2023-09-11 18:14:54.208 32 ERROR oslo_messaging.rpc.server     result = self._api.get(uuid_ref)
openstack/octavia-worker-76f48486bc-cz9gs[octavia-worker]: 2023-09-11 18:14:54.208 32 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/lib/python3.10/site-packages/barbicanclient/client.py", line 70, in get
openstack/octavia-worker-76f48486bc-cz9gs[octavia-worker]: 2023-09-11 18:14:54.208 32 ERROR oslo_messaging.rpc.server     return super(_HTTPClient, self).get(*args, **kwargs).json()
openstack/octavia-worker-76f48486bc-cz9gs[octavia-worker]: 2023-09-11 18:14:54.208 32 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/lib/python3.10/site-packages/keystoneauth1/adapter.py", line 395, in get
openstack/octavia-worker-76f48486bc-cz9gs[octavia-worker]: 2023-09-11 18:14:54.208 32 ERROR oslo_messaging.rpc.server     return self.request(url, 'GET', **kwargs)
openstack/octavia-worker-76f48486bc-cz9gs[octavia-worker]: 2023-09-11 18:14:54.208 32 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/lib/python3.10/site-packages/barbicanclient/client.py", line 63, in request
openstack/octavia-worker-76f48486bc-cz9gs[octavia-worker]: 2023-09-11 18:14:54.208 32 ERROR oslo_messaging.rpc.server     self._check_status_code(resp)
openstack/octavia-worker-76f48486bc-cz9gs[octavia-worker]: 2023-09-11 18:14:54.208 32 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/lib/python3.10/site-packages/barbicanclient/client.py", line 105, in _check_status_code
openstack/octavia-worker-76f48486bc-cz9gs[octavia-worker]: 2023-09-11 18:14:54.208 32 ERROR oslo_messaging.rpc.server     raise exceptions.HTTPClientError(
openstack/octavia-worker-76f48486bc-cz9gs[octavia-worker]: 2023-09-11 18:14:54.208 32 ERROR oslo_messaging.rpc.server barbicanclient.exceptions.HTTPClientError: Not Found: Secret not found.
openstack/octavia-worker-76f48486bc-cz9gs[octavia-worker]: 2023-09-11 18:14:54.208 32 ERROR oslo_messaging.rpc.server 
openstack/octavia-worker-76f48486bc-cz9gs[octavia-worker]: 2023-09-11 18:14:54.208 32 ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred:
openstack/octavia-worker-76f48486bc-cz9gs[octavia-worker]: 2023-09-11 18:14:54.208 32 ERROR oslo_messaging.rpc.server 
openstack/octavia-worker-76f48486bc-cz9gs[octavia-worker]: 2023-09-11 18:14:54.208 32 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
openstack/octavia-worker-76f48486bc-cz9gs[octavia-worker]: 2023-09-11 18:14:54.208 32 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/lib/python3.10/site-packages/octavia/common/tls_utils/cert_parser.py", line 359, in load_certificates_data
openstack/octavia-worker-76f48486bc-cz9gs[octavia-worker]: 2023-09-11 18:14:54.208 32 ERROR oslo_messaging.rpc.server     cert_mngr.get_cert(context,
openstack/octavia-worker-76f48486bc-cz9gs[octavia-worker]: 2023-09-11 18:14:54.208 32 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/lib/python3.10/site-packages/octavia/certificates/manager/barbican.py", line 123, in get_cert
openstack/octavia-worker-76f48486bc-cz9gs[octavia-worker]: 2023-09-11 18:14:54.208 32 ERROR oslo_messaging.rpc.server     legacy_cert = legacy_mgr.get_cert(
openstack/octavia-worker-76f48486bc-cz9gs[octavia-worker]: 2023-09-11 18:14:54.208 32 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/lib/python3.10/site-packages/octavia/certificates/manager/barbican_legacy.py", line 160, in get_cert
openstack/octavia-worker-76f48486bc-cz9gs[octavia-worker]: 2023-09-11 18:14:54.208 32 ERROR oslo_messaging.rpc.server     with excutils.save_and_reraise_exception():
openstack/octavia-worker-76f48486bc-cz9gs[octavia-worker]: 2023-09-11 18:14:54.208 32 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
openstack/octavia-worker-76f48486bc-cz9gs[octavia-worker]: 2023-09-11 18:14:54.208 32 ERROR oslo_messaging.rpc.server     self.force_reraise()
openstack/octavia-worker-76f48486bc-cz9gs[octavia-worker]: 2023-09-11 18:14:54.208 32 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
openstack/octavia-worker-76f48486bc-cz9gs[octavia-worker]: 2023-09-11 18:14:54.208 32 ERROR oslo_messaging.rpc.server     raise self.value
openstack/octavia-worker-76f48486bc-v4wr2[octavia-worker]: 2023-09-11 18:16:42.399 20 ERROR oslo_messaging.rpc.server     self._fetch_payload()
openstack/octavia-worker-76f48486bc-v4wr2[octavia-worker]: 2023-09-11 18:16:42.399 20 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/lib/python3.10/site-packages/barbicanclient/v1/secrets.py", line 260, in _fetch_payload
openstack/octavia-worker-76f48486bc-v4wr2[octavia-worker]: 2023-09-11 18:16:42.399 20 ERROR oslo_messaging.rpc.server     if not self.payload_content_type and not self.content_types:
openstack/octavia-worker-76f48486bc-v4wr2[octavia-worker]: 2023-09-11 18:16:42.399 20 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/lib/python3.10/site-packages/barbicanclient/v1/secrets.py", line 183, in payload_content_type
openstack/octavia-worker-76f48486bc-v4wr2[octavia-worker]: 2023-09-11 18:16:42.399 20 ERROR oslo_messaging.rpc.server     if not self._payload_content_type and self.content_types:
openstack/octavia-worker-76f48486bc-v4wr2[octavia-worker]: 2023-09-11 18:16:42.399 20 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/lib/python3.10/site-packages/barbicanclient/v1/secrets.py", line 33, in wrapper
openstack/octavia-worker-76f48486bc-v4wr2[octavia-worker]: 2023-09-11 18:16:42.399 20 ERROR oslo_messaging.rpc.server     self._fill_lazy_properties()
openstack/octavia-worker-76f48486bc-v4wr2[octavia-worker]: 2023-09-11 18:16:42.399 20 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/lib/python3.10/site-packages/barbicanclient/v1/secrets.py", line 416, in _fill_lazy_properties
openstack/octavia-worker-76f48486bc-v4wr2[octavia-worker]: 2023-09-11 18:16:42.399 20 ERROR oslo_messaging.rpc.server     result = self._api.get(uuid_ref)
openstack/octavia-worker-76f48486bc-v4wr2[octavia-worker]: 2023-09-11 18:16:42.399 20 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/lib/python3.10/site-packages/barbicanclient/client.py", line 70, in get
openstack/octavia-worker-76f48486bc-v4wr2[octavia-worker]: 2023-09-11 18:16:42.399 20 ERROR oslo_messaging.rpc.server     return super(_HTTPClient, self).get(*args, **kwargs).json()
openstack/octavia-worker-76f48486bc-v4wr2[octavia-worker]: 2023-09-11 18:16:42.399 20 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/lib/python3.10/site-packages/keystoneauth1/adapter.py", line 395, in get
openstack/octavia-worker-76f48486bc-v4wr2[octavia-worker]: 2023-09-11 18:16:42.399 20 ERROR oslo_messaging.rpc.server     return self.request(url, 'GET', **kwargs)
openstack/octavia-worker-76f48486bc-v4wr2[octavia-worker]: 2023-09-11 18:16:42.399 20 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/lib/python3.10/site-packages/barbicanclient/client.py", line 63, in request
openstack/octavia-worker-76f48486bc-v4wr2[octavia-worker]: 2023-09-11 18:16:42.399 20 ERROR oslo_messaging.rpc.server     self._check_status_code(resp)
openstack/octavia-worker-76f48486bc-v4wr2[octavia-worker]: 2023-09-11 18:16:42.399 20 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/lib/python3.10/site-packages/barbicanclient/client.py", line 105, in _check_status_code
openstack/octavia-worker-76f48486bc-v4wr2[octavia-worker]: 2023-09-11 18:16:42.399 20 ERROR oslo_messaging.rpc.server     raise exceptions.HTTPClientError(
openstack/octavia-worker-76f48486bc-v4wr2[octavia-worker]: 2023-09-11 18:16:42.399 20 ERROR oslo_messaging.rpc.server barbicanclient.exceptions.HTTPClientError: Not Found: Secret not found.
openstack/octavia-worker-76f48486bc-v4wr2[octavia-worker]: 2023-09-11 18:16:42.399 20 ERROR oslo_messaging.rpc.server 
openstack/octavia-worker-76f48486bc-v4wr2[octavia-worker]: 2023-09-11 18:16:42.399 20 ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred:
openstack/octavia-worker-76f48486bc-v4wr2[octavia-worker]: 2023-09-11 18:16:42.399 20 ERROR oslo_messaging.rpc.server 
openstack/octavia-worker-76f48486bc-v4wr2[octavia-worker]: 2023-09-11 18:16:42.399 20 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
openstack/octavia-worker-76f48486bc-v4wr2[octavia-worker]: 2023-09-11 18:16:42.399 20 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/lib/python3.10/site-packages/octavia/common/tls_utils/cert_parser.py", line 359, in load_certificates_data
openstack/octavia-worker-76f48486bc-v4wr2[octavia-worker]: 2023-09-11 18:16:42.399 20 ERROR oslo_messaging.rpc.server     cert_mngr.get_cert(context,
openstack/octavia-worker-76f48486bc-v4wr2[octavia-worker]: 2023-09-11 18:16:42.399 20 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/lib/python3.10/site-packages/octavia/certificates/manager/barbican.py", line 123, in get_cert
openstack/octavia-worker-76f48486bc-v4wr2[octavia-worker]: 2023-09-11 18:16:42.399 20 ERROR oslo_messaging.rpc.server     legacy_cert = legacy_mgr.get_cert(
openstack/octavia-worker-76f48486bc-v4wr2[octavia-worker]: 2023-09-11 18:16:42.399 20 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/lib/python3.10/site-packages/octavia/certificates/manager/barbican_legacy.py", line 160, in get_cert
openstack/octavia-worker-76f48486bc-v4wr2[octavia-worker]: 2023-09-11 18:16:42.399 20 ERROR oslo_messaging.rpc.server     with excutils.save_and_reraise_exception():
openstack/octavia-worker-76f48486bc-v4wr2[octavia-worker]: 2023-09-11 18:16:42.399 20 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
openstack/octavia-worker-76f48486bc-v4wr2[octavia-worker]: 2023-09-11 18:16:42.399 20 ERROR oslo_messaging.rpc.server     self.force_reraise()
openstack/octavia-worker-76f48486bc-v4wr2[octavia-worker]: 2023-09-11 18:16:42.399 20 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
openstack/octavia-worker-76f48486bc-v4wr2[octavia-worker]: 2023-09-11 18:16:42.399 20 ERROR oslo_messaging.rpc.server     raise self.value
openstack/octavia-worker-76f48486bc-cz9gs[octavia-worker]: 2023-09-11 18:14:54.208 32 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/lib/python3.10/site-packages/octavia/certificates/manager/barbican_legacy.py", line 138, in get_cert
openstack/octavia-worker-76f48486bc-cz9gs[octavia-worker]: 2023-09-11 18:14:54.208 32 ERROR oslo_messaging.rpc.server     cert_container = connection.containers.get(
openstack/octavia-worker-76f48486bc-cz9gs[octavia-worker]: 2023-09-11 18:14:54.208 32 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/lib/python3.10/site-packages/barbicanclient/v1/containers.py", line 543, in get
openstack/octavia-worker-76f48486bc-cz9gs[octavia-worker]: 2023-09-11 18:14:54.208 32 ERROR oslo_messaging.rpc.server     response = self._api.get(uuid_ref)
openstack/octavia-worker-76f48486bc-cz9gs[octavia-worker]: 2023-09-11 18:14:54.208 32 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/lib/python3.10/site-packages/barbicanclient/client.py", line 70, in get
openstack/octavia-worker-76f48486bc-cz9gs[octavia-worker]: 2023-09-11 18:14:54.208 32 ERROR oslo_messaging.rpc.server     return super(_HTTPClient, self).get(*args, **kwargs).json()
openstack/octavia-worker-76f48486bc-cz9gs[octavia-worker]: 2023-09-11 18:14:54.208 32 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/lib/python3.10/site-packages/keystoneauth1/adapter.py", line 395, in get
openstack/octavia-worker-76f48486bc-cz9gs[octavia-worker]: 2023-09-11 18:14:54.208 32 ERROR oslo_messaging.rpc.server     return self.request(url, 'GET', **kwargs)
openstack/octavia-worker-76f48486bc-cz9gs[octavia-worker]: 2023-09-11 18:14:54.208 32 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/lib/python3.10/site-packages/barbicanclient/client.py", line 63, in request
openstack/octavia-worker-76f48486bc-cz9gs[octavia-worker]: 2023-09-11 18:14:54.208 32 ERROR oslo_messaging.rpc.server     self._check_status_code(resp)
openstack/octavia-worker-76f48486bc-cz9gs[octavia-worker]: 2023-09-11 18:14:54.208 32 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/lib/python3.10/site-packages/barbicanclient/client.py", line 105, in _check_status_code
openstack/octavia-worker-76f48486bc-cz9gs[octavia-worker]: 2023-09-11 18:14:54.208 32 ERROR oslo_messaging.rpc.server     raise exceptions.HTTPClientError(
openstack/octavia-worker-76f48486bc-cz9gs[octavia-worker]: 2023-09-11 18:14:54.208 32 ERROR oslo_messaging.rpc.server barbicanclient.exceptions.HTTPClientError: Not Found: Secrets container not found.
openstack/octavia-worker-76f48486bc-cz9gs[octavia-worker]: 2023-09-11 18:14:54.208 32 ERROR oslo_messaging.rpc.server 
openstack/octavia-worker-76f48486bc-cz9gs[octavia-worker]: 2023-09-11 18:14:54.208 32 ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred:
openstack/octavia-worker-76f48486bc-cz9gs[octavia-worker]: 2023-09-11 18:14:54.208 32 ERROR oslo_messaging.rpc.server 
openstack/octavia-worker-76f48486bc-cz9gs[octavia-worker]: 2023-09-11 18:14:54.208 32 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
openstack/octavia-worker-76f48486bc-cz9gs[octavia-worker]: 2023-09-11 18:14:54.208 32 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/lib/python3.10/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
openstack/octavia-worker-76f48486bc-cz9gs[octavia-worker]: 2023-09-11 18:14:54.208 32 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
openstack/octavia-worker-76f48486bc-cz9gs[octavia-worker]: 2023-09-11 18:14:54.208 32 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/lib/python3.10/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
openstack/octavia-worker-76f48486bc-cz9gs[octavia-worker]: 2023-09-11 18:14:54.208 32 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
openstack/octavia-worker-76f48486bc-cz9gs[octavia-worker]: 2023-09-11 18:14:54.208 32 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/lib/python3.10/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
openstack/octavia-worker-76f48486bc-cz9gs[octavia-worker]: 2023-09-11 18:14:54.208 32 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
openstack/octavia-worker-76f48486bc-cz9gs[octavia-worker]: 2023-09-11 18:14:54.208 32 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/lib/python3.10/site-packages/octavia/controller/queue/v2/endpoints.py", line 56, in delete_load_balancer
openstack/octavia-worker-76f48486bc-cz9gs[octavia-worker]: 2023-09-11 18:14:54.208 32 ERROR oslo_messaging.rpc.server     self.worker.delete_load_balancer(loadbalancer, cascade)
openstack/octavia-worker-76f48486bc-cz9gs[octavia-worker]: 2023-09-11 18:14:54.208 32 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/lib/python3.10/site-packages/octavia/controller/worker/v2/controller_worker.py", line 383, in delete_load_balancer
openstack/octavia-worker-76f48486bc-cz9gs[octavia-worker]: 2023-09-11 18:14:54.208 32 ERROR oslo_messaging.rpc.server     listeners = flow_utils.get_listeners_on_lb(db_lb)
openstack/octavia-worker-76f48486bc-cz9gs[octavia-worker]: 2023-09-11 18:14:54.208 32 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/lib/python3.10/site-packages/octavia/controller/worker/v2/flows/flow_utils.py", line 52, in get_listeners_on_lb
openstack/octavia-worker-76f48486bc-cz9gs[octavia-worker]: 2023-09-11 18:14:54.208 32 ERROR oslo_messaging.rpc.server     prov_listener = provider_utils.db_listener_to_provider_listener(
openstack/octavia-worker-76f48486bc-cz9gs[octavia-worker]: 2023-09-11 18:14:54.208 32 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/lib/python3.10/site-packages/octavia/api/drivers/utils.py", line 182, in db_listener_to_provider_listener
openstack/octavia-worker-76f48486bc-cz9gs[octavia-worker]: 2023-09-11 18:14:54.208 32 ERROR oslo_messaging.rpc.server     new_listener_dict = listener_dict_to_provider_dict(
openstack/octavia-worker-76f48486bc-v4wr2[octavia-worker]: 2023-09-11 18:16:42.399 20 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/lib/python3.10/site-packages/octavia/certificates/manager/barbican_legacy.py", line 138, in get_cert
openstack/octavia-worker-76f48486bc-v4wr2[octavia-worker]: 2023-09-11 18:16:42.399 20 ERROR oslo_messaging.rpc.server     cert_container = connection.containers.get(
openstack/octavia-worker-76f48486bc-v4wr2[octavia-worker]: 2023-09-11 18:16:42.399 20 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/lib/python3.10/site-packages/barbicanclient/v1/containers.py", line 543, in get
openstack/octavia-worker-76f48486bc-v4wr2[octavia-worker]: 2023-09-11 18:16:42.399 20 ERROR oslo_messaging.rpc.server     response = self._api.get(uuid_ref)
openstack/octavia-worker-76f48486bc-v4wr2[octavia-worker]: 2023-09-11 18:16:42.399 20 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/lib/python3.10/site-packages/barbicanclient/client.py", line 70, in get
openstack/octavia-worker-76f48486bc-v4wr2[octavia-worker]: 2023-09-11 18:16:42.399 20 ERROR oslo_messaging.rpc.server     return super(_HTTPClient, self).get(*args, **kwargs).json()
openstack/octavia-worker-76f48486bc-v4wr2[octavia-worker]: 2023-09-11 18:16:42.399 20 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/lib/python3.10/site-packages/keystoneauth1/adapter.py", line 395, in get
openstack/octavia-worker-76f48486bc-v4wr2[octavia-worker]: 2023-09-11 18:16:42.399 20 ERROR oslo_messaging.rpc.server     return self.request(url, 'GET', **kwargs)
openstack/octavia-worker-76f48486bc-v4wr2[octavia-worker]: 2023-09-11 18:16:42.399 20 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/lib/python3.10/site-packages/barbicanclient/client.py", line 63, in request
openstack/octavia-worker-76f48486bc-v4wr2[octavia-worker]: 2023-09-11 18:16:42.399 20 ERROR oslo_messaging.rpc.server     self._check_status_code(resp)
openstack/octavia-worker-76f48486bc-v4wr2[octavia-worker]: 2023-09-11 18:16:42.399 20 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/lib/python3.10/site-packages/barbicanclient/client.py", line 105, in _check_status_code
openstack/octavia-worker-76f48486bc-v4wr2[octavia-worker]: 2023-09-11 18:16:42.399 20 ERROR oslo_messaging.rpc.server     raise exceptions.HTTPClientError(
openstack/octavia-worker-76f48486bc-v4wr2[octavia-worker]: 2023-09-11 18:16:42.399 20 ERROR oslo_messaging.rpc.server barbicanclient.exceptions.HTTPClientError: Not Found: Secrets container not found.
openstack/octavia-worker-76f48486bc-v4wr2[octavia-worker]: 2023-09-11 18:16:42.399 20 ERROR oslo_messaging.rpc.server 
openstack/octavia-worker-76f48486bc-v4wr2[octavia-worker]: 2023-09-11 18:16:42.399 20 ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred:
openstack/octavia-worker-76f48486bc-v4wr2[octavia-worker]: 2023-09-11 18:16:42.399 20 ERROR oslo_messaging.rpc.server 
openstack/octavia-worker-76f48486bc-v4wr2[octavia-worker]: 2023-09-11 18:16:42.399 20 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
openstack/octavia-worker-76f48486bc-v4wr2[octavia-worker]: 2023-09-11 18:16:42.399 20 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/lib/python3.10/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
openstack/octavia-worker-76f48486bc-v4wr2[octavia-worker]: 2023-09-11 18:16:42.399 20 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
openstack/octavia-worker-76f48486bc-v4wr2[octavia-worker]: 2023-09-11 18:16:42.399 20 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/lib/python3.10/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
openstack/octavia-worker-76f48486bc-v4wr2[octavia-worker]: 2023-09-11 18:16:42.399 20 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
openstack/octavia-worker-76f48486bc-v4wr2[octavia-worker]: 2023-09-11 18:16:42.399 20 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/lib/python3.10/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
openstack/octavia-worker-76f48486bc-v4wr2[octavia-worker]: 2023-09-11 18:16:42.399 20 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
openstack/octavia-worker-76f48486bc-v4wr2[octavia-worker]: 2023-09-11 18:16:42.399 20 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/lib/python3.10/site-packages/octavia/controller/queue/v2/endpoints.py", line 56, in delete_load_balancer
openstack/octavia-worker-76f48486bc-v4wr2[octavia-worker]: 2023-09-11 18:16:42.399 20 ERROR oslo_messaging.rpc.server     self.worker.delete_load_balancer(loadbalancer, cascade)
openstack/octavia-worker-76f48486bc-v4wr2[octavia-worker]: 2023-09-11 18:16:42.399 20 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/lib/python3.10/site-packages/octavia/controller/worker/v2/controller_worker.py", line 383, in delete_load_balancer
openstack/octavia-worker-76f48486bc-v4wr2[octavia-worker]: 2023-09-11 18:16:42.399 20 ERROR oslo_messaging.rpc.server     listeners = flow_utils.get_listeners_on_lb(db_lb)
openstack/octavia-worker-76f48486bc-v4wr2[octavia-worker]: 2023-09-11 18:16:42.399 20 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/lib/python3.10/site-packages/octavia/controller/worker/v2/flows/flow_utils.py", line 52, in get_listeners_on_lb
openstack/octavia-worker-76f48486bc-v4wr2[octavia-worker]: 2023-09-11 18:16:42.399 20 ERROR oslo_messaging.rpc.server     prov_listener = provider_utils.db_listener_to_provider_listener(
openstack/octavia-worker-76f48486bc-v4wr2[octavia-worker]: 2023-09-11 18:16:42.399 20 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/lib/python3.10/site-packages/octavia/api/drivers/utils.py", line 182, in db_listener_to_provider_listener
openstack/octavia-worker-76f48486bc-cz9gs[octavia-worker]: 2023-09-11 18:14:54.208 32 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/lib/python3.10/site-packages/octavia/api/drivers/utils.py", line 261, in listener_dict_to_provider_dict
openstack/octavia-worker-76f48486bc-cz9gs[octavia-worker]: 2023-09-11 18:14:54.208 32 ERROR oslo_messaging.rpc.server     with excutils.save_and_reraise_exception() as ctxt:
openstack/octavia-worker-76f48486bc-cz9gs[octavia-worker]: 2023-09-11 18:14:54.208 32 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
openstack/octavia-worker-76f48486bc-cz9gs[octavia-worker]: 2023-09-11 18:14:54.208 32 ERROR oslo_messaging.rpc.server     self.force_reraise()
openstack/octavia-worker-76f48486bc-cz9gs[octavia-worker]: 2023-09-11 18:14:54.208 32 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
openstack/octavia-worker-76f48486bc-cz9gs[octavia-worker]: 2023-09-11 18:14:54.208 32 ERROR oslo_messaging.rpc.server     raise self.value
openstack/octavia-worker-76f48486bc-cz9gs[octavia-worker]: 2023-09-11 18:14:54.208 32 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/lib/python3.10/site-packages/octavia/api/drivers/utils.py", line 258, in listener_dict_to_provider_dict
openstack/octavia-worker-76f48486bc-cz9gs[octavia-worker]: 2023-09-11 18:14:54.208 32 ERROR oslo_messaging.rpc.server     cert_dict = cert_parser.load_certificates_data(cert_manager,
openstack/octavia-worker-76f48486bc-cz9gs[octavia-worker]: 2023-09-11 18:14:54.208 32 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/lib/python3.10/site-packages/octavia/common/tls_utils/cert_parser.py", line 365, in load_certificates_data
openstack/octavia-worker-76f48486bc-cz9gs[octavia-worker]: 2023-09-11 18:14:54.208 32 ERROR oslo_messaging.rpc.server     raise exceptions.CertificateRetrievalException(
openstack/octavia-worker-76f48486bc-cz9gs[octavia-worker]: 2023-09-11 18:14:54.208 32 ERROR oslo_messaging.rpc.server octavia.common.exceptions.CertificateRetrievalException: Could not retrieve certificate: https://key-manager.openstack.cloud.local/v1/containers/8ec85cf4-7135-4eec-81a1-bd6ca823792b
openstack/octavia-worker-76f48486bc-cz9gs[octavia-worker]: 2023-09-11 18:14:54.208 32 ERROR oslo_messaging.rpc.server 
openstack/octavia-worker-76f48486bc-v4wr2[octavia-worker]: 2023-09-11 18:16:42.399 20 ERROR oslo_messaging.rpc.server     new_listener_dict = listener_dict_to_provider_dict(
openstack/octavia-worker-76f48486bc-v4wr2[octavia-worker]: 2023-09-11 18:16:42.399 20 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/lib/python3.10/site-packages/octavia/api/drivers/utils.py", line 261, in listener_dict_to_provider_dict
openstack/octavia-worker-76f48486bc-v4wr2[octavia-worker]: 2023-09-11 18:16:42.399 20 ERROR oslo_messaging.rpc.server     with excutils.save_and_reraise_exception() as ctxt:
openstack/octavia-worker-76f48486bc-v4wr2[octavia-worker]: 2023-09-11 18:16:42.399 20 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
openstack/octavia-worker-76f48486bc-v4wr2[octavia-worker]: 2023-09-11 18:16:42.399 20 ERROR oslo_messaging.rpc.server     self.force_reraise()
openstack/octavia-worker-76f48486bc-v4wr2[octavia-worker]: 2023-09-11 18:16:42.399 20 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
openstack/octavia-worker-76f48486bc-v4wr2[octavia-worker]: 2023-09-11 18:16:42.399 20 ERROR oslo_messaging.rpc.server     raise self.value
openstack/octavia-worker-76f48486bc-v4wr2[octavia-worker]: 2023-09-11 18:16:42.399 20 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/lib/python3.10/site-packages/octavia/api/drivers/utils.py", line 258, in listener_dict_to_provider_dict
openstack/octavia-worker-76f48486bc-v4wr2[octavia-worker]: 2023-09-11 18:16:42.399 20 ERROR oslo_messaging.rpc.server     cert_dict = cert_parser.load_certificates_data(cert_manager,
openstack/octavia-worker-76f48486bc-v4wr2[octavia-worker]: 2023-09-11 18:16:42.399 20 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/lib/python3.10/site-packages/octavia/common/tls_utils/cert_parser.py", line 365, in load_certificates_data
openstack/octavia-worker-76f48486bc-v4wr2[octavia-worker]: 2023-09-11 18:16:42.399 20 ERROR oslo_messaging.rpc.server     raise exceptions.CertificateRetrievalException(
openstack/octavia-worker-76f48486bc-v4wr2[octavia-worker]: 2023-09-11 18:16:42.399 20 ERROR oslo_messaging.rpc.server octavia.common.exceptions.CertificateRetrievalException: Could not retrieve certificate: https://key-manager.openstack.cloud.local/v1/containers/8ec85cf4-7135-4eec-81a1-bd6ca823792b
openstack/octavia-worker-76f48486bc-v4wr2[octavia-worker]: 2023-09-11 18:16:42.399 20 ERROR oslo_messaging.rpc.server 

From horizon or cli, we do not get any error message, just the 200 that the delete signal was sent.

[guilhermesteinmuller]

The workaround is inevitably ( when we are talking about octavia ) touching database.

First we need to locate the listener

select * from listener where id="ff617d44-65ca-4a06-80ee-26ffc9d0667c"

You should set the tls_certificate_id column from the ghost secret container to NULL:

update listener set tls_certificate_id=NULL where id="listener_id";

We also need to delete the record inside sni table:

DELETE FROM sni WHERE listener_id="listener_id";

Once thats done, you can remove the load balancer ( remember to use --cascade parameter )

[mnaser]

This should probably be reported upstream to where ever the bug tracker du jour is these days for Octavia.