search

osism / issues

This repository is used for bug reports that are cross-project or not bound to a specific repository (or to an unknown repository).
https://www.osism.tech
1 stars 1 forks source link

investigate: keystone service role missing from some deployments #907

Closed artificial-intelligence closed 7 months ago

artificial-intelligence commented 8 months ago

we had at least two reports of upgraded openstack deployments where the service role was missing.

What investigation has found so far:

service registration should take place also during upgrades since this patch:

https://review.opendev.org/c/openstack/kolla-ansible/+/909085/4/ansible/roles/keystone/tasks/upgrade.yml

however many deployment logs are spilled with: "ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service."

it's currently unclear if this is a bug in logging or just an artifact because the logs got produced before the keystone-bootstrap was run and thus the role wasn't just yet available.

I was able so far to find a bug in the code itself. a fresh test deployment is still running (had some other issues first, e.g. https://github.com/osism/testbed/issues/2082 ).

But so far other community members reported that they at least have the role present in new deployments.

one possible bug was found in keystone, which might lead to the irritating log error message pasted above ("could not find role: service"):

https://github.com/openstack/keystone/blob/2ac039b717669bf9744f72161e82bdac46dbfacf/keystone/assignment/role_backends/sql.py#L63 expects to get a reference by role_id but if this is not triggered by an id but by "service" instead - it returns the above error message.

The service_role_id of "service" is at least hard coded to "None" here:

https://github.com/openstack/keystone/blob/2ac039b717669bf9744f72161e82bdac46dbfacf/keystone/cmd/bootstrap.py#L51

artificial-intelligence commented 8 months ago

similar bug in the catlog function it seems:

https://github.com/openstack/keystone/blob/2ac039b717669bf9744f72161e82bdac46dbfacf/keystone/catalog/core.py#L150

2024-02-15 12:57:07.089 958 ERROR keystone.server.flask.application [None req-ce324daf-3e48-4c78-87ef-eda956610577 8062402ba2fa4c8c906cc24e2f8bcdfe 8f050cd71a374e5d93b24e6303133f4d - - default default] Could not find service: k
       │ eystone.: keystone.exception.ServiceNotFound: Could not find service: keystone.
 642   │ 2024-02-15 12:57:07.089 958 ERROR keystone.server.flask.application Traceback (most recent call last):
 643   │ 2024-02-15 12:57:07.089 958 ERROR keystone.server.flask.application   File "/var/lib/kolla/venv/lib/python3.9/site-packages/keystone/catalog/core.py", line 152, in get_service
 644   │ 2024-02-15 12:57:07.089 958 ERROR keystone.server.flask.application     return self.driver.get_service(service_id)
 645   │ 2024-02-15 12:57:07.089 958 ERROR keystone.server.flask.application   File "/var/lib/kolla/venv/lib/python3.9/site-packages/keystone/catalog/backends/sql.py", line 196, in get_service
 646   │ 2024-02-15 12:57:07.089 958 ERROR keystone.server.flask.application     return self._get_service(session, service_id).to_dict()
 647   │ 2024-02-15 12:57:07.089 958 ERROR keystone.server.flask.application   File "/var/lib/kolla/venv/lib/python3.9/site-packages/keystone/catalog/backends/sql.py", line 191, in _get_service
 648   │ 2024-02-15 12:57:07.089 958 ERROR keystone.server.flask.application     raise exception.ServiceNotFound(service_id=service_id)
 649   │ 2024-02-15 12:57:07.089 958 ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: keystone.
 650   │ 2024-02-15 12:57:07.089 958 ERROR keystone.server.flask.application 
 651   │ 2024-02-15 12:57:07.089 958 ERROR keystone.server.flask.application During handling of the above exception, another exception occurred:
2024-02-15 12:57:07.089 958 ERROR keystone.server.flask.application 
 653   │ 2024-02-15 12:57:07.089 958 ERROR keystone.server.flask.application Traceback (most recent call last):
 654   │ 2024-02-15 12:57:07.089 958 ERROR keystone.server.flask.application   File "/var/lib/kolla/venv/lib/python3.9/site-packages/flask/app.py", line 1820, in full_dispatch_request
 655   │ 2024-02-15 12:57:07.089 958 ERROR keystone.server.flask.application     rv = self.dispatch_request()
 656   │ 2024-02-15 12:57:07.089 958 ERROR keystone.server.flask.application   File "/var/lib/kolla/venv/lib/python3.9/site-packages/flask/app.py", line 1796, in dispatch_request
 657   │ 2024-02-15 12:57:07.089 958 ERROR keystone.server.flask.application     return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
 658   │ 2024-02-15 12:57:07.089 958 ERROR keystone.server.flask.application   File "/var/lib/kolla/venv/lib/python3.9/site-packages/flask_restful/__init__.py", line 467, in wrapper
 659   │ 2024-02-15 12:57:07.089 958 ERROR keystone.server.flask.application     resp = resource(*args, **kwargs)
 660   │ 2024-02-15 12:57:07.089 958 ERROR keystone.server.flask.application   File "/var/lib/kolla/venv/lib/python3.9/site-packages/flask/views.py", line 107, in view
 661   │ 2024-02-15 12:57:07.089 958 ERROR keystone.server.flask.application     return current_app.ensure_sync(self.dispatch_request)(**kwargs)
 662   │ 2024-02-15 12:57:07.089 958 ERROR keystone.server.flask.application   File "/var/lib/kolla/venv/lib/python3.9/site-packages/flask_restful/__init__.py", line 582, in dispatch_request
 663   │ 2024-02-15 12:57:07.089 958 ERROR keystone.server.flask.application     resp = meth(*args, **kwargs)
 664   │ 2024-02-15 12:57:07.089 958 ERROR keystone.server.flask.application   File "/var/lib/kolla/venv/lib/python3.9/site-packages/keystone/api/services.py", line 45, in get
 665   │ 2024-02-15 12:57:07.089 958 ERROR keystone.server.flask.application     return self._get_service(service_id)
 666   │ 2024-02-15 12:57:07.089 958 ERROR keystone.server.flask.application   File "/var/lib/kolla/venv/lib/python3.9/site-packages/keystone/api/services.py", line 34, in _get_service
 667   │ 2024-02-15 12:57:07.089 958 ERROR keystone.server.flask.application     return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id))
 668   │ 2024-02-15 12:57:07.089 958 ERROR keystone.server.flask.application   File "/var/lib/kolla/venv/lib/python3.9/site-packages/keystone/common/manager.py", line 115, in wrapped
 669   │ 2024-02-15 12:57:07.089 958 ERROR keystone.server.flask.application     __ret_val = __f(*args, **kwargs)
 670   │ 2024-02-15 12:57:07.089 958 ERROR keystone.server.flask.application   File "/var/lib/kolla/venv/lib/python3.9/site-packages/decorator.py", line 232, in fun
 671   │ 2024-02-15 12:57:07.089 958 ERROR keystone.server.flask.application     return caller(func, *(extras + args), **kw)
 672   │ 2024-02-15 12:57:07.089 958 ERROR keystone.server.flask.application   File "/var/lib/kolla/venv/lib/python3.9/site-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func
 673   │ 2024-02-15 12:57:07.089 958 ERROR keystone.server.flask.application     return self.get_or_create(
 674   │ 2024-02-15 12:57:07.089 958 ERROR keystone.server.flask.application   File "/var/lib/kolla/venv/lib/python3.9/site-packages/dogpile/cache/region.py", line 1042, in get_or_create
 675   │ 2024-02-15 12:57:07.089 958 ERROR keystone.server.flask.application     with Lock(
 676   │ 2024-02-15 12:57:07.089 958 ERROR keystone.server.flask.application   File "/var/lib/kolla/venv/lib/python3.9/site-packages/dogpile/lock.py", line 185, in __enter__
 677   │ 2024-02-15 12:57:07.089 958 ERROR keystone.server.flask.application     return self._enter()
 678   │ 2024-02-15 12:57:07.089 958 ERROR keystone.server.flask.application   File "/var/lib/kolla/venv/lib/python3.9/site-packages/dogpile/lock.py", line 94, in _enter
 679   │ 2024-02-15 12:57:07.089 958 ERROR keystone.server.flask.application     generated = self._enter_create(value, createdtime)
 680   │ 2024-02-15 12:57:07.089 958 ERROR keystone.server.flask.application   File "/var/lib/kolla/venv/lib/python3.9/site-packages/dogpile/lock.py", line 178, in _enter_create
 681   │ 2024-02-15 12:57:07.089 958 ERROR keystone.server.flask.application     return self.creator()
 682   │ 2024-02-15 12:57:07.089 958 ERROR keystone.server.flask.application   File "/var/lib/kolla/venv/lib/python3.9/site-packages/dogpile/cache/region.py", line 995, in gen_value
 683   │ 2024-02-15 12:57:07.089 958 ERROR keystone.server.flask.application     created_value = creator(
 684   │ 2024-02-15 12:57:07.089 958 ERROR keystone.server.flask.application   File "/var/lib/kolla/venv/lib/python3.9/site-packages/keystone/catalog/core.py", line 154, in get_service
 685   │ 2024-02-15 12:57:07.089 958 ERROR keystone.server.flask.application     raise exception.ServiceNotFound(service_id=service_id)
 686   │ 2024-02-15 12:57:07.089 958 ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: keystone.
berendt commented 8 months ago

This is a kolla-ansible issue and not a keystone issue.

This task is only called by the deploy action, not by the upgrade action.

TASK [service-ks-register : keystone | Creating roles] *************************
Friday 08 March 2024  16:20:13 +0000 (0:00:00.052)       0:01:04.112 **********
changed: [testbed-node-0.testbed.osism.xyz] => (item=service)
berendt commented 8 months ago

It should be sufficient to also add https://review.opendev.org/c/openstack/kolla-ansible/+/909002 to our 2023.2 builds to fix this problem.

artificial-intelligence commented 8 months ago

upstream fix pending review: https://review.opendev.org/c/openstack/kolla-ansible/+/912452

I'll make sure to remove our downstream patch once upstream is merged.