search

osism / issues

This repository is used for bug reports that are cross-project or not bound to a specific repository (or to an unknown repository).
https://www.osism.tech
1 stars 1 forks source link

/var/log/kolla full because of ceph rgw keystone spam #1164

Open gravityscout opened 3 weeks ago

gravityscout commented 3 weeks ago

OSISM release version

7.1.2

What's the problem?

Dear OSISM-Team,

we have been encountering an issue where keystone-logs are constantly spammed with authentication errors.

/var/log/kolla/keystone/keystone-apache-public-error.log:2024-10-11 08:05:01.082323 2024-10-11 08:05:01.081 757 ERROR keystone.server.flask.application [None req-cf8450d7-b89e-47fa-b330-d41dcbda1c91 0539c3f5e17a491db620bd40825475e6 de918c13b97640ac832d7048d6375cb2 - - default default] Could not find credential: ddc05fe374286e6ae31aa272da1882a4bb6de52bc002b9733e34f9917e242b1d.: keystone.exception.CredentialNotFound: Could not find credential: ddc05fe374286e6ae31aa272da1882a4bb6de52bc002b9733e34f9917e242b1d

These entries occur on every second of the day. It's the same credential every time. We found the ID 0539c3f5e17a491db620bd40825475e6 to be an OpenStack-User-UUID (user-name "ceph_rgw").

Would love to know if you have any thoughts on this, that could help us determine where this credential might be configured.

References to existing reports

References to existing bug reports, mailing lists, ...

Severity

high

Urgency

medium

berendt commented 3 weeks ago

That's because of the prepare Ceph RGW/Keystone integration in the Cookiecutter configuration repository: https://github.com/osism/cfg-cookiecutter/blob/main/%7B%7Bcookiecutter.project_name%7D%7D/environments/ceph/configuration.yml#L31

You can remove the Ceph RGW/Keystone integration or you can add the missing user & endpoint on the Keystone service with the help of the kolla-ceph-rgw play (osism apply kolla-ceph-rgw).

roharc commented 6 days ago

We tried the playbook: kolla-ceph-rgw with the 

    "rgw keystone admin user": "ceph_rgw"

commented out in environments/ceph/configuration.yaml got the only changed output here:

TASK [service-ks-register : ceph-rgw | Creating users] *************************
changed: [pbctr10lw1] => (item=ceph_rgw -> service)

However, keystone logs still showing same error. We'd appreciate any further hints. Thank you, Oliver.