[octavia-ingress-controller] Could not retrieve certificate

evildevel commented 3 years ago

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug

What happened: I create an Ingress resource (TLS encryption) and get error (see logs from "octavia-ingress-controller")

apiVersion: apps/v1
kind: Deployment
metadata:
  name: test
spec:
  replicas: 2
  selector:
    matchLabels:
      app: test
  template:
    metadata:
      name: test
      labels:
        app: test
    spec:
      containers:
      - name: test
        image: test/test
      imagePullSecrets:
      - name: test
---
apiVersion: v1
kind: Service
metadata:
  name: test
  labels:
    app: test
spec:
  ports:
  - port: 8780
    targetPort: 8780
    protocol: TCP
  selector:
    app: test
  type: NodePort
---
apiVersion: v1
kind: Secret
metadata:
  name: test
data:
  tls.crt: xxx
  tls.key: xxx
type: kubernetes.io/tls
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: test
  annotations:
    kubernetes.io/ingress.class: "openstack"
    octavia.ingress.kubernetes.io/internal: "false"
    ingress.kubernetes.io/ssl-redirect: "true"
spec:
  rules:
  - host: test.cloud
    http:
      paths:
      - path: /
        backend:
          serviceName: test
          servicePort: 8780
  tls:
  - hosts:
    - test.cloud
    secretName: test

Logs from "octavia-ingress-controller" in "octavia-ingress-controller-0"

I0929 11:39:30.764649       1 keymanager.go:47] OpenStack Request URL: GET http://xxx.xxx.xxx.xxx:9311/v1/secrets?name=kube_ingress_109310cf-8811-405e-a970-7d9bc8fb46cd_default_landing_landing-tls
I0929 11:39:30.764715       1 keymanager.go:47] OpenStack Request Headers:
I0929 11:39:30.764724       1 keymanager.go:47] Accept: application/json
I0929 11:39:30.764732       1 keymanager.go:47] User-Agent: octavia-ingress-controller/latest gophercloud/2.0.0
I0929 11:39:30.764740       1 keymanager.go:47] X-Auth-Token: ***
I0929 11:39:30.805040       1 keymanager.go:47] OpenStack Response Code: 200
I0929 11:39:30.805122       1 keymanager.go:47] OpenStack Response Headers:
I0929 11:39:30.805135       1 keymanager.go:47] Content-Length: 27
I0929 11:39:30.805145       1 keymanager.go:47] Content-Type: application/json
I0929 11:39:30.805179       1 keymanager.go:47] Date: Tue, 29 Sep 2020 11:39:30 GMT
I0929 11:39:30.805235       1 keymanager.go:47] Server: Apache/2.4.29 (Ubuntu)
I0929 11:39:30.805248       1 keymanager.go:47] X-Openstack-Request-Id: req-a6da3add-ca48-491b-a132-222faa192477
I0929 11:39:30.805401       1 keymanager.go:47] OpenStack Response Body: {
I0929 11:39:30.805418       1 keymanager.go:47]   "secrets": [],
I0929 11:39:30.805494       1 keymanager.go:47]   "total": 0
I0929 11:39:30.805514       1 keymanager.go:47] }

I0929 11:39:30.806045       1 keymanager.go:78] OpenStack Request URL: POST http://xxx.xxx.xxx.xxx:9311/v1/secrets
I0929 11:39:30.806105       1 keymanager.go:78] OpenStack Request Headers:
I0929 11:39:30.806117       1 keymanager.go:78] Accept: application/json
I0929 11:39:30.806129       1 keymanager.go:78] Content-Type: application/json
I0929 11:39:30.806140       1 keymanager.go:78] User-Agent: octavia-ingress-controller/latest gophercloud/2.0.0
I0929 11:39:30.806152       1 keymanager.go:78] X-Auth-Token: ***
I0929 11:39:30.806534       1 keymanager.go:78] OpenStack Request Body: {
I0929 11:39:30.806567       1 keymanager.go:78]   "algorithm": "aes",
I0929 11:39:30.806580       1 keymanager.go:78]   "bit_length": 256,
I0929 11:39:30.806591       1 keymanager.go:78]   "mode": "cbc",
I0929 11:39:30.806603       1 keymanager.go:78]   "name": "kube_ingress_109310cf-8811-405e-a970-7d9bc8fb46cd_default_landing_landing-tls",
I0929 11:39:30.806615       1 keymanager.go:78]   "payload": "***",
I0929 11:39:30.806636       1 keymanager.go:78]   "payload_content_encoding": "base64",
I0929 11:39:30.806650       1 keymanager.go:78]   "payload_content_type": "application/octet-stream",
I0929 11:39:30.806662       1 keymanager.go:78]   "secret_type": "opaque"
I0929 11:39:30.806681       1 keymanager.go:78] }
I0929 11:39:30.894500       1 keymanager.go:78] OpenStack Response Code: 201
I0929 11:39:30.894564       1 keymanager.go:78] OpenStack Response Headers:
I0929 11:39:30.894576       1 keymanager.go:78] Content-Length: 93
I0929 11:39:30.894586       1 keymanager.go:78] Content-Type: application/json
I0929 11:39:30.894596       1 keymanager.go:78] Date: Tue, 29 Sep 2020 11:39:30 GMT
I0929 11:39:30.894606       1 keymanager.go:78] Location: http://xxx.xxx.xxx.xxx:9311/v1/secrets/17579246-2f08-4420-8073-0ff252723d37
I0929 11:39:30.894617       1 keymanager.go:78] Server: Apache/2.4.29 (Ubuntu)
I0929 11:39:30.894633       1 keymanager.go:78] X-Openstack-Request-Id: req-be1d5ae0-ca94-4387-9892-44c64446ac3e
I0929 11:39:30.894747       1 keymanager.go:78] OpenStack Response Body: {
I0929 11:39:30.894768       1 keymanager.go:78]   "secret_ref": "http://xxx.xxx.xxx.xxx:9311/v1/secrets/17579246-2f08-4420-8073-0ff252723d37"
I0929 11:39:30.894779       1 keymanager.go:78] }
time="2020-09-29T11:39:30Z" level=info msg="secret created in Barbican" ingress=default/landing secretName=kube_ingress_109310cf-8811-405e-a970-7d9bc8fb46cd_default_landing_landing-tls secretRef="http://xxx.xxx.xxx.xxx:9311/v1/secrets/17579246-2f08-4420-8073-0ff252723d37"

I0929 11:39:30.895244       1 loadbalancer.go:206] OpenStack Request URL: GET http://xxx.xxx.xxx.xxx:9876/v2.0/lbaas/listeners?loadbalancer_id=03851c99-c7a3-4726-9a1b-9ba6fab61689&name=kube_ingress_109310cf-8811-405e-a970-7d9bc8fb46cd_default_landing
I0929 11:39:30.895322       1 loadbalancer.go:206] OpenStack Request Headers:
I0929 11:39:30.895342       1 loadbalancer.go:206] Accept: application/json
I0929 11:39:30.895353       1 loadbalancer.go:206] User-Agent: octavia-ingress-controller/latest gophercloud/2.0.0
I0929 11:39:30.895365       1 loadbalancer.go:206] X-Auth-Token: ***
I0929 11:39:30.959740       1 loadbalancer.go:206] OpenStack Response Code: 200
I0929 11:39:30.959829       1 loadbalancer.go:206] OpenStack Response Headers:
I0929 11:39:30.959849       1 loadbalancer.go:206] Content-Length: 40
I0929 11:39:30.959870       1 loadbalancer.go:206] Content-Type: application/json
I0929 11:39:30.959895       1 loadbalancer.go:206] Date: Tue, 29 Sep 2020 11:39:30 GMT
I0929 11:39:30.959911       1 loadbalancer.go:206] Server: WSGIServer/0.2 CPython/3.6.9
I0929 11:39:30.959925       1 loadbalancer.go:206] X-Openstack-Request-Id: req-f097de30-27e6-455f-b34a-3d784a51bfe0
time="2020-09-29T11:39:30Z" level=debug msg="creating listener" lb=03851c99-c7a3-4726-9a1b-9ba6fab61689 listenerName=kube_ingress_109310cf-8811-405e-a970-7d9bc8fb46cd_default_landing
I0929 11:39:30.960121       1 loadbalancer.go:206] OpenStack Response Body: {
I0929 11:39:30.960164       1 loadbalancer.go:206]   "listeners": [],
I0929 11:39:30.960177       1 loadbalancer.go:206]   "listeners_links": []
I0929 11:39:30.960223       1 loadbalancer.go:206] }

I0929 11:39:30.961008       1 octavia.go:263] OpenStack Request URL: POST http://xxx.xxx.xxx.xxx:9876/v2.0/lbaas/listeners
I0929 11:39:30.961095       1 octavia.go:263] OpenStack Request Headers:
I0929 11:39:30.961108       1 octavia.go:263] Accept: application/json
I0929 11:39:30.961118       1 octavia.go:263] Content-Type: application/json
I0929 11:39:30.961129       1 octavia.go:263] User-Agent: octavia-ingress-controller/latest gophercloud/2.0.0
I0929 11:39:30.961140       1 octavia.go:263] X-Auth-Token: ***
I0929 11:39:30.961323       1 octavia.go:263] OpenStack Request Body: {
I0929 11:39:30.961346       1 octavia.go:263]   "listener": {
I0929 11:39:30.961358       1 octavia.go:263]     "default_tls_container_ref": "http://xxx.xxx.xxx.xxx:9311/v1/secrets/17579246-2f08-4420-8073-0ff252723d37",
I0929 11:39:30.961391       1 octavia.go:263]     "loadbalancer_id": "03851c99-c7a3-4726-9a1b-9ba6fab61689",
I0929 11:39:30.961409       1 octavia.go:263]     "name": "kube_ingress_109310cf-8811-405e-a970-7d9bc8fb46cd_default_landing",
I0929 11:39:30.961423       1 octavia.go:263]     "protocol": "TERMINATED_HTTPS",
I0929 11:39:30.961434       1 octavia.go:263]     "protocol_port": 443,
I0929 11:39:30.961446       1 octavia.go:263]     "sni_container_refs": [
I0929 11:39:30.961477       1 octavia.go:263]       "http://xxx.xxx.xxx.xxx:9311/v1/secrets/17579246-2f08-4420-8073-0ff252723d37"
I0929 11:39:30.961497       1 octavia.go:263]     ]
I0929 11:39:30.961509       1 octavia.go:263]   }
I0929 11:39:30.961520       1 octavia.go:263] }
I0929 11:39:32.030455       1 octavia.go:263] OpenStack Response Code: 400
I0929 11:39:32.030607       1 octavia.go:263] OpenStack Response Headers:
I0929 11:39:32.030660       1 octavia.go:263] Content-Length: 251
I0929 11:39:32.030680       1 octavia.go:263] Content-Type: application/json
I0929 11:39:32.030692       1 octavia.go:263] Date: Tue, 29 Sep 2020 11:39:32 GMT
I0929 11:39:32.030704       1 octavia.go:263] Server: WSGIServer/0.2 CPython/3.6.9
I0929 11:39:32.030746       1 octavia.go:263] X-Openstack-Request-Id: req-c47f6584-e44f-42d4-8112-5a0f995b0272
I0929 11:39:32.031070       1 octavia.go:263] OpenStack Response Body: {
I0929 11:39:32.031101       1 octavia.go:263]   "debuginfo": null,
I0929 11:39:32.031113       1 octavia.go:263]   "faultcode": "Client",
I0929 11:39:32.031150       1 octavia.go:263]   "faultstring": "Could not retrieve certificate: ['http://xxx.xxx.xxx.xxx:9311/v1/secrets/17579246-2f08-4420-8073-0ff252723d37', 'http://xxx.xxx.xxx.xxx:9311/v1/secrets/17579246-2f08-4420-8073-0ff252723d37']"
I0929 11:39:32.031165       1 octavia.go:263] }
E0929 11:39:32.031371       1 controller.go:485] failed to create openstack resources for ingress default/landing: error creating listener: Bad request with: [POST http://xxx.xxx.xxx.xxx:9876/v2.0/lbaas/listeners], error message: {"faultcode": "Client", "faultstring": "Could not retrieve certificate: ['http://xxx.xxx.xxx.xxx:9311/v1/secrets/17579246-2f08-4420-8073-0ff252723d37', 'http://xxx.xxx.xxx.xxx:9311/v1/secrets/17579246-2f08-4420-8073-0ff252723d37']", "debuginfo": null}
I0929 11:39:32.031516       1 event.go:278] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"landing", UID:"792bbf60-2396-4c8d-92e2-3f28699af1ae", APIVersion:"networking.k8s.io/v1beta1", ResourceVersion:"2491", FieldPath:""}): type: 'Warning' reason: 'Failed' Failed to create openstack resources for ingress default/landing: error creating listener: Bad request with: [POST http://xxx.xxx.xxx.xxx:9876/v2.0/lbaas/listeners], error message: {"faultcode": "Client", "faultstring": "Could not retrieve certificate: ['http://xxx.xxx.xxx.xxx:9311/v1/secrets/17579246-2f08-4420-8073-0ff252723d37', 'http://xxx.xxx.xxx.xxx:9311/v1/secrets/17579246-2f08-4420-8073-0ff252723d37']", "debuginfo": null}

Logs from "/var/log/octavia/octavia-api.log"

2020-09-29 14:38:47.766 8047 INFO octavia.api.v2.controllers.load_balancer [xxx] Sending create Load Balancer 03851c99-c7a3-4726-9a1b-9ba6fab61689 to provider octavia
2020-09-29 14:39:31.012 8047 DEBUG octavia.db.repositories [xxx] Checking quota for project: 6212190dc8724fe79455dcf37c6f310e object: <class 'octavia.common.data_models.Listener'> check_quota_met /home/xxx/install/src/octavia/octavia/db/repositories.py:374
2020-09-29 14:39:31.021 8047 DEBUG octavia.certificates.manager.barbican [xxx] Setting project ACL for certificate secret... set_acls /home/xxx/install/src/octavia/octavia/certificates/manager/barbican.py:150
2020-09-29 14:39:31.309 8047 DEBUG barbicanclient.client [xxx] Creating Client object Client /usr/local/lib/python3.6/dist-packages/barbicanclient/client.py:156
2020-09-29 14:39:31.310 8047 DEBUG barbicanclient.v1.acls [xxx] Getting ACL for secret href: http://xxx.xxx.xxx.xxx:9311/v1/secrets/17579246-2f08-4420-8073-0ff252723d37/acl get /usr/local/lib/python3.6/dist-packages/barbicanclient/v1/acls.py:485
2020-09-29 14:39:31.545 8047 DEBUG octavia.certificates.manager.barbican [xxx] Setting project ACL for certificate secret... set_acls /home/xxx/install/src/octavia/octavia/certificates/manager/barbican.py:150
2020-09-29 14:39:31.833 8047 DEBUG barbicanclient.client [xxx] Creating Client object Client /usr/local/lib/python3.6/dist-packages/barbicanclient/client.py:156
2020-09-29 14:39:31.833 8047 DEBUG barbicanclient.v1.acls [xxx] Getting ACL for secret href: http://xxx.xxx.xxx.xxx:9311/v1/secrets/17579246-2f08-4420-8073-0ff252723d37/acl get /usr/local/lib/python3.6/dist-packages/barbicanclient/v1/acls.py:485
2020-09-29 14:39:32.027 8047 DEBUG wsme.api [xxx] Client-side error: Could not retrieve certificate: ['http://xxx.xxx.xxx.xxx:9311/v1/secrets/17579246-2f08-4420-8073-0ff252723d37', 'http://xxx.xxx.xxx.xxx:9311/v1/secrets/17579246-2f08-4420-8073-0ff252723d37'] format_exception /usr/local/lib/python3.6/dist-packages/wsme/api.py:223

Logs from "/var/log/apache2/keystone.log"

2020-09-29 14:06:59.658136 2020-09-29 14:06:59.657 29071 DEBUG keystone.server.flask.request_processing.middleware.auth_context [req-b4c6b6a3-f7e3-45ba-8988-53edb5578dfe d3ac14c5568142db9dd5158c85cc92af 6212190dc8724fe79455dcf37c6f310e - 9b424a4b11264d418ec77460de992d18 9b424a4b11264d418ec77460de992d18] RBAC: auth_context: {'token': <TokenModel (audit_id=-hJjWVKuSlaxH3Mm0ZmOtg, audit_chain_id=['-hJjWVKuSlaxH3Mm0ZmOtg']) at 0x7f9f431ccb70>, 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd3ac14c5568142db9dd5158c85cc92af', 'user_domain_id': '9b424a4b11264d418ec77460de992d18', 'system_scope': None, 'project_id': '6212190dc8724fe79455dcf37c6f310e', 'project_domain_id': '9b424a4b11264d418ec77460de992d18', 'roles': ['reader', 'creator', 'member', 'heat_stack_owner', 'k8s_admin', 'load-balancer_member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} fill_context /home/xxx/install/src/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478\x1b[00m
2020-09-29 14:06:59.658487 2020-09-29 14:06:59.658 29071 DEBUG keystone.server.flask.request_processing.req_logging [req-b4c6b6a3-f7e3-45ba-8988-53edb5578dfe d3ac14c5568142db9dd5158c85cc92af 6212190dc8724fe79455dcf37c6f310e - 9b424a4b11264d418ec77460de992d18 9b424a4b11264d418ec77460de992d18] REQUEST_METHOD: `DELETE` log_request_info /home/xxx/install/src/keystone/keystone/server/flask/request_processing/req_logging.py:27\x1b[00m
2020-09-29 14:06:59.658602 2020-09-29 14:06:59.658 29071 DEBUG keystone.server.flask.request_processing.req_logging [req-b4c6b6a3-f7e3-45ba-8988-53edb5578dfe d3ac14c5568142db9dd5158c85cc92af 6212190dc8724fe79455dcf37c6f310e - 9b424a4b11264d418ec77460de992d18 9b424a4b11264d418ec77460de992d18] SCRIPT_NAME: `` log_request_info /home/xxx/install/src/keystone/keystone/server/flask/request_processing/req_logging.py:28\x1b[00m
2020-09-29 14:06:59.658709 2020-09-29 14:06:59.658 29071 DEBUG keystone.server.flask.request_processing.req_logging [req-b4c6b6a3-f7e3-45ba-8988-53edb5578dfe d3ac14c5568142db9dd5158c85cc92af 6212190dc8724fe79455dcf37c6f310e - 9b424a4b11264d418ec77460de992d18 9b424a4b11264d418ec77460de992d18] PATH_INFO: `/v3/OS-TRUST/trusts/87de3225a3e743dd983f02c779c20c39` log_request_info /home/xxx/install/src/keystone/keystone/server/flask/request_processing/req_logging.py:29\x1b[00m
2020-09-29 14:06:59.662450 2020-09-29 14:06:59.662 29071 DEBUG keystone.common.rbac_enforcer.enforcer [req-b4c6b6a3-f7e3-45ba-8988-53edb5578dfe d3ac14c5568142db9dd5158c85cc92af 6212190dc8724fe79455dcf37c6f310e - 9b424a4b11264d418ec77460de992d18 9b424a4b11264d418ec77460de992d18] RBAC: Authorizing `identity:delete_trust(trust_id=87de3225a3e743dd983f02c779c20c39)` enforce_call /home/xxx/install/src/keystone/keystone/common/rbac_enforcer/enforcer.py:442\x1b[00m
2020-09-29 14:06:59.671205 2020-09-29 14:06:59.671 29071 DEBUG keystone.common.rbac_enforcer.enforcer [req-b4c6b6a3-f7e3-45ba-8988-53edb5578dfe d3ac14c5568142db9dd5158c85cc92af 6212190dc8724fe79455dcf37c6f310e - 9b424a4b11264d418ec77460de992d18 9b424a4b11264d418ec77460de992d18] RBAC: Authorization granted enforce_call /home/xxx/install/src/keystone/keystone/common/rbac_enforcer/enforcer.py:449\x1b[00m
2020-09-29 14:06:59.704409 2020-09-29 14:06:59.703 29071 DEBUG keystone.notifications [req-b4c6b6a3-f7e3-45ba-8988-53edb5578dfe d3ac14c5568142db9dd5158c85cc92af 6212190dc8724fe79455dcf37c6f310e - 9b424a4b11264d418ec77460de992d18 9b424a4b11264d418ec77460de992d18] Invoking callback _trust_callback for event identity OS-TRUST:trust deleted for {'resource_info': '87de3225a3e743dd983f02c779c20c39', 'request_id': 'req-b4c6b6a3-f7e3-45ba-8988-53edb5578dfe'} notify_event_callbacks /home/xxx/install/src/keystone/keystone/notifications.py:357\x1b[00m
2020-09-29 14:06:59.733131 2020-09-29 14:06:59.732 29071 DEBUG keystone.notifications [req-b4c6b6a3-f7e3-45ba-8988-53edb5578dfe d3ac14c5568142db9dd5158c85cc92af 6212190dc8724fe79455dcf37c6f310e - 9b424a4b11264d418ec77460de992d18 9b424a4b11264d418ec77460de992d18] Invoking callback _drop_token_cache for event identity OS-TRUST:trust deleted for {'resource_info': '87de3225a3e743dd983f02c779c20c39', 'request_id': 'req-b4c6b6a3-f7e3-45ba-8988-53edb5578dfe'} notify_event_callbacks /home/xxx/install/src/keystone/keystone/notifications.py:357\x1b[00m
2020-09-29 14:06:59.733920 2020-09-29 14:06:59.733 29071 DEBUG keystone.notifications [req-b4c6b6a3-f7e3-45ba-8988-53edb5578dfe d3ac14c5568142db9dd5158c85cc92af 6212190dc8724fe79455dcf37c6f310e - 9b424a4b11264d418ec77460de992d18 9b424a4b11264d418ec77460de992d18] Invoking callback _drop_receipt_cache for event identity OS-TRUST:trust deleted for {'resource_info': '87de3225a3e743dd983f02c779c20c39', 'request_id': 'req-b4c6b6a3-f7e3-45ba-8988-53edb5578dfe'} notify_event_callbacks /home/xxx/install/src/keystone/keystone/notifications.py:357\x1b[00m
2020-09-29 14:09:09.885933 2020-09-29 14:09:09.885 7860 DEBUG keystone.server.flask.request_processing.middleware.auth_context [req-69b452bf-7aa0-44e9-ad25-b0d763ca820c d3ac14c5568142db9dd5158c85cc92af 6212190dc8724fe79455dcf37c6f310e - 9b424a4b11264d418ec77460de992d18 9b424a4b11264d418ec77460de992d18] RBAC: auth_context: {'token': <TokenModel (audit_id=xZW5N05XRKaPB-Yi19qS1A, audit_chain_id=['xZW5N05XRKaPB-Yi19qS1A']) at 0x7fcff76267f0>, 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd3ac14c5568142db9dd5158c85cc92af', 'user_domain_id': '9b424a4b11264d418ec77460de992d18', 'system_scope': None, 'project_id': '6212190dc8724fe79455dcf37c6f310e', 'project_domain_id': '9b424a4b11264d418ec77460de992d18', 'roles': ['heat_stack_owner', 'k8s_admin', 'load-balancer_member', 'member', 'reader', 'creator'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} fill_context /home/xxx/install/src/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478\x1b[00m
2020-09-29 14:09:09.886204 2020-09-29 14:09:09.886 7860 DEBUG keystone.server.flask.request_processing.req_logging [req-69b452bf-7aa0-44e9-ad25-b0d763ca820c d3ac14c5568142db9dd5158c85cc92af 6212190dc8724fe79455dcf37c6f310e - 9b424a4b11264d418ec77460de992d18 9b424a4b11264d418ec77460de992d18] REQUEST_METHOD: `POST` log_request_info /home/xxx/install/src/keystone/keystone/server/flask/request_processing/req_logging.py:27\x1b[00m
2020-09-29 14:09:09.886319 2020-09-29 14:09:09.886 7860 DEBUG keystone.server.flask.request_processing.req_logging [req-69b452bf-7aa0-44e9-ad25-b0d763ca820c d3ac14c5568142db9dd5158c85cc92af 6212190dc8724fe79455dcf37c6f310e - 9b424a4b11264d418ec77460de992d18 9b424a4b11264d418ec77460de992d18] SCRIPT_NAME: `` log_request_info /home/xxx/install/src/keystone/keystone/server/flask/request_processing/req_logging.py:28\x1b[00m
2020-09-29 14:09:09.886430 2020-09-29 14:09:09.886 7860 DEBUG keystone.server.flask.request_processing.req_logging [req-69b452bf-7aa0-44e9-ad25-b0d763ca820c d3ac14c5568142db9dd5158c85cc92af 6212190dc8724fe79455dcf37c6f310e - 9b424a4b11264d418ec77460de992d18 9b424a4b11264d418ec77460de992d18] PATH_INFO: `/v3/OS-TRUST/trusts` log_request_info /home/xxx/install/src/keystone/keystone/server/flask/request_processing/req_logging.py:29\x1b[00m
2020-09-29 14:09:09.887209 2020-09-29 14:09:09.887 7860 DEBUG keystone.common.rbac_enforcer.enforcer [req-69b452bf-7aa0-44e9-ad25-b0d763ca820c d3ac14c5568142db9dd5158c85cc92af 6212190dc8724fe79455dcf37c6f310e - 9b424a4b11264d418ec77460de992d18 9b424a4b11264d418ec77460de992d18] RBAC: Authorizing `identity:create_trust()` enforce_call /home/xxx/install/src/keystone/keystone/common/rbac_enforcer/enforcer.py:442\x1b[00m
2020-09-29 14:09:09.900167 2020-09-29 14:09:09.899 7860 DEBUG keystone.common.rbac_enforcer.enforcer [req-69b452bf-7aa0-44e9-ad25-b0d763ca820c d3ac14c5568142db9dd5158c85cc92af 6212190dc8724fe79455dcf37c6f310e - 9b424a4b11264d418ec77460de992d18 9b424a4b11264d418ec77460de992d18] RBAC: Authorization granted enforce_call /home/xxx/install/src/keystone/keystone/common/rbac_enforcer/enforcer.py:449\x1b[00m
2020-09-29 14:09:17.999259 2020-09-29 14:09:17.999 7860 DEBUG keystone.server.flask.request_processing.middleware.auth_context [req-43daec0d-547a-46d5-a26c-607987b05479 d3ac14c5568142db9dd5158c85cc92af 6212190dc8724fe79455dcf37c6f310e - 9b424a4b11264d418ec77460de992d18 9b424a4b11264d418ec77460de992d18] RBAC: auth_context: {'token': <TokenModel (audit_id=xZW5N05XRKaPB-Yi19qS1A, audit_chain_id=['xZW5N05XRKaPB-Yi19qS1A']) at 0x7fcff74776a0>, 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd3ac14c5568142db9dd5158c85cc92af', 'user_domain_id': '9b424a4b11264d418ec77460de992d18', 'system_scope': None, 'project_id': '6212190dc8724fe79455dcf37c6f310e', 'project_domain_id': '9b424a4b11264d418ec77460de992d18', 'roles': ['heat_stack_owner', 'k8s_admin', 'load-balancer_member', 'member', 'reader', 'creator'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} fill_context /home/xxx/install/src/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478\x1b[00m
2020-09-29 14:09:17.999533 2020-09-29 14:09:17.999 7860 DEBUG keystone.server.flask.request_processing.req_logging [req-43daec0d-547a-46d5-a26c-607987b05479 d3ac14c5568142db9dd5158c85cc92af 6212190dc8724fe79455dcf37c6f310e - 9b424a4b11264d418ec77460de992d18 9b424a4b11264d418ec77460de992d18] REQUEST_METHOD: `GET` log_request_info /home/xxx/install/src/keystone/keystone/server/flask/request_processing/req_logging.py:27\x1b[00m
2020-09-29 14:09:17.999647 2020-09-29 14:09:17.999 7860 DEBUG keystone.server.flask.request_processing.req_logging [req-43daec0d-547a-46d5-a26c-607987b05479 d3ac14c5568142db9dd5158c85cc92af 6212190dc8724fe79455dcf37c6f310e - 9b424a4b11264d418ec77460de992d18 9b424a4b11264d418ec77460de992d18] SCRIPT_NAME: `` log_request_info /home/xxx/install/src/keystone/keystone/server/flask/request_processing/req_logging.py:28\x1b[00m
2020-09-29 14:09:17.999763 2020-09-29 14:09:17.999 7860 DEBUG keystone.server.flask.request_processing.req_logging [req-43daec0d-547a-46d5-a26c-607987b05479 d3ac14c5568142db9dd5158c85cc92af 6212190dc8724fe79455dcf37c6f310e - 9b424a4b11264d418ec77460de992d18 9b424a4b11264d418ec77460de992d18] PATH_INFO: `/v3/regions` log_request_info /home/xxx/install/src/keystone/keystone/server/flask/request_processing/req_logging.py:29\x1b[00m
2020-09-29 14:09:18.000086 2020-09-29 14:09:17.999 7860 DEBUG keystone.common.rbac_enforcer.enforcer [req-43daec0d-547a-46d5-a26c-607987b05479 d3ac14c5568142db9dd5158c85cc92af 6212190dc8724fe79455dcf37c6f310e - 9b424a4b11264d418ec77460de992d18 9b424a4b11264d418ec77460de992d18] RBAC: Authorizing `identity:list_regions()` enforce_call /home/xxx/install/src/keystone/keystone/common/rbac_enforcer/enforcer.py:442\x1b[00m
2020-09-29 14:09:18.008989 2020-09-29 14:09:18.008 7860 DEBUG keystone.common.rbac_enforcer.enforcer [req-43daec0d-547a-46d5-a26c-607987b05479 d3ac14c5568142db9dd5158c85cc92af 6212190dc8724fe79455dcf37c6f310e - 9b424a4b11264d418ec77460de992d18 9b424a4b11264d418ec77460de992d18] RBAC: Authorization granted enforce_call /home/xxx/install/src/keystone/keystone/common/rbac_enforcer/enforcer.py:449\x1b[00m
2020-09-29 14:09:25.138007 2020-09-29 14:09:25.137 7861 DEBUG keystone.server.flask.request_processing.middleware.auth_context [req-b0501c23-ba6d-421c-a604-901c30f4196f d3ac14c5568142db9dd5158c85cc92af 6212190dc8724fe79455dcf37c6f310e - 9b424a4b11264d418ec77460de992d18 9b424a4b11264d418ec77460de992d18] RBAC: auth_context: {'token': <TokenModel (audit_id=xZW5N05XRKaPB-Yi19qS1A, audit_chain_id=['xZW5N05XRKaPB-Yi19qS1A']) at 0x7fcff743dba8>, 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd3ac14c5568142db9dd5158c85cc92af', 'user_domain_id': '9b424a4b11264d418ec77460de992d18', 'system_scope': None, 'project_id': '6212190dc8724fe79455dcf37c6f310e', 'project_domain_id': '9b424a4b11264d418ec77460de992d18', 'roles': ['k8s_admin', 'creator', 'heat_stack_owner', 'member', 'load-balancer_member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} fill_context /home/xxx/install/src/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478\x1b[00m
2020-09-29 14:09:25.138289 2020-09-29 14:09:25.138 7861 DEBUG keystone.server.flask.request_processing.req_logging [req-b0501c23-ba6d-421c-a604-901c30f4196f d3ac14c5568142db9dd5158c85cc92af 6212190dc8724fe79455dcf37c6f310e - 9b424a4b11264d418ec77460de992d18 9b424a4b11264d418ec77460de992d18] REQUEST_METHOD: `POST` log_request_info /home/xxx/install/src/keystone/keystone/server/flask/request_processing/req_logging.py:27\x1b[00m
2020-09-29 14:09:25.138409 2020-09-29 14:09:25.138 7861 DEBUG keystone.server.flask.request_processing.req_logging [req-b0501c23-ba6d-421c-a604-901c30f4196f d3ac14c5568142db9dd5158c85cc92af 6212190dc8724fe79455dcf37c6f310e - 9b424a4b11264d418ec77460de992d18 9b424a4b11264d418ec77460de992d18] SCRIPT_NAME: `` log_request_info /home/xxx/install/src/keystone/keystone/server/flask/request_processing/req_logging.py:28\x1b[00m
2020-09-29 14:09:25.138535 2020-09-29 14:09:25.138 7861 DEBUG keystone.server.flask.request_processing.req_logging [req-b0501c23-ba6d-421c-a604-901c30f4196f d3ac14c5568142db9dd5158c85cc92af 6212190dc8724fe79455dcf37c6f310e - 9b424a4b11264d418ec77460de992d18 9b424a4b11264d418ec77460de992d18] PATH_INFO: `/v3/OS-TRUST/trusts` log_request_info /home/xxx/install/src/keystone/keystone/server/flask/request_processing/req_logging.py:29\x1b[00m
2020-09-29 14:09:25.139163 2020-09-29 14:09:25.139 7861 DEBUG keystone.common.rbac_enforcer.enforcer [req-b0501c23-ba6d-421c-a604-901c30f4196f d3ac14c5568142db9dd5158c85cc92af 6212190dc8724fe79455dcf37c6f310e - 9b424a4b11264d418ec77460de992d18 9b424a4b11264d418ec77460de992d18] RBAC: Authorizing `identity:create_trust()` enforce_call /home/xxx/install/src/keystone/keystone/common/rbac_enforcer/enforcer.py:442\x1b[00m
2020-09-29 14:09:25.147895 2020-09-29 14:09:25.147 7861 DEBUG keystone.common.rbac_enforcer.enforcer [req-b0501c23-ba6d-421c-a604-901c30f4196f d3ac14c5568142db9dd5158c85cc92af 6212190dc8724fe79455dcf37c6f310e - 9b424a4b11264d418ec77460de992d18 9b424a4b11264d418ec77460de992d18] RBAC: Authorization granted enforce_call /home/xxx/install/src/keystone/keystone/common/rbac_enforcer/enforcer.py:449\x1b[00m
2020-09-29 14:21:00.299969 2020-09-29 14:21:00.299 7863 DEBUG keystone.server.flask.request_processing.middleware.auth_context [req-4e1236a2-514d-4ca9-b452-15ba67f74125 d3ac14c5568142db9dd5158c85cc92af 6212190dc8724fe79455dcf37c6f310e - 9b424a4b11264d418ec77460de992d18 9b424a4b11264d418ec77460de992d18] RBAC: auth_context: {'token': <TokenModel (audit_id=-rKeME3RRRSHdwFvxqFVPA, audit_chain_id=['-rKeME3RRRSHdwFvxqFVPA']) at 0x7fcff6fd8668>, 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd3ac14c5568142db9dd5158c85cc92af', 'user_domain_id': '9b424a4b11264d418ec77460de992d18', 'system_scope': None, 'project_id': '6212190dc8724fe79455dcf37c6f310e', 'project_domain_id': '9b424a4b11264d418ec77460de992d18', 'roles': ['k8s_admin', 'member', 'heat_stack_owner', 'creator', 'load-balancer_member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} fill_context /home/xxx/install/src/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478\x1b[00m
2020-09-29 14:21:00.300324 2020-09-29 14:21:00.300 7863 DEBUG keystone.server.flask.request_processing.req_logging [req-4e1236a2-514d-4ca9-b452-15ba67f74125 d3ac14c5568142db9dd5158c85cc92af 6212190dc8724fe79455dcf37c6f310e - 9b424a4b11264d418ec77460de992d18 9b424a4b11264d418ec77460de992d18] REQUEST_METHOD: `DELETE` log_request_info /home/xxx/install/src/keystone/keystone/server/flask/request_processing/req_logging.py:27\x1b[00m
2020-09-29 14:21:00.300467 2020-09-29 14:21:00.300 7863 DEBUG keystone.server.flask.request_processing.req_logging [req-4e1236a2-514d-4ca9-b452-15ba67f74125 d3ac14c5568142db9dd5158c85cc92af 6212190dc8724fe79455dcf37c6f310e - 9b424a4b11264d418ec77460de992d18 9b424a4b11264d418ec77460de992d18] SCRIPT_NAME: `` log_request_info /home/xxx/install/src/keystone/keystone/server/flask/request_processing/req_logging.py:28\x1b[00m
2020-09-29 14:21:00.300592 2020-09-29 14:21:00.300 7863 DEBUG keystone.server.flask.request_processing.req_logging [req-4e1236a2-514d-4ca9-b452-15ba67f74125 d3ac14c5568142db9dd5158c85cc92af 6212190dc8724fe79455dcf37c6f310e - 9b424a4b11264d418ec77460de992d18 9b424a4b11264d418ec77460de992d18] PATH_INFO: `/v3/OS-TRUST/trusts/e4c2dc960b28452a82ee09ead7ebd489` log_request_info /home/xxx/install/src/keystone/keystone/server/flask/request_processing/req_logging.py:29\x1b[00m
2020-09-29 14:21:00.303528 2020-09-29 14:21:00.303 7863 DEBUG keystone.common.rbac_enforcer.enforcer [req-4e1236a2-514d-4ca9-b452-15ba67f74125 d3ac14c5568142db9dd5158c85cc92af 6212190dc8724fe79455dcf37c6f310e - 9b424a4b11264d418ec77460de992d18 9b424a4b11264d418ec77460de992d18] RBAC: Authorizing `identity:delete_trust(trust_id=e4c2dc960b28452a82ee09ead7ebd489)` enforce_call /home/xxx/install/src/keystone/keystone/common/rbac_enforcer/enforcer.py:442\x1b[00m
2020-09-29 14:21:00.311998 2020-09-29 14:21:00.311 7863 DEBUG keystone.common.rbac_enforcer.enforcer [req-4e1236a2-514d-4ca9-b452-15ba67f74125 d3ac14c5568142db9dd5158c85cc92af 6212190dc8724fe79455dcf37c6f310e - 9b424a4b11264d418ec77460de992d18 9b424a4b11264d418ec77460de992d18] RBAC: Authorization granted enforce_call /home/xxx/install/src/keystone/keystone/common/rbac_enforcer/enforcer.py:449\x1b[00m
2020-09-29 14:21:00.341321 2020-09-29 14:21:00.341 7863 DEBUG keystone.notifications [req-4e1236a2-514d-4ca9-b452-15ba67f74125 d3ac14c5568142db9dd5158c85cc92af 6212190dc8724fe79455dcf37c6f310e - 9b424a4b11264d418ec77460de992d18 9b424a4b11264d418ec77460de992d18] Invoking callback _trust_callback for event identity OS-TRUST:trust deleted for {'resource_info': 'e4c2dc960b28452a82ee09ead7ebd489', 'request_id': 'req-4e1236a2-514d-4ca9-b452-15ba67f74125'} notify_event_callbacks /home/xxx/install/src/keystone/keystone/notifications.py:357\x1b[00m
2020-09-29 14:21:00.385770 2020-09-29 14:21:00.385 7863 DEBUG keystone.notifications [req-4e1236a2-514d-4ca9-b452-15ba67f74125 d3ac14c5568142db9dd5158c85cc92af 6212190dc8724fe79455dcf37c6f310e - 9b424a4b11264d418ec77460de992d18 9b424a4b11264d418ec77460de992d18] Invoking callback _drop_token_cache for event identity OS-TRUST:trust deleted for {'resource_info': 'e4c2dc960b28452a82ee09ead7ebd489', 'request_id': 'req-4e1236a2-514d-4ca9-b452-15ba67f74125'} notify_event_callbacks /home/xxx/install/src/keystone/keystone/notifications.py:357\x1b[00m
2020-09-29 14:21:00.385919 2020-09-29 14:21:00.385 7863 DEBUG keystone.notifications [req-4e1236a2-514d-4ca9-b452-15ba67f74125 d3ac14c5568142db9dd5158c85cc92af 6212190dc8724fe79455dcf37c6f310e - 9b424a4b11264d418ec77460de992d18 9b424a4b11264d418ec77460de992d18] Invoking callback _drop_receipt_cache for event identity OS-TRUST:trust deleted for {'resource_info': 'e4c2dc960b28452a82ee09ead7ebd489', 'request_id': 'req-4e1236a2-514d-4ca9-b452-15ba67f74125'} notify_event_callbacks /home/xxx/install/src/keystone/keystone/notifications.py:357\x1b[00m
2020-09-29 14:26:30.477101 2020-09-29 14:26:30.476 7860 DEBUG keystone.server.flask.request_processing.middleware.auth_context [req-829cf863-37ae-4960-b367-d121770715cd d3ac14c5568142db9dd5158c85cc92af 6212190dc8724fe79455dcf37c6f310e - 9b424a4b11264d418ec77460de992d18 9b424a4b11264d418ec77460de992d18] RBAC: auth_context: {'token': <TokenModel (audit_id=INNLmzrfTdWxKjSOB0qqWQ, audit_chain_id=['INNLmzrfTdWxKjSOB0qqWQ']) at 0x7fcff9dbbd30>, 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd3ac14c5568142db9dd5158c85cc92af', 'user_domain_id': '9b424a4b11264d418ec77460de992d18', 'system_scope': None, 'project_id': '6212190dc8724fe79455dcf37c6f310e', 'project_domain_id': '9b424a4b11264d418ec77460de992d18', 'roles': ['heat_stack_owner', 'k8s_admin', 'load-balancer_member', 'member', 'reader', 'creator'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} fill_context /home/xxx/install/src/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478\x1b[00m
2020-09-29 14:26:30.477372 2020-09-29 14:26:30.477 7860 DEBUG keystone.server.flask.request_processing.req_logging [req-829cf863-37ae-4960-b367-d121770715cd d3ac14c5568142db9dd5158c85cc92af 6212190dc8724fe79455dcf37c6f310e - 9b424a4b11264d418ec77460de992d18 9b424a4b11264d418ec77460de992d18] REQUEST_METHOD: `POST` log_request_info /home/xxx/install/src/keystone/keystone/server/flask/request_processing/req_logging.py:27\x1b[00m
2020-09-29 14:26:30.477491 2020-09-29 14:26:30.477 7860 DEBUG keystone.server.flask.request_processing.req_logging [req-829cf863-37ae-4960-b367-d121770715cd d3ac14c5568142db9dd5158c85cc92af 6212190dc8724fe79455dcf37c6f310e - 9b424a4b11264d418ec77460de992d18 9b424a4b11264d418ec77460de992d18] SCRIPT_NAME: `` log_request_info /home/xxx/install/src/keystone/keystone/server/flask/request_processing/req_logging.py:28\x1b[00m
2020-09-29 14:26:30.477608 2020-09-29 14:26:30.477 7860 DEBUG keystone.server.flask.request_processing.req_logging [req-829cf863-37ae-4960-b367-d121770715cd d3ac14c5568142db9dd5158c85cc92af 6212190dc8724fe79455dcf37c6f310e - 9b424a4b11264d418ec77460de992d18 9b424a4b11264d418ec77460de992d18] PATH_INFO: `/v3/OS-TRUST/trusts` log_request_info /home/xxx/install/src/keystone/keystone/server/flask/request_processing/req_logging.py:29\x1b[00m
2020-09-29 14:26:30.478239 2020-09-29 14:26:30.478 7860 DEBUG keystone.common.rbac_enforcer.enforcer [req-829cf863-37ae-4960-b367-d121770715cd d3ac14c5568142db9dd5158c85cc92af 6212190dc8724fe79455dcf37c6f310e - 9b424a4b11264d418ec77460de992d18 9b424a4b11264d418ec77460de992d18] RBAC: Authorizing `identity:create_trust()` enforce_call /home/xxx/install/src/keystone/keystone/common/rbac_enforcer/enforcer.py:442\x1b[00m
2020-09-29 14:26:30.486626 2020-09-29 14:26:30.486 7860 DEBUG keystone.common.rbac_enforcer.enforcer [req-829cf863-37ae-4960-b367-d121770715cd d3ac14c5568142db9dd5158c85cc92af 6212190dc8724fe79455dcf37c6f310e - 9b424a4b11264d418ec77460de992d18 9b424a4b11264d418ec77460de992d18] RBAC: Authorization granted enforce_call /home/xxx/install/src/keystone/keystone/common/rbac_enforcer/enforcer.py:449\x1b[00m
2020-09-29 14:26:32.148959 2020-09-29 14:26:32.148 7860 DEBUG keystone.server.flask.request_processing.middleware.auth_context [req-829cf863-37ae-4960-b367-d121770715cd d3ac14c5568142db9dd5158c85cc92af 6212190dc8724fe79455dcf37c6f310e - 9b424a4b11264d418ec77460de992d18 9b424a4b11264d418ec77460de992d18] Authenticating user token process_request /usr/local/lib/python3.6/dist-packages/keystonemiddleware/auth_token/__init__.py:407\x1b[00m
2020-09-29 14:26:32.149388 2020-09-29 14:26:32.149 7860 DEBUG keystone.common.fernet_utils [req-829cf863-37ae-4960-b367-d121770715cd d3ac14c5568142db9dd5158c85cc92af 6212190dc8724fe79455dcf37c6f310e - 9b424a4b11264d418ec77460de992d18 9b424a4b11264d418ec77460de992d18] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 5`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? load_keys /home/xxx/install/src/keystone/keystone/common/fernet_utils.py:289\x1b[00m
2020-09-29 14:26:32.158666 2020-09-29 14:26:32.158 7860 DEBUG keystone.server.flask.request_processing.middleware.auth_context [req-829cf863-37ae-4960-b367-d121770715cd d3ac14c5568142db9dd5158c85cc92af 6212190dc8724fe79455dcf37c6f310e - 9b424a4b11264d418ec77460de992d18 9b424a4b11264d418ec77460de992d18] Validating token access rules against request validate_allowed_request /usr/local/lib/python3.6/dist-packages/keystonemiddleware/auth_token/__init__.py:545\x1b[00m
2020-09-29 14:26:32.214425 2020-09-29 14:26:32.214 7860 DEBUG keystone.server.flask.request_processing.middleware.auth_context [req-0408b5f9-8381-4020-8e7f-fca692e6a712 d3ac14c5568142db9dd5158c85cc92af 6212190dc8724fe79455dcf37c6f310e - 9b424a4b11264d418ec77460de992d18 9b424a4b11264d418ec77460de992d18] RBAC: auth_context: {'token': <TokenModel (audit_id=INNLmzrfTdWxKjSOB0qqWQ, audit_chain_id=['INNLmzrfTdWxKjSOB0qqWQ']) at 0x7fcff6f4c978>, 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd3ac14c5568142db9dd5158c85cc92af', 'user_domain_id': '9b424a4b11264d418ec77460de992d18', 'system_scope': None, 'project_id': '6212190dc8724fe79455dcf37c6f310e', 'project_domain_id': '9b424a4b11264d418ec77460de992d18', 'roles': ['heat_stack_owner', 'k8s_admin', 'load-balancer_member', 'member', 'reader', 'creator'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} fill_context /home/xxx/install/src/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478\x1b[00m
2020-09-29 14:26:32.214705 2020-09-29 14:26:32.214 7860 DEBUG keystone.server.flask.request_processing.req_logging [req-0408b5f9-8381-4020-8e7f-fca692e6a712 d3ac14c5568142db9dd5158c85cc92af 6212190dc8724fe79455dcf37c6f310e - 9b424a4b11264d418ec77460de992d18 9b424a4b11264d418ec77460de992d18] REQUEST_METHOD: `GET` log_request_info /home/xxx/install/src/keystone/keystone/server/flask/request_processing/req_logging.py:27\x1b[00m
2020-09-29 14:26:32.214830 2020-09-29 14:26:32.214 7860 DEBUG keystone.server.flask.request_processing.req_logging [req-0408b5f9-8381-4020-8e7f-fca692e6a712 d3ac14c5568142db9dd5158c85cc92af 6212190dc8724fe79455dcf37c6f310e - 9b424a4b11264d418ec77460de992d18 9b424a4b11264d418ec77460de992d18] SCRIPT_NAME: `` log_request_info /home/xxx/install/src/keystone/keystone/server/flask/request_processing/req_logging.py:28\x1b[00m
2020-09-29 14:26:32.214937 2020-09-29 14:26:32.214 7860 DEBUG keystone.server.flask.request_processing.req_logging [req-0408b5f9-8381-4020-8e7f-fca692e6a712 d3ac14c5568142db9dd5158c85cc92af 6212190dc8724fe79455dcf37c6f310e - 9b424a4b11264d418ec77460de992d18 9b424a4b11264d418ec77460de992d18] PATH_INFO: `/v3/regions` log_request_info /home/xxx/install/src/keystone/keystone/server/flask/request_processing/req_logging.py:29\x1b[00m
2020-09-29 14:26:32.215207 2020-09-29 14:26:32.215 7860 DEBUG keystone.common.rbac_enforcer.enforcer [req-0408b5f9-8381-4020-8e7f-fca692e6a712 d3ac14c5568142db9dd5158c85cc92af 6212190dc8724fe79455dcf37c6f310e - 9b424a4b11264d418ec77460de992d18 9b424a4b11264d418ec77460de992d18] RBAC: Authorizing `identity:list_regions()` enforce_call /home/xxx/install/src/keystone/keystone/common/rbac_enforcer/enforcer.py:442\x1b[00m
2020-09-29 14:26:32.223500 2020-09-29 14:26:32.223 7860 DEBUG keystone.common.rbac_enforcer.enforcer [req-0408b5f9-8381-4020-8e7f-fca692e6a712 d3ac14c5568142db9dd5158c85cc92af 6212190dc8724fe79455dcf37c6f310e - 9b424a4b11264d418ec77460de992d18 9b424a4b11264d418ec77460de992d18] RBAC: Authorization granted enforce_call /home/xxx/install/src/keystone/keystone/common/rbac_enforcer/enforcer.py:449\x1b[00m
2020-09-29 14:26:34.722417 2020-09-29 14:26:34.722 7860 DEBUG keystone.server.flask.request_processing.middleware.auth_context [req-0408b5f9-8381-4020-8e7f-fca692e6a712 d3ac14c5568142db9dd5158c85cc92af 6212190dc8724fe79455dcf37c6f310e - 9b424a4b11264d418ec77460de992d18 9b424a4b11264d418ec77460de992d18] Authenticating user token process_request /usr/local/lib/python3.6/dist-packages/keystonemiddleware/auth_token/__init__.py:407\x1b[00m
2020-09-29 14:26:34.722847 2020-09-29 14:26:34.722 7860 DEBUG keystone.common.fernet_utils [req-0408b5f9-8381-4020-8e7f-fca692e6a712 d3ac14c5568142db9dd5158c85cc92af 6212190dc8724fe79455dcf37c6f310e - 9b424a4b11264d418ec77460de992d18 9b424a4b11264d418ec77460de992d18] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 5`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? load_keys /home/xxx/install/src/keystone/keystone/common/fernet_utils.py:289\x1b[00m
2020-09-29 14:26:34.739035 2020-09-29 14:26:34.738 7860 DEBUG keystone.server.flask.request_processing.middleware.auth_context [req-0408b5f9-8381-4020-8e7f-fca692e6a712 d3ac14c5568142db9dd5158c85cc92af 6212190dc8724fe79455dcf37c6f310e - 9b424a4b11264d418ec77460de992d18 9b424a4b11264d418ec77460de992d18] Validating token access rules against request validate_allowed_request /usr/local/lib/python3.6/dist-packages/keystonemiddleware/auth_token/__init__.py:545\x1b[00m
2020-09-29 14:26:34.793262 2020-09-29 14:26:34.793 7860 DEBUG keystone.server.flask.request_processing.middleware.auth_context [req-8a92fda5-0f09-43ef-bab5-089d6c9ec792 d3ac14c5568142db9dd5158c85cc92af 6212190dc8724fe79455dcf37c6f310e - 9b424a4b11264d418ec77460de992d18 9b424a4b11264d418ec77460de992d18] RBAC: auth_context: {'token': <TokenModel (audit_id=INNLmzrfTdWxKjSOB0qqWQ, audit_chain_id=['INNLmzrfTdWxKjSOB0qqWQ']) at 0x7fcff700d550>, 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd3ac14c5568142db9dd5158c85cc92af', 'user_domain_id': '9b424a4b11264d418ec77460de992d18', 'system_scope': None, 'project_id': '6212190dc8724fe79455dcf37c6f310e', 'project_domain_id': '9b424a4b11264d418ec77460de992d18', 'roles': ['heat_stack_owner', 'k8s_admin', 'load-balancer_member', 'member', 'reader', 'creator'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} fill_context /home/xxx/install/src/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478\x1b[00m
2020-09-29 14:26:34.793529 2020-09-29 14:26:34.793 7860 DEBUG keystone.server.flask.request_processing.req_logging [req-8a92fda5-0f09-43ef-bab5-089d6c9ec792 d3ac14c5568142db9dd5158c85cc92af 6212190dc8724fe79455dcf37c6f310e - 9b424a4b11264d418ec77460de992d18 9b424a4b11264d418ec77460de992d18] REQUEST_METHOD: `POST` log_request_info /home/xxx/install/src/keystone/keystone/server/flask/request_processing/req_logging.py:27\x1b[00m
2020-09-29 14:26:34.793643 2020-09-29 14:26:34.793 7860 DEBUG keystone.server.flask.request_processing.req_logging [req-8a92fda5-0f09-43ef-bab5-089d6c9ec792 d3ac14c5568142db9dd5158c85cc92af 6212190dc8724fe79455dcf37c6f310e - 9b424a4b11264d418ec77460de992d18 9b424a4b11264d418ec77460de992d18] SCRIPT_NAME: `` log_request_info /home/xxx/install/src/keystone/keystone/server/flask/request_processing/req_logging.py:28\x1b[00m
2020-09-29 14:26:34.793750 2020-09-29 14:26:34.793 7860 DEBUG keystone.server.flask.request_processing.req_logging [req-8a92fda5-0f09-43ef-bab5-089d6c9ec792 d3ac14c5568142db9dd5158c85cc92af 6212190dc8724fe79455dcf37c6f310e - 9b424a4b11264d418ec77460de992d18 9b424a4b11264d418ec77460de992d18] PATH_INFO: `/v3/OS-TRUST/trusts` log_request_info /home/xxx/install/src/keystone/keystone/server/flask/request_processing/req_logging.py:29\x1b[00m
2020-09-29 14:26:34.794044 2020-09-29 14:26:34.793 7860 DEBUG keystone.common.rbac_enforcer.enforcer [req-8a92fda5-0f09-43ef-bab5-089d6c9ec792 d3ac14c5568142db9dd5158c85cc92af 6212190dc8724fe79455dcf37c6f310e - 9b424a4b11264d418ec77460de992d18 9b424a4b11264d418ec77460de992d18] RBAC: Authorizing `identity:create_trust()` enforce_call /home/xxx/install/src/keystone/keystone/common/rbac_enforcer/enforcer.py:442\x1b[00m
2020-09-29 14:26:34.803008 2020-09-29 14:26:34.802 7860 DEBUG keystone.common.rbac_enforcer.enforcer [req-8a92fda5-0f09-43ef-bab5-089d6c9ec792 d3ac14c5568142db9dd5158c85cc92af 6212190dc8724fe79455dcf37c6f310e - 9b424a4b11264d418ec77460de992d18 9b424a4b11264d418ec77460de992d18] RBAC: Authorization granted enforce_call /home/xxx/install/src/keystone/keystone/common/rbac_enforcer/enforcer.py:449\x1b[00m
2020-09-29 14:39:31.491168 2020-09-29 14:39:31.490 7862 DEBUG keystone.server.flask.request_processing.middleware.auth_context [req-90f4de9b-d4d8-4d7e-a839-851f8c0384e7 d3ac14c5568142db9dd5158c85cc92af 6212190dc8724fe79455dcf37c6f310e - 9b424a4b11264d418ec77460de992d18 9b424a4b11264d418ec77460de992d18] RBAC: auth_context: {'token': <TokenModel (audit_id=rSjM0AEJQVG-eKc2F36VSw, audit_chain_id=['rSjM0AEJQVG-eKc2F36VSw']) at 0x7fcff7143710>, 'domain_id': None, 'trust_id': '91326f347a2d49e0bb3bc7cec87a12b7', 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd3ac14c5568142db9dd5158c85cc92af', 'user_domain_id': '9b424a4b11264d418ec77460de992d18', 'system_scope': None, 'project_id': '6212190dc8724fe79455dcf37c6f310e', 'project_domain_id': '9b424a4b11264d418ec77460de992d18', 'roles': ['creator', 'reader', 'member', 'k8s_admin', 'heat_stack_owner', 'load-balancer_member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} fill_context /home/xxx/install/src/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478\x1b[00m
2020-09-29 14:39:31.491493 2020-09-29 14:39:31.491 7862 DEBUG keystone.server.flask.request_processing.req_logging [req-90f4de9b-d4d8-4d7e-a839-851f8c0384e7 d3ac14c5568142db9dd5158c85cc92af 6212190dc8724fe79455dcf37c6f310e - 9b424a4b11264d418ec77460de992d18 9b424a4b11264d418ec77460de992d18] REQUEST_METHOD: `POST` log_request_info /home/xxx/install/src/keystone/keystone/server/flask/request_processing/req_logging.py:27\x1b[00m
2020-09-29 14:39:31.491840 2020-09-29 14:39:31.491 7862 DEBUG keystone.server.flask.request_processing.req_logging [req-90f4de9b-d4d8-4d7e-a839-851f8c0384e7 d3ac14c5568142db9dd5158c85cc92af 6212190dc8724fe79455dcf37c6f310e - 9b424a4b11264d418ec77460de992d18 9b424a4b11264d418ec77460de992d18] SCRIPT_NAME: `` log_request_info /home/xxx/install/src/keystone/keystone/server/flask/request_processing/req_logging.py:28\x1b[00m
2020-09-29 14:39:31.491973 2020-09-29 14:39:31.491 7862 DEBUG keystone.server.flask.request_processing.req_logging [req-90f4de9b-d4d8-4d7e-a839-851f8c0384e7 d3ac14c5568142db9dd5158c85cc92af 6212190dc8724fe79455dcf37c6f310e - 9b424a4b11264d418ec77460de992d18 9b424a4b11264d418ec77460de992d18] PATH_INFO: `/v3/auth/tokens` log_request_info /home/xxx/install/src/keystone/keystone/server/flask/request_processing/req_logging.py:29\x1b[00m
2020-09-29 14:39:31.492978 2020-09-29 14:39:31.492 7862 DEBUG keystone.common.fernet_utils [req-90f4de9b-d4d8-4d7e-a839-851f8c0384e7 d3ac14c5568142db9dd5158c85cc92af 6212190dc8724fe79455dcf37c6f310e - 9b424a4b11264d418ec77460de992d18 9b424a4b11264d418ec77460de992d18] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 5`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? load_keys /home/xxx/install/src/keystone/keystone/common/fernet_utils.py:289\x1b[00m
2020-09-29 14:39:31.544141 2020-09-29 14:39:31.543 7862 WARNING keystone.server.flask.application [req-90f4de9b-d4d8-4d7e-a839-851f8c0384e7 d3ac14c5568142db9dd5158c85cc92af 6212190dc8724fe79455dcf37c6f310e - 9b424a4b11264d418ec77460de992d18 9b424a4b11264d418ec77460de992d18] You are not authorized to perform the requested action: Using trust-scoped token to create another token. Create a new trust-scoped token instead.: keystone.exception.ForbiddenAction: You are not authorized to perform the requested action: Using trust-scoped token to create another token. Create a new trust-scoped token instead.\x1b[00m
2020-09-29 14:39:31.838101 2020-09-29 14:39:31.837 7862 DEBUG keystone.server.flask.request_processing.middleware.auth_context [req-90f4de9b-d4d8-4d7e-a839-851f8c0384e7 d3ac14c5568142db9dd5158c85cc92af 6212190dc8724fe79455dcf37c6f310e - 9b424a4b11264d418ec77460de992d18 9b424a4b11264d418ec77460de992d18] Authenticating user token process_request /usr/local/lib/python3.6/dist-packages/keystonemiddleware/auth_token/__init__.py:407\x1b[00m
2020-09-29 14:39:31.862346 2020-09-29 14:39:31.862 7862 DEBUG keystone.server.flask.request_processing.middleware.auth_context [req-90f4de9b-d4d8-4d7e-a839-851f8c0384e7 d3ac14c5568142db9dd5158c85cc92af 6212190dc8724fe79455dcf37c6f310e - 9b424a4b11264d418ec77460de992d18 9b424a4b11264d418ec77460de992d18] Validating token access rules against request validate_allowed_request /usr/local/lib/python3.6/dist-packages/keystonemiddleware/auth_token/__init__.py:545\x1b[00m
2020-09-29 14:39:31.925976 2020-09-29 14:39:31.925 7862 DEBUG keystone.server.flask.request_processing.middleware.auth_context [req-e04f9686-dd6a-498c-a01e-eb5b4ee228c1 d3ac14c5568142db9dd5158c85cc92af 6212190dc8724fe79455dcf37c6f310e - 9b424a4b11264d418ec77460de992d18 9b424a4b11264d418ec77460de992d18] RBAC: auth_context: {'token': <TokenModel (audit_id=rSjM0AEJQVG-eKc2F36VSw, audit_chain_id=['rSjM0AEJQVG-eKc2F36VSw']) at 0x7fcff718df98>, 'domain_id': None, 'trust_id': '91326f347a2d49e0bb3bc7cec87a12b7', 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd3ac14c5568142db9dd5158c85cc92af', 'user_domain_id': '9b424a4b11264d418ec77460de992d18', 'system_scope': None, 'project_id': '6212190dc8724fe79455dcf37c6f310e', 'project_domain_id': '9b424a4b11264d418ec77460de992d18', 'roles': ['creator', 'reader', 'member', 'k8s_admin', 'heat_stack_owner', 'load-balancer_member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} fill_context /home/xxx/install/src/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478\x1b[00m
2020-09-29 14:39:31.926242 2020-09-29 14:39:31.926 7862 DEBUG keystone.server.flask.request_processing.req_logging [req-e04f9686-dd6a-498c-a01e-eb5b4ee228c1 d3ac14c5568142db9dd5158c85cc92af 6212190dc8724fe79455dcf37c6f310e - 9b424a4b11264d418ec77460de992d18 9b424a4b11264d418ec77460de992d18] REQUEST_METHOD: `POST` log_request_info /home/xxx/install/src/keystone/keystone/server/flask/request_processing/req_logging.py:27\x1b[00m
2020-09-29 14:39:31.926361 2020-09-29 14:39:31.926 7862 DEBUG keystone.server.flask.request_processing.req_logging [req-e04f9686-dd6a-498c-a01e-eb5b4ee228c1 d3ac14c5568142db9dd5158c85cc92af 6212190dc8724fe79455dcf37c6f310e - 9b424a4b11264d418ec77460de992d18 9b424a4b11264d418ec77460de992d18] SCRIPT_NAME: `` log_request_info /home/xxx/install/src/keystone/keystone/server/flask/request_processing/req_logging.py:28\x1b[00m
2020-09-29 14:39:31.926469 2020-09-29 14:39:31.926 7862 DEBUG keystone.server.flask.request_processing.req_logging [req-e04f9686-dd6a-498c-a01e-eb5b4ee228c1 d3ac14c5568142db9dd5158c85cc92af 6212190dc8724fe79455dcf37c6f310e - 9b424a4b11264d418ec77460de992d18 9b424a4b11264d418ec77460de992d18] PATH_INFO: `/v3/auth/tokens` log_request_info /home/xxx/install/src/keystone/keystone/server/flask/request_processing/req_logging.py:29\x1b[00m
2020-09-29 14:39:31.927343 2020-09-29 14:39:31.927 7862 DEBUG keystone.common.fernet_utils [req-e04f9686-dd6a-498c-a01e-eb5b4ee228c1 d3ac14c5568142db9dd5158c85cc92af 6212190dc8724fe79455dcf37c6f310e - 9b424a4b11264d418ec77460de992d18 9b424a4b11264d418ec77460de992d18] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 5`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? load_keys /home/xxx/install/src/keystone/keystone/common/fernet_utils.py:289\x1b[00m
2020-09-29 14:39:31.977038 2020-09-29 14:39:31.976 7862 WARNING keystone.server.flask.application [req-e04f9686-dd6a-498c-a01e-eb5b4ee228c1 d3ac14c5568142db9dd5158c85cc92af 6212190dc8724fe79455dcf37c6f310e - 9b424a4b11264d418ec77460de992d18 9b424a4b11264d418ec77460de992d18] You are not authorized to perform the requested action: Using trust-scoped token to create another token. Create a new trust-scoped token instead.: keystone.exception.ForbiddenAction: You are not authorized to perform the requested action: Using trust-scoped token to create another token. Create a new trust-scoped token instead.\x1b[00m
2020-09-29 14:39:51.133154 2020-09-29 14:39:51.133 7862 DEBUG keystone.server.flask.request_processing.middleware.auth_context [req-e04f9686-dd6a-498c-a01e-eb5b4ee228c1 d3ac14c5568142db9dd5158c85cc92af 6212190dc8724fe79455dcf37c6f310e - 9b424a4b11264d418ec77460de992d18 9b424a4b11264d418ec77460de992d18] Authenticating user token process_request /usr/local/lib/python3.6/dist-packages/keystonemiddleware/auth_token/__init__.py:407\x1b[00m
2020-09-29 14:39:51.133548 2020-09-29 14:39:51.133 7862 DEBUG keystone.common.fernet_utils [req-e04f9686-dd6a-498c-a01e-eb5b4ee228c1 d3ac14c5568142db9dd5158c85cc92af 6212190dc8724fe79455dcf37c6f310e - 9b424a4b11264d418ec77460de992d18 9b424a4b11264d418ec77460de992d18] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 5`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? load_keys /home/xxx/install/src/keystone/keystone/common/fernet_utils.py:289\x1b[00m
2020-09-29 14:39:51.189102 2020-09-29 14:39:51.188 7862 DEBUG keystone.server.flask.request_processing.middleware.auth_context [req-e04f9686-dd6a-498c-a01e-eb5b4ee228c1 d3ac14c5568142db9dd5158c85cc92af 6212190dc8724fe79455dcf37c6f310e - 9b424a4b11264d418ec77460de992d18 9b424a4b11264d418ec77460de992d18] Validating token access rules against request validate_allowed_request /usr/local/lib/python3.6/dist-packages/keystonemiddleware/auth_token/__init__.py:545\x1b[00m

Environment:

openstack-cloud-controller-manager version: 1.18
OpenStack version: Ussuri

lingxiankong commented 3 years ago

Hi, the only thing suspicious is:

2020-09-29 14:39:31.544141 2020-09-29 14:39:31.543 7862 WARNING keystone.server.flask.application [req-90f4de9b-d4d8-4d7e-a839-851f8c0384e7 d3ac14c5568142db9dd5158c85cc92af 6212190dc8724fe79455dcf37c6f310e - 9b424a4b11264d418ec77460de992d18 9b424a4b11264d418ec77460de992d18] You are not authorized to perform the requested action: Using trust-scoped token to create another token. Create a new trust-scoped token instead.: keystone.exception.ForbiddenAction: You are not authorized to perform the requested action: Using trust-scoped token to create another token. Create a new trust-scoped token instead.

Are you using trust identity in octavia-ingress-controller config?

evildevel commented 3 years ago

Are you using trust identity in octavia-ingress-controller config?

[octavia-ingress-controller] user-id: addc8b12915043da8d1ee9dbe5e8a830 trust-id: 91326f347a2d49e0bb3bc7cec87a12b7

$ openstack user show addc8b12915043da8d1ee9dbe5e8a830
+---------------------+-----------------------------------------------------------------------+
| Field               | Value                                                                 |
+---------------------+-----------------------------------------------------------------------+
| domain_id           | e71ccce17e014079a682fc6142fcbb52                                      |
| enabled             | True                                                                  |
| id                  | addc8b12915043da8d1ee9dbe5e8a830                                      |
| name                | 109310cf-8811-405e-a970-7d9bc8fb46cd_6212190dc8724fe79455dcf37c6f310e |
| options             | {}                                                                    |
| password_expires_at | None                                                                  |
+---------------------+-----------------------------------------------------------------------+

$ openstack domain show e71ccce17e014079a682fc6142fcbb52
+-------------+-------------------------------------------+
| Field       | Value                                     |
+-------------+-------------------------------------------+
| description | Owns users and projects created by magnum |
| enabled     | True                                      |
| id          | e71ccce17e014079a682fc6142fcbb52          |
| name        | magnum                                    |
| options     | {}                                        |
| tags        | []                                        |
+-------------+-------------------------------------------+

openstack trust show 91326f347a2d49e0bb3bc7cec87a12b7
+----------------------+-----------------------------------------------------------------------+
| Field                | Value                                                                 |
+----------------------+-----------------------------------------------------------------------+
| delegation_depth     | 0                                                                     |
| deleted_at           | None                                                                  |
| expires_at           | None                                                                  |
| id                   | 91326f347a2d49e0bb3bc7cec87a12b7                                      |
| impersonation        | True                                                                  |
| project_id           | 6212190dc8724fe79455dcf37c6f310e                                      |
| redelegated_trust_id | None                                                                  |
| redelegation_count   | 0                                                                     |
| remaining_uses       | None                                                                  |
| roles                | reader load-balancer_member member k8s_admin creator heat_stack_owner |
| trustee_user_id      | addc8b12915043da8d1ee9dbe5e8a830                                      |
| trustor_user_id      | d3ac14c5568142db9dd5158c85cc92af                                      |
+----------------------+-----------------------------------------------------------------------+

user "d3ac14c5568142db9dd5158c85cc92af" - creator k8s cluster

Why is "trust" (91326f347a2d49e0bb3bc7cec87a12b7) not taken into account when executing a request to create a listener via Octavia?

data:
  config: |
    cluster-name: 109310cf-8811-405e-a970-7d9bc8fb46cd
    openstack:
      auth-url: http://ct1.xxx.net:5000/v3
      user-id: addc8b12915043da8d1ee9dbe5e8a830
      password: PYt6jZ5ZxYZYxHb4Se
      trust-id: 91326f347a2d49e0bb3bc7cec87a12b7
      region: ru-msk-1
      ca-file: /etc/kubernetes/ca-bundle.crt
    octavia:
      subnet-id: 484758da-37c2-4f5d-b50e-42e68ff3ab3e
      floating-network-id: a4c97db3-6080-484f-82b5-70963dc1c5c3

lingxiankong commented 3 years ago

I've fixed the same issue for neutron client in Octavia before, but not for barbican client, ref: https://storyboard.openstack.org/#!/story/2007619. So for now, we either need to wait for someone fix the bug or don't use trust in octavia-ingress-controller config.

evildevel commented 3 years ago

@lingxiankong Thanks! Can you fix this problem? I would be very grateful to you.

lingxiankong commented 3 years ago

@lingxiankong Thanks! Can you fix this problem? I would be very grateful to you.

Unfortunately, I have no extra bandwidth for fixing that, you can either add your comment in that issue or ask someone in the Octavia team for help.

Anyway, I will close this issue as it's actually an Octavia bug.

guibrazlima commented 3 years ago

I have the same problem. It’s possible to fix?

lingxiankong commented 3 years ago

@guibrazlima Yes, it's possible to fix, please read the comment above.

guibrazlima commented 3 years ago

@lingxiankong Thanks for the update. Do you know if someone in octavia team is looking to this issue?

lingxiankong commented 3 years ago

@lingxiankong Thanks for the update. Do you know if someone in octavia team is looking to this issue?

I don't think there is someone working on this ATM.

lebonez commented 2 years ago

Seems to be a good place to ask this question. I am having issues with this at the moment. I have found even after removing trust. The octavia service user still cannot access the certificate secret because the project scope does not match the creator scope. Is there something missing? I know in barbican to access secrets the user accessing the secret either needs to be in the acls/project access true and be scoped and have access to creator's project.

Has this really ever worked? All Octavia seems to do is set ACLs for the secret but even after still cannot access with the octavia user in the secret's ACLs because Octavia is "service" project scoped authed. Maybe I'm missing something does the kubernetes cluster need to also be in the service project? My users are on a seperate OpenStack domain so that is impossible in my case.

lingxiankong commented 2 years ago

@lebonez Since Rocky, Octavia is able to set Barbican ACLs on behalf of the user automatically to enable users to create TLS-terminated listeners without having to add the Octavia keystone user id to the ACL list by themselves. Please check your Octavia version and Octavia service logs (if you have the permission)

kubernetes / cloud-provider-openstack

[octavia-ingress-controller] Could not retrieve certificate #1250