search

zalando / postgres-operator

Postgres operator creates and manages PostgreSQL clusters running in Kubernetes
https://postgres-operator.readthedocs.io/
MIT License
4.28k stars 973 forks source link

Install fails in OC version 4.8 with permission issue #2133

Open batchamalick opened 1 year ago

batchamalick commented 1 year ago

Please, answer some short questions which should help us to understand your problem / question better?

Some general remarks when posting a bug report: My OC version 

oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.8.0     True        False         8m37s   Cluster version is 4.8.0

My config 

apiVersion: v1
items:
- apiVersion: acid.zalan.do/v1
  kind: postgresql
  metadata:
    generation: 
    name: db-ds
    namespace: postgres
  spec:
    databases:
      test: testdb
    dockerImage: registry.opensource.zalan.do/acid/spilo-14:2.1-p7
    numberOfInstances: 1
    postgresql:
      version: "12"
    spiloFSGroup: 103
    spiloRunAsGroup: 103
    spiloRunAsUser: 101
    teamId: test
    users:
      dbuser:
      - superuser
    volume:
      size: 10Gi
  status:
    PostgresClusterStatus: UpdateFailed
kind: List
metadata:
  resourceVersion: ""

I'm seeing permission denied error 

2022-12-07 10:12:43,419 ERROR: failed to update leader lock
2022-12-07 10:12:43,419 INFO: not promoting because failed to update leader lock in DCS
2022-12-07 10:12:49.038 32 LOG {ticks: 0, maint: 0, retry: 0}
2022-12-07 10:12:53,370 INFO: Lock owner: tsb-ds-0; I am tsb-ds-0
2022-12-07 10:12:53,419 ERROR: Permission denied
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/patroni/dcs/kubernetes.py", line 974, in _update_leader_with_retry
    return self._patch_or_create(self.leader_path, annotations, resource_version, ips=ips, retry=_retry)
  File "/usr/local/lib/python3.6/dist-packages/patroni/dcs/kubernetes.py", line 921, in _patch_or_create
    ret = retry(func, self._namespace, body) if retry else func(self._namespace, body)
  File "/usr/local/lib/python3.6/dist-packages/patroni/dcs/kubernetes.py", line 971, in _retry
    return retry(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/patroni/utils.py", line 334, in __call__
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/patroni/dcs/kubernetes.py", line 483, in wrapper
    return getattr(self._core_v1_api, func)(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/patroni/dcs/kubernetes.py", line 419, in wrapper
    return self._api_client.call_api(method, path, headers, body, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/patroni/dcs/kubernetes.py", line 388, in call_api
    return self._handle_server_response(response, _preload_content)
  File "/usr/local/lib/python3.6/dist-packages/patroni/dcs/kubernetes.py", line 218, in _handle_server_response
    raise k8s_client.rest.ApiException(http_resp=response)
patroni.dcs.kubernetes.K8sClient.rest.ApiException: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Audit-Id': '9b001521-bf55-4d0c-a266-8d3c7ebb4fe5', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': 'f7b3b62e-bbd5-4f55-ba41-d2bce7edfeef', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'e026dd8d-e274-45b4-a104-2dc5eb0eb290', 'Date': 'Wed, 07 Dec 2022 10:12:53 GMT', 'Content-Length': '239'})
HTTP response body: b'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"endpoints \\"tsb-ds\\" is forbidden: endpoint address 10.131.0.37 is not allowed","reason":"Forbidden","details":{"name":"tsb-ds","kind":"endpoints"},"code":403}\n'

2022-12-07 10:12:53,419 ERROR: failed to update leader lock
2022-12-07 10:12:53,419 INFO: not promoting because failed to update leader lock in DCS

I also have this clusterrole created 

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    app.kubernetes.io/instance: postgres-operator
    app.kubernetes.io/name: postgres-operator
  name: postgres-pod
rules:
- apiGroups:
  - ""
  resources:
  - configmaps
  verbs:
  - create
  - delete
  - deletecollection
  - get
  - list
  - patch
  - update
  - watch
- apiGroups:
  - ""
  resources:
  - endpoints
  verbs:
  - get
- apiGroups:
  - ""
  resources:
  - endpoints
  verbs:
  - create
  - delete
  - deletecollection
  - get
  - list
  - patch
  - update
  - watch
- apiGroups:
  - ""
  resources:
  - pods
  verbs:
  - get
  - list
  - patch
  - update
  - watch
- apiGroups:
  - ""
  resources:
  - services
  verbs:
  - create
- apiGroups:
  - extensions
  resourceNames:
  - privileged
  resources:
  - podsecuritypolicies
  verbs:
  - use
- apiGroups:
  - ""
  resources:
  - services
  verbs:
  - create
  - patch
  - get
  - list
- apiGroups:
  - ""
  resources:
  - configmaps
  verbs:
  - get
  - list
  - patch
  - update
  - watch
- apiGroups:
  - ""
  resources:
  - endpoints
  verbs:
  - create
  - delete
  - deletecollection
  - get
  - list
  - patch
  - update
  - watch
- apiGroups:
  - ""
  resources:
  - pods
  verbs:
  - get
  - list
  - patch
  - update
  - watch
- apiGroups:
  - ""
  resources:
  - services
  verbs:
  - create
  - patch
  - get
  - list
- apiGroups:
  - ""
  resources:
  - nodes
  verbs:
  - get
  - list
  - watch
samuelchen commented 1 year ago

same issue

2023-01-05 09:04:46,549 ERROR: failed to update leader lock
2023-01-05 09:04:46,550 INFO: not promoting because failed to update leader lock in DCS
2023-01-05 09:04:56,536 INFO: Lock owner: ccr-postgres-cluster-0; I am ccr-postgres-cluster-0
2023-01-05 09:04:56,548 ERROR: Permission denied
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/patroni/dcs/kubernetes.py", line 975, in _update_leader_with_retry
    return self._patch_or_create(self.leader_path, annotations, resource_version, ips=ips, retry=_retry)
  File "/usr/local/lib/python3.10/dist-packages/patroni/dcs/kubernetes.py", line 921, in _patch_or_create
    ret = retry(func, self._namespace, body) if retry else func(self._namespace, body)
  File "/usr/local/lib/python3.10/dist-packages/patroni/dcs/kubernetes.py", line 972, in _retry
    return retry(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/patroni/utils.py", line 334, in __call__
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/patroni/dcs/kubernetes.py", line 483, in wrapper
    return getattr(self._core_v1_api, func)(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/patroni/dcs/kubernetes.py", line 419, in wrapper
    return self._api_client.call_api(method, path, headers, body, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/patroni/dcs/kubernetes.py", line 388, in call_api
    return self._handle_server_response(response, _preload_content)
  File "/usr/local/lib/python3.10/dist-packages/patroni/dcs/kubernetes.py", line 218, in _handle_server_response
    raise k8s_client.rest.ApiException(http_resp=response)
patroni.dcs.kubernetes.K8sClient.rest.ApiException: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Audit-Id': 'b1fb4eb5-8e25-4a35-83c8-fddae0510d61', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': '415f970c-6a42-4b9e-8066-271cd048f096', 'X-Kubernetes-Pf-Prioritylevel-Uid': '3fb00e9b-6c7f-499a-8bb0-e72fa577bc84', 'Date': 'Thu, 05 Jan 2023 09:04:56 GMT', 'Content-Length': '268'})
HTTP response body: b'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"endpoints \\"ccr-postgres-cluster\\" is forbidden: endpoint address 10.128.3.135 is not allowed","reason":"Forbidden","details":{"name":"ccr-postgres-cluster","kind":"endpoints"},"code":403}\n'
Samusername commented 1 year ago

We had troubles with permissions in Openshift.

We ended up adding permissions to make "endpoints mechanism" work in Openshift (instead of using "ConfigMap" mechanisms/mode). Following permissions were needed for endpoints "mode" to work:

endpoints/restricted endpointslices/restricted

https://github.com/zalando/patroni/issues/2148