Closed jgerry2002 closed 1 year ago
This error also happens during a new install of 17.17.0
2022-11-15 10:17:31,167 ERROR: create_config_service failed
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/patroni/dcs/kubernetes.py", line 950, in _create_config_service
if not self._api.create_namespaced_service(self._namespace, body):
File "/usr/lib/python3/dist-packages/patroni/dcs/kubernetes.py", line 483, in wrapper
return getattr(self._core_v1_api, func)(*args, **kwargs)
File "/usr/lib/python3/dist-packages/patroni/dcs/kubernetes.py", line 419, in wrapper
return self._api_client.call_api(method, path, headers, body, **kwargs)
File "/usr/lib/python3/dist-packages/patroni/dcs/kubernetes.py", line 388, in call_api
return self._handle_server_response(response, _preload_content)
File "/usr/lib/python3/dist-packages/patroni/dcs/kubernetes.py", line 218, in _handle_server_response
raise k8s_client.rest.ApiException(http_resp=response)
patroni.dcs.kubernetes.K8sClient.rest.ApiException: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Audit-Id': '6d0823fa-d627-4ab7-9342-894587a0b624', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': '43fde79c-ff65-437e-b03c-edbdbb29013b', 'X-Kubernetes-Pf-Prioritylevel-Uid': '73f0f59e-1d05-46ee-8872-1f406aa1b018', 'Date': 'Tue, 15 Nov 2022 10:17:31 GMT', 'Content-Length': '299'})
HTTP response body: b'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"services is forbidden: User \\"system:serviceaccount:tobs:tobs-timescaledb\\" cannot create resource \\"services\\" in API group \\"\\" in the namespace \\"tobs\\"","reason":"Forbidden","details":{"kind":"services"},"code":403}\n'
This issue is explained in some detail here with a workaround. I just re-deployed the latest Tobs in hopes that this error was fixed, but can confirm I'm still seeing the issue. https://github.com/timescale/helm-charts/issues/405
After applying the recommended fix, the error does go away, however I'm still not getting any data in Grafana...
This is/was a bug in Patroni which was fixed with https://github.com/zalando/patroni/issues/1132 & https://github.com/zalando/patroni/pull/2390.
We have merged the change to add these fixes into the timescale-db container image https://github.com/timescale/timescaledb-docker-ha/pull/319
The image changes are currently in the process of being tested. I hope to have a release soon.
facing the same issue on latest version.
timescale/timescaledb-ha:pg14.6-ts2.8.1-p0 has been released with the Patroni fix.
I performed a fresh tobs installation with a modified my_values.yaml file that specified the new timescaledb image (timescaledb-ha:pg14.6-ts2.8.1-p0 and I still see the 403 error in the tobs-timescaledb-0 pod logs. The only item I changed in the tobs values file was tag: pg14.6-ts2.8.1-p0. I also confirmed this is the image that was pulled for tobs-timescaledb-0.
I performed a fresh tobs installation with a modified my_values.yaml file that specified the new timescaledb image (timescaledb-ha:pg14.6-ts2.8.1-p0 and I still see the 403 error in the tobs-timescaledb-0 pod logs. The only item I changed in the tobs values file was tag: pg14.6-ts2.8.1-p0. I also confirmed this is the image that was pulled for tobs-timescaledb-0.
Yup you are correct, sorry about that. Can you try again with this image and let me know. timescale/timescaledb:pg14.6-ts2.8.1-patroni-static-primary-p0
I am currently no longer seeing the error logs when using that image.
timescale/timescaledb:pg14.6-ts2.8.1-patroni-static-primary-p0
I also no longer see the error in the logs after re-deploying with this image. Thank you!
Awesome. I am going to close this, please re-open if needed.
What happened? During update to 17.17.0 from 17.15.0 there was a RBAC related error with the service account rights for the tobs-timescaledb role.
Did you expect to see something different? The rights for the service account should be checked/confirmed. The error causes timescaledb to crashloop.
How to reproduce it (as minimally and precisely as possible): Install 17.50.0 via helm, proceed with upgrade to 17.17.0
Environment Kubernetes 1.21.14
tobs version:
17.15.0 with an upgrade to 17.17.0
Kubernetes version information:
kubectl version
Client Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.1", GitCommit:"3ddd0f45aa91e2f30c70734b175631bec5b5825a", GitTreeState:"clean", BuildDate:"2022-05-24T12:26:19Z", GoVersion:"go1.18.2", Compiler:"gc", Platform:"linux/amd64"} Kustomize Version: v4.5.4 Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.14", GitCommit:"0f77da5bd4809927e15d1658fb4aa8f13ad890a5", GitTreeState:"clean", BuildDate:"2022-06-15T14:11:36Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}tobs Logs: 2022-11-15 04:32:07,937 ERROR: create_config_service failed Traceback (most recent call last): File "/usr/lib/python3/dist-packages/patroni/dcs/kubernetes.py", line 950, in _create_config_service if not self._api.create_namespaced_service(self._namespace, body): File "/usr/lib/python3/dist-packages/patroni/dcs/kubernetes.py", line 483, in wrapper return getattr(self._core_v1_api, func)(*args, kwargs) File "/usr/lib/python3/dist-packages/patroni/dcs/kubernetes.py", line 419, in wrapper return self._api_client.call_api(method, path, headers, body, kwargs) File "/usr/lib/python3/dist-packages/patroni/dcs/kubernetes.py", line 388, in call_api return self._handle_server_response(response, _preload_content) File "/usr/lib/python3/dist-packages/patroni/dcs/kubernetes.py", line 218, in _handle_server_response raise k8s_client.rest.ApiException(http_resp=response) patroni.dcs.kubernetes.K8sClient.rest.ApiException: (403) Reason: Forbidden HTTP response headers: HTTPHeaderDict({'Audit-Id': 'f4d1cc89-9193-4363-8ee4-be89d8ea2e2a', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': '34201944-ab28-47ee-8f64-6d9e778d06b0', 'X-Kubernetes-Pf-Prioritylevel-Uid': '1e12bbf7-cd0e-40cf-be91-2bd42febd6e0', 'Date': 'Tue, 15 Nov 2022 04: 32:07 GMT', 'Content-Length': '299'}) HTTP response body: b'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"services is forbidden: User \"system:serviceaccount:tobs:tobs-timescaledb\" cannot creat e resource \"services\" in API group \"\" in the namespace \"tobs\"","reason":"Forbidden","details":{"kind":"services"},"code":403}\n'
Notes I was able to proceed with an update when I manually modified and added the ability for the service account to alter services.