search

opensearch-project / opensearch-k8s-operator

OpenSearch Kubernetes Operator
Apache License 2.0
385 stars 202 forks source link

[BUG] Operator certificate generation / renewals not working #815

Open albgus opened 4 months ago

albgus commented 4 months ago

I have recently updated the OpenSearch operator to version 2.6.0. This seems to have actually triggered some sort of certificate genration process, as seen in the log entries. However, it seems that only the admin certificate was updated, the http and transport certificate is still the same old version.

The operator has been logging this for hours with no apparent progress.

{"level":"info","ts":"2024-05-16T12:43:40.132Z","msg":"Starting workers","controller":"opensearchactiongroup","controllerGroup":"opensearch.opster.io","controllerKind":"OpensearchActionGroup","worker count":1}
{"level":"info","ts":"2024-05-16T12:43:40.132Z","msg":"Starting workers","controller":"opensearchcomponenttemplate","controllerGroup":"opensearch.opster.io","controllerKind":"OpensearchComponentTemplate","worker count":1}
{"level":"info","ts":"2024-05-16T12:43:40.132Z","msg":"Starting workers","controller":"opensearchindextemplate","controllerGroup":"opensearch.opster.io","controllerKind":"OpensearchIndexTemplate","worker count":1}
{"level":"info","ts":"2024-05-16T12:43:40.132Z","msg":"Starting workers","controller":"opensearchismpolicy","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchISMPolicy","worker count":1}
{"level":"info","ts":"2024-05-16T12:43:40.224Z","msg":"Starting workers","controller":"opensearchrole","controllerGroup":"opensearch.opster.io","controllerKind":"OpensearchRole","worker count":1}
{"level":"info","ts":"2024-05-16T12:43:40.225Z","msg":"Starting workers","controller":"opensearchtenant","controllerGroup":"opensearch.opster.io","controllerKind":"OpensearchTenant","worker count":1}
{"level":"info","ts":"2024-05-16T12:43:40.228Z","msg":"Starting workers","controller":"opensearchuserrolebinding","controllerGroup":"opensearch.opster.io","controllerKind":"OpensearchUserRoleBinding","worker count":1}
{"level":"info","ts":"2024-05-16T12:43:40.228Z","msg":"Starting workers","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","worker count":1}
{"level":"info","ts":"2024-05-16T12:43:40.228Z","msg":"Reconciling OpenSearchCluster","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"opensearch-deployment"},"namespace":"opensearch-deployment","name":"opensearch","reconcileID":"d511df94-9763-429c-b28f-f7df986f5997","cluster":{"name":"opensearch","namespace":"opensearch-deployment"}}
{"level":"info","ts":"2024-05-16T12:43:40.229Z","msg":"Starting workers","controller":"opensearchuser","controllerGroup":"opensearch.opster.io","controllerKind":"OpensearchUser","worker count":1}
{"level":"info","ts":"2024-05-16T12:43:40.244Z","msg":"Generating certificates","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"opensearch-deployment"},"namespace":"opensearch-deployment","name":"opensearch","reconcileID":"d511df94-9763-429c-b28f-f7df986f5997","interface":"transport"}
{"level":"info","ts":"2024-05-16T12:43:40.244Z","msg":"Generating certificates","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"opensearch-deployment"},"namespace":"opensearch-deployment","name":"opensearch","reconcileID":"d511df94-9763-429c-b28f-f7df986f5997","interface":"http"}
{"level":"info","ts":"2024-05-16T12:43:40.848Z","msg":"Generating certificates","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"opensearch-deployment"},"namespace":"opensearch-deployment","name":"opensearch","reconcileID":"d511df94-9763-429c-b28f-f7df986f5997"}
{"level":"info","ts":"2024-05-16T12:43:40.946Z","logger":"KubeAPIWarningLogger","msg":"would violate PodSecurity \"restricted:latest\": allowPrivilegeEscalation != false (container \"dashboards\" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container \"dashboards\" must set securityContext.capabilities.drop=[\"ALL\"]), runAsNonRoot != true (pod or container \"dashboards\" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container \"dashboards\" must set securityContext.seccompProfile.type to \"RuntimeDefault\" or \"Localhost\")"}
{"level":"info","ts":"2024-05-16T12:44:10.985Z","msg":"Reconciling OpenSearchCluster","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"opensearch-deployment"},"namespace":"opensearch-deployment","name":"opensearch","reconcileID":"ce3f1596-3baf-4887-b0e8-a628fb891a21","cluster":{"name":"opensearch","namespace":"opensearch-deployment"}}
{"level":"info","ts":"2024-05-16T12:44:11.023Z","msg":"Generating certificates","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"opensearch-deployment"},"namespace":"opensearch-deployment","name":"opensearch","reconcileID":"ce3f1596-3baf-4887-b0e8-a628fb891a21","interface":"transport"}
{"level":"info","ts":"2024-05-16T12:44:11.023Z","msg":"Generating certificates","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"opensearch-deployment"},"namespace":"opensearch-deployment","name":"opensearch","reconcileID":"ce3f1596-3baf-4887-b0e8-a628fb891a21","interface":"http"}
{"level":"info","ts":"2024-05-16T12:44:11.173Z","msg":"Generating certificates","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"opensearch-deployment"},"namespace":"opensearch-deployment","name":"opensearch","reconcileID":"ce3f1596-3baf-4887-b0e8-a628fb891a21"}
{"level":"info","ts":"2024-05-16T12:44:41.274Z","msg":"Reconciling OpenSearchCluster","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"opensearch-deployment"},"namespace":"opensearch-deployment","name":"opensearch","reconcileID":"2d3f6e99-51e9-4975-aa28-d1b6eba80095","cluster":{"name":"opensearch","namespace":"opensearch-deployment"}}
{"level":"info","ts":"2024-05-16T12:44:41.293Z","msg":"Generating certificates","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"opensearch-deployment"},"namespace":"opensearch-deployment","name":"opensearch","reconcileID":"2d3f6e99-51e9-4975-aa28-d1b6eba80095","interface":"transport"}
{"level":"info","ts":"2024-05-16T12:44:41.294Z","msg":"Generating certificates","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"opensearch-deployment"},"namespace":"opensearch-deployment","name":"opensearch","reconcileID":"2d3f6e99-51e9-4975-aa28-d1b6eba80095","interface":"http"}
{"level":"info","ts":"2024-05-16T12:44:41.536Z","msg":"Generating certificates","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"opensearch-deployment"},"namespace":"opensearch-deployment","name":"opensearch","reconcileID":"2d3f6e99-51e9-4975-aa28-d1b6eba80095"}
{"level":"info","ts":"2024-05-16T12:45:11.647Z","msg":"Reconciling OpenSearchCluster","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"opensearch-deployment"},"namespace":"opensearch-deployment","name":"opensearch","reconcileID":"ee9e9109-d68a-4882-a7b7-de318b2bffa2","cluster":{"name":"opensearch","namespace":"opensearch-deployment"}}
{"level":"info","ts":"2024-05-16T12:45:11.666Z","msg":"Generating certificates","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"opensearch-deployment"},"namespace":"opensearch-deployment","name":"opensearch","reconcileID":"ee9e9109-d68a-4882-a7b7-de318b2bffa2","interface":"transport"}
{"level":"info","ts":"2024-05-16T12:45:11.666Z","msg":"Generating certificates","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"opensearch-deployment"},"namespace":"opensearch-deployment","name":"opensearch","reconcileID":"ee9e9109-d68a-4882-a7b7-de318b2bffa2","interface":"http"}
{"level":"info","ts":"2024-05-16T12:45:11.924Z","msg":"Generating certificates","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"opensearch-deployment"},"namespace":"opensearch-deployment","name":"opensearch","reconcileID":"ee9e9109-d68a-4882-a7b7-de318b2bffa2"}
{"level":"info","ts":"2024-05-16T12:45:42.029Z","msg":"Reconciling OpenSearchCluster","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"opensearch-deployment"},"namespace":"opensearch-deployment","name":"opensearch","reconcileID":"49f68700-b1a1-420c-8d0b-961c38e623e3","cluster":{"name":"opensearch","namespace":"opensearch-deployment"}}
{"level":"info","ts":"2024-05-16T12:45:42.055Z","msg":"Generating certificates","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"opensearch-deployment"},"namespace":"opensearch-deployment","name":"opensearch","reconcileID":"49f68700-b1a1-420c-8d0b-961c38e623e3","interface":"transport"}
{"level":"info","ts":"2024-05-16T12:45:42.055Z","msg":"Generating certificates","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"opensearch-deployment"},"namespace":"opensearch-deployment","name":"opensearch","reconcileID":"49f68700-b1a1-420c-8d0b-961c38e623e3","interface":"http"}
{"level":"info","ts":"2024-05-16T12:45:42.239Z","msg":"Generating certificates","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"opensearch-deployment"},"namespace":"opensearch-deployment","name":"opensearch","reconcileID":"49f68700-b1a1-420c-8d0b-961c38e623e3"}
{"level":"info","ts":"2024-05-16T12:46:12.334Z","msg":"Reconciling OpenSearchCluster","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"opensearch-deployment"},"namespace":"opensearch-deployment","name":"opensearch","reconcileID":"3ddf126d-81dc-456b-b1b6-9e4842f72ba7","cluster":{"name":"opensearch","namespace":"opensearch-deployment"}}
{"level":"info","ts":"2024-05-16T12:46:12.352Z","msg":"Generating certificates","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"opensearch-deployment"},"namespace":"opensearch-deployment","name":"opensearch","reconcileID":"3ddf126d-81dc-456b-b1b6-9e4842f72ba7","interface":"transport"}
{"level":"info","ts":"2024-05-16T12:46:12.352Z","msg":"Generating certificates","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"opensearch-deployment"},"namespace":"opensearch-deployment","name":"opensearch","reconcileID":"3ddf126d-81dc-456b-b1b6-9e4842f72ba7","interface":"http"}
{"level":"info","ts":"2024-05-16T12:46:12.488Z","msg":"Generating certificates","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"opensearch-deployment"},"namespace":"opensearch-deployment","name":"opensearch","reconcileID":"3ddf126d-81dc-456b-b1b6-9e4842f72ba7"}
{"level":"info","ts":"2024-05-16T12:46:42.639Z","msg":"Reconciling OpenSearchCluster","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"opensearch-deployment"},"namespace":"opensearch-deployment","name":"opensearch","reconcileID":"db98e17b-7501-468d-b79c-743b7aa66690","cluster":{"name":"opensearch","namespace":"opensearch-deployment"}}
{"level":"info","ts":"2024-05-16T12:46:42.735Z","msg":"Generating certificates","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"opensearch-deployment"},"namespace":"opensearch-deployment","name":"opensearch","reconcileID":"db98e17b-7501-468d-b79c-743b7aa66690","interface":"transport"}
{"level":"info","ts":"2024-05-16T12:46:42.735Z","msg":"Generating certificates","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"opensearch-deployment"},"namespace":"opensearch-deployment","name":"opensearch","reconcileID":"db98e17b-7501-468d-b79c-743b7aa66690","interface":"http"}
{"level":"info","ts":"2024-05-16T12:46:42.923Z","msg":"Generating certificates","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"opensearch-deployment"},"namespace":"opensearch-deployment","name":"opensearch","reconcileID":"db98e17b-7501-468d-b79c-743b7aa66690"}
{"level":"info","ts":"2024-05-16T12:47:13.027Z","msg":"Reconciling OpenSearchCluster","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"opensearch-deployment"},"namespace":"opensearch-deployment","name":"opensearch","reconcileID":"017ad2d7-2d7a-4251-8c73-bee0c2609db1","cluster":{"name":"opensearch","namespace":"opensearch-deployment"}}
{"level":"info","ts":"2024-05-16T12:47:13.044Z","msg":"Generating certificates","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"opensearch-deployment"},"namespace":"opensearch-deployment","name":"opensearch","reconcileID":"017ad2d7-2d7a-4251-8c73-bee0c2609db1","interface":"transport"}
{"level":"info","ts":"2024-05-16T12:47:13.045Z","msg":"Generating certificates","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"opensearch-deployment"},"namespace":"opensearch-deployment","name":"opensearch","reconcileID":"017ad2d7-2d7a-4251-8c73-bee0c2609db1","interface":"http"}
{"level":"info","ts":"2024-05-16T12:47:13.274Z","msg":"Generating certificates","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"opensearch-deployment"},"namespace":"opensearch-deployment","name":"opensearch","reconcileID":"017ad2d7-2d7a-4251-8c73-bee0c2609db1"}
{"level":"info","ts":"2024-05-16T12:47:43.371Z","msg":"Reconciling OpenSearchCluster","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"opensearch-deployment"},"namespace":"opensearch-deployment","name":"opensearch","reconcileID":"09e712b6-8591-4bff-bd0d-bb14a8a76c50","cluster":{"name":"opensearch","namespace":"opensearch-deployment"}}
{"level":"info","ts":"2024-05-16T12:47:43.389Z","msg":"Generating certificates","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"opensearch-deployment"},"namespace":"opensearch-deployment","name":"opensearch","reconcileID":"09e712b6-8591-4bff-bd0d-bb14a8a76c50","interface":"transport"}
{"level":"info","ts":"2024-05-16T12:47:43.390Z","msg":"Generating certificates","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"opensearch-deployment"},"namespace":"opensearch-deployment","name":"opensearch","reconcileID":"09e712b6-8591-4bff-bd0d-bb14a8a76c50","interface":"http"}
pasztorl commented 4 months ago

+1

pasztorl commented 4 months ago

I've checked with a new cluster install. 2.11.0 works as expected, 2.12.0 the same issue above.

pasztorl commented 4 months ago

update: if I retry multiple times sometimes works sometimes not, race condition?

Jerrimikkihvatai commented 4 months ago

+1 Catch the same error while testing cert renewal after trying this method. Tested on operator versions 2.5.0 and 2.6.0, opensearch 2.13. The operator didn't recreate certs so I restarted pod and got the error. But certs were regenerated. The error disappeared only after cluster redeploy.

prudhvigodithi commented 3 months ago

[Triage] Thanks everyone, I assume this method posted here by @swoehrl-mw worked? @albgus @Jerrimikkihvatai @pasztorl @getsaurabh02