search

rancher / opni

Multi Cluster Observability with AIOps
https://opni.io
Apache License 2.0
338 stars 53 forks source link

Opni fails to create admin Opensearch user in EKS #1183

Open joshmeranda opened 1 year ago

joshmeranda commented 1 year ago

When installing the logging capability Opni is not able to create the opensearch admin user.

2023-03-14T14:05:48Z DEBUG plugin.logging.opensearch-manager opensearchdata/admin.go:101 creating opensearch admin user
2023-03-14T14:05:48Z ERROR plugin.logging.opensearch-manager opensearchdata/admin.go:69 failed to create admin user: response from API is [403 Forbidden] {"status":"FORBIDDEN","message":"No permission to access REST API: User internalopni with Security roles [own_index] does not have any role privileged for admin access. Security admin permissions required but CN=internalopni is not an admin"}
2023-03-14T14:05:51Z DEBUG apiext management/extensions.go:242 handling http request {"method": "GetOpensearchStatus", "path": "/logging/status"}
2023-03-14T14:05:51Z ERROR plugin.logging.opensearch-manager opensearchdata/status.go:23 failure response from cluster status {"respError": "json: unsupported type: func() string"}

Kubernetes Version: v1.23.16 Provider: Amazon EKS

Logging Primary Pod Replicas: 3 Logging Ingest Pods: Disabled Logging Controlplane Pods: Disabled Logging Dashboard Replicas: 3

@dbason

dbason commented 1 year ago

@joshmeranda can you take a look at the logs from the opni-manager pod. In particular I'm looking for if there are errors there as this use should have been created as part of the initial installation.

joshmeranda commented 1 year ago

@dbason Looks like things start up ok but theres something goping on with tls certs

[14:22:45] ESC[31mERRORESC[0m Reconciler error {"controller": "multiclusterrolebinding", "controllerGroup": "logging.opni.io", "controllerKind": "MulticlusterRoleBinding", "MulticlusterRoleBinding": {"name":"opni","namespace":"opni"}, "namespace": "opni", "name": "opni", "reconcileID": "f7009fcd-6fb9-4dc0-b8e2-4e2a5a4c05f2", "error": "remote error: tls: unknown certificat
e"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
        sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
        sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
[14:22:46] ESC[34mINFOESC[0m Reconciling OpenSearchCluster {"controller": "opensearchcluster", "controllerGroup": "opensearch.opster.io", "controllerKind": "OpenSearchCluster", "OpenSearchCluster": {"name":"opni","namespace":"opni"}, "namespace": "opni", "name": "opni", "reconcileID": "8008c978-612a-4e43-8253-5127895d99ac", "cluster": "opni/opni"}
[14:22:46] ESC[34mINFOESC[0m Generating certificates {"controller": "opensearchcluster", "controllerGroup": "opensearch.opster.io", "controllerKind": "OpenSearchCluster", "OpenSearchCluster": {"name":"opni","namespace":"opni"}, "namespace": "opni", "name": "opni", "reconcileID": "8008c978-612a-4e43-8253-5127895d99ac", "interface": "transport"}
[14:22:48] ESC[34mINFOESC[0m Generating certificates {"controller": "opensearchcluster", "controllerGroup": "opensearch.opster.io", "controllerKind": "OpenSearchCluster", "OpenSearchCluster": {"name":"opni","namespace":"opni"}, "namespace": "opni", "name": "opni", "reconcileID": "8008c978-612a-4e43-8253-5127895d99ac", "interface": "http"}
[14:22:48] ESC[34mINFOESC[0m Generating certificates {"controller": "opensearchcluster", "controllerGroup": "opensearch.opster.io", "controllerKind": "OpenSearchCluster", "OpenSearchCluster": {"name":"opni","namespace":"opni"}, "namespace": "opni", "name": "opni", "reconcileID": "8008c978-612a-4e43-8253-5127895d99ac"}
E0313 14:22:48.651324       7 event.go:340] Unsupported event type: 'Info'
[14:22:58] ESC[34mINFOESC[0m Reconciling OpenSearchCluster {"controller": "opensearchcluster", "controllerGroup": "opensearch.opster.io", "controllerKind": "OpenSearchCluster", "OpenSearchCluster": {"name":"opni","namespace":"opni"}, "namespace": "opni", "name": "opni", "reconcileID": "58468234-7641-4f6f-8a08-07f2d5cb00b7", "cluster": "opni/opni"}
[14:22:58] ESC[34mINFOESC[0m Generating certificates {"controller": "opensearchcluster", "controllerGroup": "opensearch.opster.io", "controllerKind": "OpenSearchCluster", "OpenSearchCluster": {"name":"opni","namespace":"opni"}, "namespace": "opni", "name": "opni", "reconcileID": "58468234-7641-4f6f-8a08-07f2d5cb00b7", "interface": "transport"}
[14:22:59] ESC[34mINFOESC[0m Generating certificates {"controller": "opensearchcluster", "controllerGroup": "opensearch.opster.io", "controllerKind": "OpenSearchCluster", "OpenSearchCluster": {"name":"opni","namespace":"opni"}, "namespace": "opni", "name": "opni", "reconcileID": "58468234-7641-4f6f-8a08-07f2d5cb00b7", "interface": "http"}
[14:23:00] ESC[34mINFOESC[0m Generating certificates {"controller": "opensearchcluster", "controllerGroup": "opensearch.opster.io", "controllerKind": "OpenSearchCluster", "OpenSearchCluster": {"name":"opni","namespace":"opni"}, "namespace": "opni", "name": "opni", "reconcileID": "58468234-7641-4f6f-8a08-07f2d5cb00b7"}
E0313 14:23:00.071465       7 event.go:340] Unsupported event type: 'Info'
[14:23:10] ESC[34mINFOESC[0m Reconciling OpenSearchCluster {"controller": "opensearchcluster", "controllerGroup": "opensearch.opster.io", "controllerKind": "OpenSearchCluster", "OpenSearchCluster": {"name":"opni","namespace":"opni"}, "namespace": "opni", "name": "opni", "reconcileID": "acb6fc60-a55c-482d-a706-8f1340c92c7c", "cluster": "opni/opni"}
[14:23:10] ESC[34mINFOESC[0m Generating certificates {"controller": "opensearchcluster", "controllerGroup": "opensearch.opster.io", "controllerKind": "OpenSearchCluster", "OpenSearchCluster": {"name":"opni","namespace":"opni"}, "namespace": "opni", "name": "opni", "reconcileID": "acb6fc60-a55c-482d-a706-8f1340c92c7c", "interface": "transport"}
[14:23:13] ESC[34mINFOESC[0m Generating certificates {"controller": "opensearchcluster", "controllerGroup": "opensearch.opster.io", "controllerKind": "OpenSearchCluster", "OpenSearchCluster": {"name":"opni","namespace":"opni"}, "namespace": "opni", "name": "opni", "reconcileID": "acb6fc60-a55c-482d-a706-8f1340c92c7c", "interface": "http"}
[14:23:13] ESC[34mINFOESC[0m Generating certificates {"controller": "opensearchcluster", "controllerGroup": "opensearch.opster.io", "controllerKind": "OpenSearchCluster", "OpenSearchCluster": {"name":"opni","namespace":"opni"}, "namespace": "opni", "name": "opni", "reconcileID": "acb6fc60-a55c-482d-a706-8f1340c92c7c"}
E0313 14:23:13.735948       7 event.go:340] Unsupported event type: 'Info'
[14:23:15] ESC[34mINFOESC[0m Reconciling OpenSearchCluster {"controller": "opensearchcluster", "controllerGroup": "opensearch.opster.io", "controllerKind": "OpenSearchCluster", "OpenSearchCluster": {"name":"opni","namespace":"opni"}, "namespace": "opni", "name": "opni", "reconcileID": "ff1f1a8a-17b5-4007-ae1a-1e42943ef1e0", "cluster": "opni/opni"}
[14:23:16] ESC[34mINFOESC[0m Generating certificates {"controller": "opensearchcluster", "controllerGroup": "opensearch.opster.io", "controllerKind": "OpenSearchCluster", "OpenSearchCluster": {"name":"opni","namespace":"opni"}, "namespace": "opni", "name": "opni", "reconcileID": "ff1f1a8a-17b5-4007-ae1a-1e42943ef1e0", "interface": "transport"}
[14:23:17] ESC[34mINFOESC[0m Generating certificates {"controller": "opensearchcluster", "controllerGroup": "opensearch.opster.io", "controllerKind": "OpenSearchCluster", "OpenSearchCluster": {"name":"opni","namespace":"opni"}, "namespace": "opni", "name": "opni", "reconcileID": "ff1f1a8a-17b5-4007-ae1a-1e42943ef1e0", "interface": "http"}
[14:23:17] ESC[34mINFOESC[0m Generating certificates {"controller": "opensearchcluster", "controllerGroup": "opensearch.opster.io", "controllerKind": "OpenSearchCluster", "OpenSearchCluster": {"name":"opni","namespace":"opni"}, "namespace": "opni", "name": "opni", "reconcileID": "ff1f1a8a-17b5-4007-ae1a-1e42943ef1e0"}
E0313 14:23:17.929820       7 event.go:340] Unsupported event type: 'Info'
[14:23:18] ESC[34mINFOESC[0m Reconciling OpenSearchCluster {"controller": "opensearchcluster", "controllerGroup": "opensearch.opster.io", "controllerKind": "OpenSearchCluster", "OpenSearchCluster": {"name":"opni","namespace":"opni"}, "namespace": "opni", "name": "opni", "reconcileID": "72ce0f00-47f5-496c-bd24-648b7ac23758", "cluster": "opni/opni"}
[14:23:18] ESC[34mINFOESC[0m Generating certificates {"controller": "opensearchcluster", "controllerGroup": "opensearch.opster.io", "controllerKind": "OpenSearchCluster", "OpenSearchCluster": {"name":"opni","namespace":"opni"}, "namespace": "opni", "name": "opni", "reconcileID": "72ce0f00-47f5-496c-bd24-648b7ac23758", "interface": "transport"}
[14:23:22] ESC[34mINFOESC[0m Generating certificates {"controller": "opensearchcluster", "controllerGroup": "opensearch.opster.io", "controllerKind": "OpenSearchCluster", "OpenSearchCluster": {"name":"opni","namespace":"opni"}, "namespace": "opni", "name": "opni", "reconcileID": "72ce0f00-47f5-496c-bd24-648b7ac23758", "interface": "http"}
[14:23:22] ESC[34mINFOESC[0m Generating certificates {"controller": "opensearchcluster", "controllerGroup": "opensearch.opster.io", "controllerKind": "OpenSearchCluster", "OpenSearchCluster": {"name":"opni","namespace":"opni"}, "namespace": "opni", "name": "opni", "reconcileID": "72ce0f00-47f5-496c-bd24-648b7ac23758"}
E0313 14:23:22.817938       7 event.go:340] Unsupported event type: 'Info'
[14:23:22] ESC[34mINFOESC[0m Reconciling OpenSearchCluster {"controller": "opensearchcluster", "controllerGroup": "opensearch.opster.io", "controllerKind": "OpenSearchCluster", "OpenSearchCluster": {"name":"opni","namespace":"opni"}, "namespace": "opni", "name": "opni", "reconcileID": "cc0c9266-1402-41e0-b168-c963e4f4c745", "cluster": "opni/opni"}
[14:23:22] ESC[34mINFOESC[0m Generating certificates {"controller": "opensearchcluster", "controllerGroup": "opensearch.opster.io", "controllerKind": "OpenSearchCluster", "OpenSearchCluster": {"name":"opni","namespace":"opni"}, "namespace": "opni", "name": "opni", "reconcileID": "cc0c9266-1402-41e0-b168-c963e4f4c745", "interface": "transport"}
[14:23:24] ESC[34mINFOESC[0m Generating certificates {"controller": "opensearchcluster", "controllerGroup": "opensearch.opster.io", "controllerKind": "OpenSearchCluster", "OpenSearchCluster": {"name":"opni","namespace":"opni"}, "namespace": "opni", "name": "opni", "reconcileID": "cc0c9266-1402-41e0-b168-c963e4f4c745", "interface": "http"}
[14:23:24] ESC[34mINFOESC[0m Generating certificates {"controller": "opensearchcluster", "controllerGroup": "opensearch.opster.io", "controllerKind": "OpenSearchCluster", "OpenSearchCluster": {"name":"opni","namespace":"opni"}, "namespace": "opni", "name": "opni", "reconcileID": "cc0c9266-1402-41e0-b168-c963e4f4c745"}
E0313 14:23:24.490338       7 event.go:340] Unsupported event type: 'Info'
[14:23:24] ESC[34mINFOESC[0m Reconciling OpenSearchCluster {"controller": "opensearchcluster", "controllerGroup": "opensearch.opster.io", "controllerKind": "OpenSearchCluster", "OpenSearchCluster": {"name":"opni","namespace":"opni"}, "namespace": "opni", "name": "opni", "reconcileID": "39f24bf8-5cf4-4f4c-abf2-7a184aaab256", "cluster": "opni/opni"}
[14:23:24] ESC[34mINFOESC[0m Generating certificates {"controller": "opensearchcluster", "controllerGroup": "opensearch.opster.io", "controllerKind": "OpenSearchCluster", "OpenSearchCluster": {"name":"opni","namespace":"opni"}, "namespace": "opni", "name": "opni", "reconcileID": "39f24bf8-5cf4-4f4c-abf2-7a184aaab256", "interface": "transport"}
[14:23:29] ESC[34mINFOESC[0m Generating certificates {"controller": "opensearchcluster", "controllerGroup": "opensearch.opster.io", "controllerKind": "OpenSearchCluster", "OpenSearchCluster": {"name":"opni","namespace":"opni"}, "namespace": "opni", "name": "opni", "reconcileID": "39f24bf8-5cf4-4f4c-abf2-7a184aaab256", "interface": "http"}
[14:23:29] ESC[34mINFOESC[0m Generating certificates {"controller": "opensearchcluster", "controllerGroup": "opensearch.opster.io", "controllerKind": "OpenSearchCluster", "OpenSearchCluster": {"name":"opni","namespace":"opni"}, "namespace": "opni", "name": "opni", "reconcileID": "39f24bf8-5cf4-4f4c-abf2-7a184aaab256"}
E0313 14:23:29.880075       7 event.go:340] Unsupported event type: 'Info'
[14:23:54] ESC[34mINFOESC[0m Reconciling OpenSearchCluster {"controller": "opensearchcluster", "controllerGroup": "opensearch.opster.io", "controllerKind": "OpenSearchCluster", "OpenSearchCluster": {"name":"opni","namespace":"opni"}, "namespace": "opni", "name": "opni", "reconcileID": "3e2ed369-06ce-43ee-b8e3-3461f17dc9dd", "cluster": "opni/opni"}
[14:23:54] ESC[34mINFOESC[0m Generating certificates {"controller": "opensearchcluster", "controllerGroup": "opensearch.opster.io", "controllerKind": "OpenSearchCluster", "OpenSearchCluster": {"name":"opni","namespace":"opni"}, "namespace": "opni", "name": "opni", "reconcileID": "3e2ed369-06ce-43ee-b8e3-3461f17dc9dd", "interface": "transport"}
[14:23:58] ESC[34mINFOESC[0m Generating certificates {"controller": "opensearchcluster", "controllerGroup": "opensearch.opster.io", "controllerKind": "OpenSearchCluster", "OpenSearchCluster": {"name":"opni","namespace":"opni"}, "namespace": "opni", "name": "opni", "reconcileID": "3e2ed369-06ce-43ee-b8e3-3461f17dc9dd", "interface": "http"}
[14:23:58] ESC[34mINFOESC[0m Generating certificates {"controller": "opensearchcluster", "controllerGroup": "opensearch.opster.io", "controllerKind": "OpenSearchCluster", "OpenSearchCluster": {"name":"opni","namespace":"opni"}, "namespace": "opni", "name": "opni", "reconcileID": "3e2ed369-06ce-43ee-b8e3-3461f17dc9dd"}
E0313 14:23:58.934357       7 event.go:340] Unsupported event type: 'Info'
[14:24:28] ESC[34mINFOESC[0m Reconciling OpenSearchCluster {"controller": "opensearchcluster", "controllerGroup": "opensearch.opster.io", "controllerKind": "OpenSearchCluster", "OpenSearchCluster": {"name":"opni","namespace":"opni"}, "namespace": "opni", "name": "opni", "reconcileID": "87dbc0cc-2bb3-4a6d-a4b6-32af5937a8d0", "cluster": "opni/opni"}
[14:24:29] ESC[34mINFOESC[0m Generating certificates {"controller": "opensearchcluster", "controllerGroup": "opensearch.opster.io", "controllerKind": "OpenSearchCluster", "OpenSearchCluster": {"name":"opni","namespace":"opni"}, "namespace": "opni", "name": "opni", "reconcileID": "87dbc0cc-2bb3-4a6d-a4b6-32af5937a8d0", "interface": "transport"}
[14:24:30] ESC[34mINFOESC[0m Generating certificates {"controller": "opensearchcluster", "controllerGroup": "opensearch.opster.io", "controllerKind": "OpenSearchCluster", "OpenSearchCluster": {"name":"opni","namespace":"opni"}, "namespace": "opni", "name": "opni", "reconcileID": "87dbc0cc-2bb3-4a6d-a4b6-32af5937a8d0", "interface": "http"}
[14:24:30] ESC[34mINFOESC[0m Generating certificates {"controller": "opensearchcluster", "controllerGroup": "opensearch.opster.io", "controllerKind": "OpenSearchCluster", "OpenSearchCluster": {"name":"opni","namespace":"opni"}, "namespace": "opni", "name": "opni", "reconcileID": "87dbc0cc-2bb3-4a6d-a4b6-32af5937a8d0"}
E0313 14:24:30.454950       7 event.go:340] Unsupported event type: 'Info'
[14:25:00] ESC[34mINFOESC[0m Reconciling OpenSearchCluster {"controller": "opensearchcluster", "controllerGroup": "opensearch.opster.io", "controllerKind": "OpenSearchCluster", "OpenSearchCluster": {"name":"opni","namespace":"opni"}, "namespace": "opni", "name": "opni", "reconcileID": "759bb583-2e8b-4635-b9f3-c92e38859db5", "cluster": "opni/opni"}
[14:25:00] ESC[34mINFOESC[0m Generating certificates {"controller": "opensearchcluster", "controllerGroup": "opensearch.opster.io", "controllerKind": "OpenSearchCluster", "OpenSearchCluster": {"name":"opni","namespace":"opni"}, "namespace": "opni", "name": "opni", "reconcileID": "759bb583-2e8b-4635-b9f3-c92e38859db5", "interface": "transport"}
[14:25:03] ESC[34mINFOESC[0m Generating certificates {"controller": "opensearchcluster", "controllerGroup": "opensearch.opster.io", "controllerKind": "OpenSearchCluster", "OpenSearchCluster": {"name":"opni","namespace":"opni"}, "namespace": "opni", "name": "opni", "reconcileID": "759bb583-2e8b-4635-b9f3-c92e38859db5", "interface": "http"}
[14:25:03] ESC[34mINFOESC[0m Generating certificates {"controller": "opensearchcluster", "controllerGroup": "opensearch.opster.io", "controllerKind": "OpenSearchCluster", "OpenSearchCluster": {"name":"opni","namespace":"opni"}, "namespace": "opni", "name": "opni", "reconcileID": "759bb583-2e8b-4635-b9f3-c92e38859db5"}
E0313 14:25:03.676007       7 event.go:340] Unsupported event type: 'Info'
[14:25:33] ESC[34mINFOESC[0m Reconciling OpenSearchCluster {"controller": "opensearchcluster", "controllerGroup": "opensearch.opster.io", "controllerKind": "OpenSearchCluster", "OpenSearchCluster": {"name":"opni","namespace":"opni"}, "namespace": "opni", "name": "opni", "reconcileID": "f9b22fd7-89dd-4c91-9e2c-db2f4be611e3", "cluster": "opni/opni"}
[14:25:33] ESC[34mINFOESC[0m Generating certificates {"controller": "opensearchcluster", "controllerGroup": "opensearch.opster.io", "controllerKind": "OpenSearchCluster", "OpenSearchCluster": {"name":"opni","namespace":"opni"}, "namespace": "opni", "name": "opni", "reconcileID": "f9b22fd7-89dd-4c91-9e2c-db2f4be611e3", "interface": "transport"}
[14:25:37] ESC[31mERRORESC[0m Reconciler error {"controller": "loggingcluster", "controllerGroup": "core.opni.io", "controllerKind": "LoggingCluster", "LoggingCluster": {"name":"logging-bmqsd","namespace":"opni"}, "namespace": "opni", "name": "logging-bmqsd", "reconcileID": "97b6515f-ada5-4197-bcd0-64450a61a3b0", "error": "remote error: tls: unknown certificate"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
        sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
        sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
[14:25:37] ESC[31mERRORESC[0m Reconciler error {"controller": "loggingcluster", "controllerGroup": "core.opni.io", "controllerKind": "LoggingCluster", "LoggingCluster": {"name":"logging-bmqsd","namespace":"opni"}, "namespace": "opni", "name": "logging-bmqsd", "reconcileID": "f53ac244-d2f9-4e87-9fc5-eb86ffebe3a8", "error": "remote error: tls: unknown certificate"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
        sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
        sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
[14:25:37] ESC[31mERRORESC[0m Reconciler error {"controller": "loggingcluster", "controllerGroup": "core.opni.io", "controllerKind": "LoggingCluster", "LoggingCluster": {"name":"logging-bmqsd","namespace":"opni"}, "namespace": "opni", "name": "logging-bmqsd", "reconcileID": "8401a259-b385-46fb-a1f8-aa7c14e02b64", "error": "remote error: tls: unknown certificate"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
        sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
        sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
dbason commented 1 year ago

And cert manager is installed and working correctly?

joshmeranda commented 1 year ago

Looks like no. Looks like there is something 

# cert-manager
I0313 13:20:19.037631       1 trigger_controller.go:200] cert-manager/certificates-trigger "msg"="Certificate must be re-issued" "key"="opni/opensearch-opni-admin" "message"="Issuing certificate as Secret does not exist" "reason"="DoesNotExist"
I0313 13:20:19.038654       1 conditions.go:203] Setting lastTransitionTime for Certificate "opensearch-opni-admin" condition "Issuing" to 2023-03-13 13:20:19.037919336 +0000 UTC m=+192.955571104
I0313 13:20:19.039990       1 conditions.go:192] Found status change for Certificate "opensearch-opni-admin" condition "Ready": "True" -> "False"; setting lastTransitionTime to 2023-03-13 13:20:19.039980908 +0000 UTC m=+192.957632676
I0313 13:20:19.082785       1 controller.go:162] cert-manager/certificates-trigger "msg"="re-queuing item due to optimistic locking on resource" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"opensearch-opni-admin\": the object has been modified; please apply your changes to the latest version and try again" "key"="opni/opensearch-opni-admin"
I0313 13:20:19.083038       1 trigger_controller.go:200] cert-manager/certificates-trigger "msg"="Certificate must be re-issued" "key"="opni/opensearch-opni-admin" "message"="Issuing certificate as Secret does not exist" "reason"="DoesNotExist"
I0313 13:20:19.083064       1 conditions.go:203] Setting lastTransitionTime for Certificate "opensearch-opni-admin" condition "Issuing" to 2023-03-13 13:20:19.083057947 +0000 UTC m=+193.000709716
I0313 13:20:19.128197       1 controller.go:162] cert-manager/certificates-key-manager "msg"="re-queuing item due to optimistic locking on resource" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"opensearch-opni-admin\": the object has been modified; please apply your changes to the latest version and try again" "key"="opni/opensearch-opni-admin"
I0313 13:20:19.135430       1 conditions.go:263] Setting lastTransitionTime for CertificateRequest "opensearch-opni-admin-lbf4s" condition "Approved" to 2023-03-13 13:20:19.135419994 +0000 UTC m=+193.053071759
I0313 13:20:19.166693       1 conditions.go:263] Setting lastTransitionTime for CertificateRequest "opensearch-opni-admin-lbf4s" condition "Ready" to 2023-03-13 13:20:19.166683035 +0000 UTC m=+193.084334785
I0313 13:20:19.206949       1 conditions.go:192] Found status change for Certificate "opensearch-opni-admin" condition "Ready": "False" -> "True"; setting lastTransitionTime to 2023-03-13 13:20:19.206938637 +0000 UTC m=+193.124590406
I0313 13:20:19.221446       1 controller.go:162] cert-manager/certificates-readiness "msg"="re-queuing item due to optimistic locking on resource" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"opensearch-opni-admin\": the object has been modified; please apply your changes to the latest version and try again" "key"="opni/opensearch-opni-admin"
I0313 13:20:19.221999       1 conditions.go:192] Found status change for Certificate "opensearch-opni-admin" condition "Ready": "False" -> "True"; setting lastTransitionTime to 2023-03-13 13:20:19.221989186 +0000 UTC m=+193.139640955
I0313 13:20:19.236348       1 conditions.go:192] Found status change for Certificate "opensearch-opni-ca" condition "Ready": "True" -> "False"; setting lastTransitionTime to 2023-03-13 13:20:19.236338972 +0000 UTC m=+193.153990740
I0313 13:20:19.242453       1 trigger_controller.go:200] cert-manager/certificates-trigger "msg"="Certificate must be re-issued" "key"="opni/opensearch-opni-ca" "message"="Issuing certificate as Secret does not exist" "reason"="DoesNotExist"
I0313 13:20:19.244362       1 conditions.go:203] Setting lastTransitionTime for Certificate "opensearch-opni-ca" condition "Issuing" to 2023-03-13 13:20:19.244354085 +0000 UTC m=+193.162005850
# cert-manager-webhook
W0314 18:26:23.006976   20457 transport.go:243] Unable to cancel request for *exec.roundTripper
I0314 18:26:23.007252   20457 versioner.go:56] Remote kubernetes server unreachable
I0313 13:16:52.320816       1 feature_gate.go:245] feature gates: &{map[]}
W0313 13:16:52.320904       1 client_config.go:617] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0313 13:16:52.328234       1 webhook.go:129] cert-manager "msg"="using dynamic certificate generating using CA stored in Secret resource" "secret_name"="cert-manager-webhook-ca" "secret_namespace"="cert-manager"
I0313 13:16:52.328411       1 server.go:133] cert-manager/webhook "msg"="listening for insecure healthz connections" "address"=":6080"
I0313 13:16:52.328447       1 server.go:197] cert-manager/webhook "msg"="listening for secure connections" "address"=":10250"
I0313 13:16:53.336634       1 dynamic_source.go:266] cert-manager/webhook "msg"="Updated cert-manager webhook TLS certificate" "DNSNames"=["cert-manager-webhook","cert-manager-webhook.cert-manager","cert-manager-webhook.cert-manager.svc"]
I0313 13:20:19.069304       1 logs.go:59] http: TLS handshake error from 192.168.176.29:37714: EOF
I0313 13:20:19.254485       1 logs.go:59] http: TLS handshake error from 192.168.176.29:37726: EOF
I0313 13:20:19.257206       1 logs.go:59] http: TLS handshake error from 192.168.176.29:37740: EOF
I0313 13:21:41.058861       1 logs.go:59] http: TLS handshake error from 192.168.176.29:37932: EOF
I0313 13:21:41.060050       1 logs.go:59] http: TLS handshake error from 192.168.176.29:37936: EOF
I0313 13:21:41.063343       1 logs.go:59] http: TLS handshake error from 192.168.176.29:37920: EOF
I0313 13:21:41.064895       1 logs.go:59] http: TLS handshake error from 192.168.176.29:37950: EOF
I0313 13:21:41.065911       1 logs.go:59] http: TLS handshake error from 192.168.176.29:37940: EOF
I0313 13:21:41.066545       1 logs.go:59] http: TLS handshake error from 192.168.176.29:37958: EOF
I0313 13:21:41.130727       1 logs.go:59] http: TLS handshake error from 192.168.176.29:37968: EOF
I0313 13:21:41.134802       1 logs.go:59] http: TLS handshake error from 192.168.176.29:37970: EOF
I0313 13:22:08.033160       1 logs.go:59] http: TLS handshake error from 192.168.176.29:45994: EOF
I0313 13:22:08.035939       1 logs.go:59] http: TLS handshake error from 192.168.176.29:46006: EOF
I0313 13:22:08.036588       1 logs.go:59] http: TLS handshake error from 192.168.176.29:46008: EOF
I0313 13:22:08.038885       1 logs.go:59] http: TLS handshake error from 192.168.176.29:46016: EOF
I0313 13:22:08.061528       1 logs.go:59] http: TLS handshake error from 192.168.176.29:46028: EOF
I0313 13:22:08.068611       1 logs.go:59] http: TLS handshake error from 192.168.176.29:46052: EOF
I0313 13:22:08.070427       1 logs.go:59] http: TLS handshake error from 192.168.176.29:46082: EOF
I0313 13:22:08.077975       1 logs.go:59] http: TLS handshake error from 192.168.176.29:46092: EOF
I0313 13:22:08.078553       1 logs.go:59] http: TLS handshake error from 192.168.176.29:46112: EOF
I0314 09:37:07.603194       1 logs.go:59] http: TLS handshake error from 192.168.193.145:57770: EOF
I0314 09:37:07.609707       1 logs.go:59] http: TLS handshake error from 192.168.193.145:57778: EOF
dbason commented 1 year ago

It seems that there was a mismatched certificate secret persisting. By clearing out all of the cert manager objects and secrets and redeploying the logging cluster this was fixed.

alexandreLamarre commented 1 year ago

I'd like to reopen this, since its possible to hit this problem on a fresh install in a new cluster and does not make for a fun experience

dbason commented 1 year ago

I've never been able to reproduce this issue. If this occurs again can we please collect logs from the manager and gateway, along with cert-manager to try and diagnose what is going on. Also please let me know what version of cert manager is being used

alexandreLamarre commented 1 year ago

No relevant error messages from the manager pod or certmanager

There were some 403s and 401s from the gateway