Open silentacorn opened 1 month ago
Pod turned to running status only indicates that the observer process is up and listens the desired part, but there's still extra steps to successfully bootstrap an OceanBase cluster. The events show that ob-operator failed to obtain a connection after tried all the observers, and the time was about 1 minute after obzone was created, maybe observer is not ready to handle connection cause it still has to do some initializing procedure after the process is started. You may try to connect to the observer using mysql client using root user and without password, see if the observer is connectable, and get the first log file of observer, the file may be observer.log.20.... We've encountered some similar issues due to slow IO, you may also check the disk iops as well.
Describe the bug
Try deploy a simple oceanbase cluster by ob-operator but always failed as follow error logs while pods shows running already:
{"level":"INFO","ts":"2024-08-01T19:41:02+08:00","msg":"Create observers","controller":"obzone","controllerGroup":"oceanbase.oceanbase.com","controllerKind":"OBZone","OBZone":{"name":"obcluster-1-zone1","namespace":"oceanbase"},"namespace":"oceanbase","name":"obcluster-1-zone1","reconcileID":"fe450abf-de5d-440b-b5ed-e7b6fb85d396"} {"level":"INFO","ts":"2024-08-01T19:41:02+08:00","msg":"Create observer","controller":"obzone","controllerGroup":"oceanbase.oceanbase.com","controllerKind":"OBZone","OBZone":{"name":"obcluster-1-zone1","namespace":"oceanbase"},"namespace":"oceanbase","name":"obcluster-1-zone1","reconcileID":"fe450abf-de5d-440b-b5ed-e7b6fb85d396","server":"obcluster-1-zone1-272987e9035e"} {"level":"INFO","ts":"2024-08-01T19:41:02+08:00","msg":"Newly created server, init status","controller":"observer","controllerGroup":"oceanbase.oceanbase.com","controllerKind":"OBServer","OBServer":{"name":"obcluster-1-zone1-272987e9035e","namespace":"oceanbase"},"namespace":"oceanbase","name":"obcluster-1-zone1-272987e9035e","reconcileID":"cab46ab5-f960-4474-be95-2a0dadbdf53a"} {"level":"INFO","ts":"2024-08-01T19:41:02+08:00","msg":"Create observers","controller":"obzone","controllerGroup":"oceanbase.oceanbase.com","controllerKind":"OBZone","OBZone":{"name":"obcluster-1-zone2","namespace":"oceanbase"},"namespace":"oceanbase","name":"obcluster-1-zone2","reconcileID":"17e81701-0ccd-43ba-b3b2-bf99b6ed0623"} {"level":"INFO","ts":"2024-08-01T19:41:02+08:00","msg":"Create observer","controller":"obzone","controllerGroup":"oceanbase.oceanbase.com","controllerKind":"OBZone","OBZone":{"name":"obcluster-1-zone2","namespace":"oceanbase"},"namespace":"oceanbase","name":"obcluster-1-zone2","reconcileID":"17e81701-0ccd-43ba-b3b2-bf99b6ed0623","server":"obcluster-1-zone2-c557ffed90ab"} {"level":"INFO","ts":"2024-08-01T19:41:02+08:00","msg":"Newly created server, init status","controller":"observer","controllerGroup":"oceanbase.oceanbase.com","controllerKind":"OBServer","OBServer":{"name":"obcluster-1-zone2-c557ffed90ab","namespace":"oceanbase"},"namespace":"oceanbase","name":"obcluster-1-zone2-c557ffed90ab","reconcileID":"ec6f9e99-3765-4ac8-897e-f0a00a0a6a9f"} {"level":"INFO","ts":"2024-08-01T19:41:02+08:00","msg":"Create observers","controller":"obzone","controllerGroup":"oceanbase.oceanbase.com","controllerKind":"OBZone","OBZone":{"name":"obcluster-1-zone3","namespace":"oceanbase"},"namespace":"oceanbase","name":"obcluster-1-zone3","reconcileID":"09711cad-b115-48a5-b497-846cb93840aa"} {"level":"INFO","ts":"2024-08-01T19:41:02+08:00","msg":"Create observer","controller":"obzone","controllerGroup":"oceanbase.oceanbase.com","controllerKind":"OBZone","OBZone":{"name":"obcluster-1-zone3","namespace":"oceanbase"},"namespace":"oceanbase","name":"obcluster-1-zone3","reconcileID":"09711cad-b115-48a5-b497-846cb93840aa","server":"obcluster-1-zone3-b92a1921f4dd"} {"level":"INFO","ts":"2024-08-01T19:41:02+08:00","msg":"Newly created server, init status","controller":"observer","controllerGroup":"oceanbase.oceanbase.com","controllerKind":"OBServer","OBServer":{"name":"obcluster-1-zone3-b92a1921f4dd","namespace":"oceanbase"},"namespace":"oceanbase","name":"obcluster-1-zone3-b92a1921f4dd","reconcileID":"e628e425-3d9d-4afd-a21f-83ca582afe83"} {"level":"INFO","ts":"2024-08-01T19:41:03+08:00","msg":"Create observer when create obcluster","controller":"observer","controllerGroup":"oceanbase.oceanbase.com","controllerKind":"OBServer","OBServer":{"name":"obcluster-1-zone1-272987e9035e","namespace":"oceanbase"},"namespace":"oceanbase","name":"obcluster-1-zone1-272987e9035e","reconcileID":"0eac2fb7-ec9f-4400-a6c3-3433eec52d7d"} {"level":"INFO","ts":"2024-08-01T19:41:03+08:00","msg":"Create observer when create obcluster","controller":"observer","controllerGroup":"oceanbase.oceanbase.com","controllerKind":"OBServer","OBServer":{"name":"obcluster-1-zone2-c557ffed90ab","namespace":"oceanbase"},"namespace":"oceanbase","name":"obcluster-1-zone2-c557ffed90ab","reconcileID":"a3366ec0-475f-422f-b976-60f4e6d67eb9"} {"level":"INFO","ts":"2024-08-01T19:41:03+08:00","msg":"Create observer when create obcluster","controller":"observer","controllerGroup":"oceanbase.oceanbase.com","controllerKind":"OBServer","OBServer":{"name":"obcluster-1-zone3-b92a1921f4dd","namespace":"oceanbase"},"namespace":"oceanbase","name":"obcluster-1-zone3-b92a1921f4dd","reconcileID":"34dbd673-0d8c-4272-9a51-77c6e1930ae3"} {"level":"INFO","ts":"2024-08-01T19:41:10+08:00","msg":"static ip not supported, set empty annotation","controller":"observer","controllerGroup":"oceanbase.oceanbase.com","controllerKind":"OBServer","OBServer":{"name":"obcluster-1-zone1-272987e9035e","namespace":"oceanbase"},"namespace":"oceanbase","name":"obcluster-1-zone1-272987e9035e","reconcileID":"095a8d75-4893-493d-ada8-f6d30f5976b4"} {"level":"INFO","ts":"2024-08-01T19:41:10+08:00","msg":"static ip not supported, set empty annotation","controller":"observer","controllerGroup":"oceanbase.oceanbase.com","controllerKind":"OBServer","OBServer":{"name":"obcluster-1-zone2-c557ffed90ab","namespace":"oceanbase"},"namespace":"oceanbase","name":"obcluster-1-zone2-c557ffed90ab","reconcileID":"03fccac2-3b74-457b-b2ca-46109752f957"} {"level":"INFO","ts":"2024-08-01T19:41:10+08:00","msg":"static ip not supported, set empty annotation","controller":"observer","controllerGroup":"oceanbase.oceanbase.com","controllerKind":"OBServer","OBServer":{"name":"obcluster-1-zone3-b92a1921f4dd","namespace":"oceanbase"},"namespace":"oceanbase","name":"obcluster-1-zone3-b92a1921f4dd","reconcileID":"ad92b607-f1ed-412e-9e69-dda2adf0e4c0"} {"level":"INFO","ts":"2024-08-01T19:41:23+08:00","msg":"Pod is ready","controller":"observer","controllerGroup":"oceanbase.oceanbase.com","controllerKind":"OBServer","OBServer":{"name":"obcluster-1-zone1-272987e9035e","namespace":"oceanbase"},"namespace":"oceanbase","name":"obcluster-1-zone1-272987e9035e","reconcileID":"0033dca3-0681-44ac-b720-9a3b0bcc2843"} {"level":"INFO","ts":"2024-08-01T19:41:25+08:00","msg":"Pod is ready","controller":"observer","controllerGroup":"oceanbase.oceanbase.com","controllerKind":"OBServer","OBServer":{"name":"obcluster-1-zone2-c557ffed90ab","namespace":"oceanbase"},"namespace":"oceanbase","name":"obcluster-1-zone2-c557ffed90ab","reconcileID":"9dd82dbe-0c90-4a08-ba8d-808ea5bb7133"} {"level":"INFO","ts":"2024-08-01T19:41:33+08:00","msg":"Pod is ready","controller":"observer","controllerGroup":"oceanbase.oceanbase.com","controllerKind":"OBServer","OBServer":{"name":"obcluster-1-zone3-b92a1921f4dd","namespace":"oceanbase"},"namespace":"oceanbase","name":"obcluster-1-zone3-b92a1921f4dd","reconcileID":"c5b30b83-a796-4e88-9ed9-6ede5e9e1252"} {"level":"ERROR","ts":"2024-08-01T19:42:12+08:00","msg":"get oceanbase operation manager failed","controller":"obcluster","controllerGroup":"oceanbase.oceanbase.com","controllerKind":"OBCluster","OBCluster":{"name":"obcluster","namespace":"oceanbase"},"namespace":"oceanbase","name":"obcluster","reconcileID":"44f8cc7e-6701-4fa7-8f20-788b32676d0f","error":"Can not get oceanbase operation manager of obcluster obcluster after checked all server","errorVerbose":"Can not get oceanbase operation manager of obcluster obcluster after checked all server\ngithub.com/oceanbase/ob-operator/internal/resource/utils.getSysClient\n\t/workspace/internal/resource/utils/util.go:138\ngithub.com/oceanbase/ob-operator/internal/resource/utils.GetSysOperationClient\n\t/workspace/internal/resource/utils/util.go:53\ngithub.com/oceanbase/ob-operator/internal/resource/obcluster.(OBClusterManager).getOceanbaseOperationManager\n\t/workspace/internal/resource/obcluster/obcluster_task.go:266\ngithub.com/oceanbase/ob-operator/internal/resource/obcluster.(OBClusterManager).Bootstrap\n\t/workspace/internal/resource/obcluster/obcluster_task.go:281\ngithub.com/oceanbase/ob-operator/pkg/task.runTask\n\t/workspace/pkg/task/task_manager.go:63\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1598","stacktrace":"github.com/oceanbase/ob-operator/internal/resource/obcluster.(*OBClusterManager).Bootstrap\n\t/workspace/internal/resource/obcluster/obcluster_task.go:291\ngithub.com/oceanbase/ob-operator/pkg/task.runTask\n\t/workspace/pkg/task/task_manager.go:63"}
The obcluster.yaml file: ` apiVersion: oceanbase.oceanbase.com/v1alpha1 kind: OBCluster metadata: name: obcluster namespace: oceanbase spec: clusterName: obcluster clusterId: 1 serviceAccount: "default" userSecrets: root: root-password proxyro: proxyro-password monitor: monitor-password operator: operator-password topology:
Environment
ob-operator:2.2.2 observer: image: oceanbase/oceanbase-cloud-native:4.2.1.6-106000012024042515 monitor: image: oceanbase/obagent:4.2.1-100000092023101717 k8s: v1.28.2
Fast reproduce steps
Expected behavior
No response
Actual behavior
kubectl -n oceanbase get obcluster NAME STATUS AGE obcluster failed 14m
kubectl -n oceanbase describe obcluster obcluster ... Events: Type Reason Age From Message
Normal 13m obcluster-controller newly created cluster, init status Normal 13m obcluster-controller Create obzone obcluster-1-zone1 successfully Normal 13m obcluster-controller Create obzone obcluster-1-zone2 successfully Normal 13m obcluster-controller Create obzone obcluster-1-zone3 successfully Warning Task failed 12m obcluster-controller get oceanbase operation manager: Can not get oceanbase operation manager of obcluster obcluster after checked all server
Additional context
No response