search

apecloud / kubeblocks

KubeBlocks is an open-source control plane software that runs and manages databases, message queues and other stateful applications on K8s.
https://kubeblocks.io
GNU Affero General Public License v3.0
1.77k stars 156 forks source link

[BUG] mongodb-4.4 cluster restore volume-snapshot or datafile backup failed #6487

Open JashBook opened 5 months ago

JashBook commented 5 months ago

Describe the bug A clear and concise description of what the bug is.

kbcli version
Kubernetes: v1.26.6-gke.1700
KubeBlocks: 0.8.1-beta.4
kbcli: 0.8.0

To Reproduce Steps to reproduce the behavior:

  1. create cluster with mongodb-4.4 
    kbcli cluster create  mongo-cluster --termination-policy=WipeOut --cluster-definition=mongodb --cluster-version=mongodb-4.4 --set cpu=100m,memory=0.5Gi,replicas=1,storage=3Gi
  2. hscale cluster 
    kbcli cluster hscale mongo-cluster --replicas 3 --components mongodb --auto-approve
  3. do volume-snapshot or datafile backup 
    kbcli cluster backup  mongo-cluster --method volume-snapshot
kbcli cluster backup  mongo-cluster --method datafile
  4. restore cluster 
    
kubectl get backup
NAME                                            POLICY                                  METHOD            REPO                    STATUS      TOTAL-SIZE   DURATION   CREATION-TIME          COMPLETION-TIME        EXPIRATION-TIME
backup-default-mongo-cluster-20240119123625     mongo-cluster-mongodb-backup-policy     volume-snapshot                           Completed   3Gi          25s        2024-01-19T04:36:29Z   2024-01-19T04:36:54Z

kbcli cluster restore mongo-cluster-bk1 --backup backup-default-mongo-cluster-20240119123625

5. See error

kubectl get pod -l app.kubernetes.io/instance=mongo-cluster-bk1 NAME READY STATUS RESTARTS AGE mongo-cluster-bk1-mongodb-0 3/3 Running 1 (3m41s ago) 5m36s mongo-cluster-bk1-mongodb-1 3/3 Running 0 5m36s mongo-cluster-bk1-mongodb-2 3/3 Running 0 5m36s ➜ ~ ➜ ~ kubectl get cluster mongo-cluster-bk1 NAME CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS AGE mongo-cluster-bk1 mongodb mongodb-4.4 WipeOut Creating 7m32s

logs pod

kubectl logs mongo-cluster-bk1-mongodb-0 mongodb --previous Error: couldn't connect to server 127.0.0.1:27027, connection attempt failed: SocketException: Error connecting to 127.0.0.1:27027 :: caused by :: Connection refused : connect@src/mongo/shell/mongo.js:374:17 @(connect):3:6 exception: connect failed exiting with code 1 Error: couldn't connect to server 127.0.0.1:27027, connection attempt failed: SocketException: Error connecting to 127.0.0.1:27027 :: caused by :: Connection refused : connect@src/mongo/shell/mongo.js:374:17 @(connect):3:6 exception: connect failed exiting with code 1 Error: couldn't connect to server 127.0.0.1:27027, connection attempt failed: SocketException: Error connecting to 127.0.0.1:27027 :: caused by :: Connection refused : connect@src/mongo/shell/mongo.js:374:17 @(connect):3:6 exception: connect failed exiting with code 1 Error: couldn't connect to server 127.0.0.1:27027, connection attempt failed: SocketException: Error connecting to 127.0.0.1:27027 :: caused by :: Connection refused : connect@src/mongo/shell/mongo.js:374:17 @(connect):3:6 exception: connect failed exiting with code 1 Error: couldn't connect to server 127.0.0.1:27027, connection attempt failed: SocketException: Error connecting to 127.0.0.1:27027 :: caused by :: Connection refused : connect@src/mongo/shell/mongo.js:374:17 @(connect):3:6 exception: connect failed exiting with code 1 Error: couldn't connect to server 127.0.0.1:27027, connection attempt failed: SocketException: Error connecting to 127.0.0.1:27027 :: caused by :: Connection refused : connect@src/mongo/shell/mongo.js:374:17 @(connect):3:6 exception: connect failed exiting with code 1 restore process is ready { "acknowledged" : true, "deletedCount" : 1 } true INFO: restore set-up configuration successfully.

kubectl logs mongo-cluster-bk1-mongodb-0 kb-checkrole 2024-01-19T04:48:07Z INFO HA Create Root. 2024-01-19T04:48:07Z INFO MongoDB Create user: root, passwd: pn7979km, roles: map[db:admin role:root] 2024-01-19T04:48:07Z INFO MongoDB Create Root failed {"error": "failed to create user: (NotWritablePrimary) not primary", "errorVerbose": "(NotWritablePrimary) not primary\nfailed to create user\ngithub.com/apecloud/kubeblocks/pkg/lorry/engines/mongodb.CreateUser\n\t/src/pkg/lorry/engines/mongodb/users.go:39\ngithub.com/apecloud/kubeblocks/pkg/lorry/engines/mongodb.(Manager).CreateRoot\n\t/src/pkg/lorry/engines/mongodb/manager.go:246\ngithub.com/apecloud/kubeblocks/pkg/lorry/highavailability.(Ha).Start\n\t/src/pkg/lorry/highavailability/ha.go:258\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650"} 2024-01-19T04:48:07Z ERROR HA Cluster initialize failed {"error": "failed to create user: (NotWritablePrimary) not primary", "errorVerbose": "(NotWritablePrimary) not primary\nfailed to create user\ngithub.com/apecloud/kubeblocks/pkg/lorry/engines/mongodb.CreateUser\n\t/src/pkg/lorry/engines/mongodb/users.go:39\ngithub.com/apecloud/kubeblocks/pkg/lorry/engines/mongodb.(Manager).CreateRoot\n\t/src/pkg/lorry/engines/mongodb/manager.go:246\ngithub.com/apecloud/kubeblocks/pkg/lorry/highavailability.(Ha).Start\n\t/src/pkg/lorry/highavailability/ha.go:258\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650"} github.com/apecloud/kubeblocks/pkg/lorry/highavailability.(Ha).Start /src/pkg/lorry/highavailability/ha.go:260 2024-01-19T04:48:08Z INFO MongoDB DB is not ready {"error": "connection() error occurred during connection handshake: auth error: sasl conversation error: unable to authenticate using mechanism \"SCRAM-SHA-1\": (AuthenticationFailed) Authentication failed."} 2024-01-19T04:48:10Z INFO MongoDB DB is not ready {"error": "connection() error occurred during connection handshake: auth error: sasl conversation error: unable to authenticate using mechanism \"SCRAM-SHA-1\": (AuthenticationFailed) Authentication failed."} 2024-01-19T04:48:12Z INFO MongoDB DB is not ready {"error": "connection() error occurred during connection handshake: auth error: sasl conversation error: unable to authenticate using mechanism \"SCRAM-SHA-1\": (AuthenticationFailed) Authentication failed."} 2024-01-19T04:48:12Z INFO HA Create Root. 2024-01-19T04:48:12Z INFO MongoDB Create user: root, passwd: pn7979km, roles: map[db:admin role:root] 2024-01-19T04:48:12Z INFO MongoDB Create Root failed {"error": "failed to create user: (NotWritablePrimary) not primary", "errorVerbose": "(NotWritablePrimary) not primary\nfailed to create user\ngithub.com/apecloud/kubeblocks/pkg/lorry/engines/mongodb.CreateUser\n\t/src/pkg/lorry/engines/mongodb/users.go:39\ngithub.com/apecloud/kubeblocks/pkg/lorry/engines/mongodb.(Manager).CreateRoot\n\t/src/pkg/lorry/engines/mongodb/manager.go:246\ngithub.com/apecloud/kubeblocks/pkg/lorry/highavailability.(Ha).Start\n\t/src/pkg/lorry/highavailability/ha.go:258\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650"} 2024-01-19T04:48:12Z ERROR HA Cluster initialize failed {"error": "failed to create user: (NotWritablePrimary) not primary", "errorVerbose": "(NotWritablePrimary) not primary\nfailed to create user\ngithub.com/apecloud/kubeblocks/pkg/lorry/engines/mongodb.CreateUser\n\t/src/pkg/lorry/engines/mongodb/users.go:39\ngithub.com/apecloud/kubeblocks/pkg/lorry/engines/mongodb.(Manager).CreateRoot\n\t/src/pkg/lorry/engines/mongodb/manager.go:246\ngithub.com/apecloud/kubeblocks/pkg/lorry/highavailability.(Ha).Start\n\t/src/pkg/lorry/highavailability/ha.go:258\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650"} github.com/apecloud/kubeblocks/pkg/lorry/highavailability.(Ha).Start /src/pkg/lorry/highavailability/ha.go:260 2024-01-19T04:48:14Z INFO MongoDB DB is not ready {"error": "connection() error occurred during connection handshake: auth error: sasl conversation error: unable to authenticate using mechanism \"SCRAM-SHA-1\": (AuthenticationFailed) Authentication failed."} 2024-01-19T04:48:16Z INFO MongoDB DB is not ready {"error": "connection() error occurred during connection handshake: auth error: sasl conversation error: unable to authenticate using mechanism \"SCRAM-SHA-1\": (AuthenticationFailed) Authentication failed."} 2024-01-19T04:48:17Z INFO HA Create Root. 2024-01-19T04:48:17Z INFO MongoDB Create user: root, passwd: pn7979km, roles: map[db:admin role:root] 2024-01-19T04:48:17Z INFO MongoDB Create Root failed {"error": "failed to create user: (NotWritablePrimary) not primary", "errorVerbose": "(NotWritablePrimary) not primary\nfailed to create user\ngithub.com/apecloud/kubeblocks/pkg/lorry/engines/mongodb.CreateUser\n\t/src/pkg/lorry/engines/mongodb/users.go:39\ngithub.com/apecloud/kubeblocks/pkg/lorry/engines/mongodb.(Manager).CreateRoot\n\t/src/pkg/lorry/engines/mongodb/manager.go:246\ngithub.com/apecloud/kubeblocks/pkg/lorry/highavailability.(Ha).Start\n\t/src/pkg/lorry/highavailability/ha.go:258\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650"} 2024-01-19T04:48:17Z ERROR HA Cluster initialize failed {"error": "failed to create user: (NotWritablePrimary) not primary", "errorVerbose": "(NotWritablePrimary) not primary\nfailed to create user\ngithub.com/apecloud/kubeblocks/pkg/lorry/engines/mongodb.CreateUser\n\t/src/pkg/lorry/engines/mongodb/users.go:39\ngithub.com/apecloud/kubeblocks/pkg/lorry/engines/mongodb.(Manager).CreateRoot\n\t/src/pkg/lorry/engines/mongodb/manager.go:246\ngithub.com/apecloud/kubeblocks/pkg/lorry/highavailability.(Ha).Start\n\t/src/pkg/lorry/highavailability/ha.go:258\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650"} github.com/apecloud/kubeblocks/pkg/lorry/highavailability.(*Ha).Start /src/pkg/lorry/highavailability/ha.go:260 2024-01-19T04:48:18Z INFO MongoDB DB is not ready {"error": "connection() error occurred during connection handshake: auth error: sasl conversation error: unable to authenticate using mechanism \"SCRAM-SHA-1\": (AuthenticationFailed) Authentication failed."} 2024-01-19T04:48:20Z INFO MongoDB DB is not ready {"error": "connection() error occurred during connection handshake: auth error: sasl conversation error: unable to authenticate using mechanism \"SCRAM-SHA-1\": (AuthenticationFailed) Authentication failed."}



**Expected behavior**
mongodb-4.4 cluster restore volume-snapshot or datafile backup success .

**Screenshots**
If applicable, add screenshots to help explain your problem.

**Desktop (please complete the following information):**
 - OS: [e.g. iOS]
 - Browser [e.g. chrome, safari]
 - Version [e.g. 22]

**Additional context**
Add any other context about the problem here.
github-actions[bot] commented 4 months ago

This issue has been marked as stale because it has been open for 30 days with no activity