search

codership / galera-manager-support

Galera Manager Support Repository
7 stars 2 forks source link

All nodes go into Donor/Desync after joining new node #89

Open dbucher-datasport opened 5 months ago

dbucher-datasport commented 5 months ago

I currently had the same issue 2 times.

Setup: Galera Manager 1.8.3 on host 1 Cluster with Ubuntu 20.04 and mysql 8 N1 synched N2 synched N3 synched

N1 is used from a application and has read/write actions The database servers have about 80GB of data each

Issue reproduction: Setting up a new node (n4) on ubuntu 20.04

Node setup runs until it trys to start the node.

Then n3 got into donor status, shortly after n2 gets into donor status and in the end n1 goes into donor status.

The log goes endless : 2024-01-26T16:36:55.449549Z 0 [Warning] [MY-000000] [WSREP] P: Member 0.0 (DSR-Core-n4-v2) requested state transfer from 'any', but it is impossible to select State Transfer donor: Resource temporarily unavailable

First time it happend I had to force delete the new node. All other nodes stayed in donor/desync. Had to shutdown all nodes and bootstrap the cluster to get it working again.

Second time i tried on a complete new installation of ubuntu 20.04 and got the same issuee. I turned off the n4. Dint help I restartet n3...now gives me the same error 2024-01-26T16:36:56.184421Z 0 [Warning] [MY-000000] [WSREP] P: Member 1.0 (DSR-Core-n3-hv) requested state transfer from 'any', but it is impossible to select State Transfer donor: Resource temporarily unavailable

I dint find any solutions online so far.

dbucher-datasport commented 5 months ago

I found that the 3 existing nodes run version 8.0.34-26.15 and the new created one 8.0.35-26.16...

Could this be related. Does the Manager not check the version of mysql at the existing nodes to install the same version?

ayurchen commented 5 months ago

No, this should not affect things at all. Could you please provide error logs from all nodes from the point where the new node joined the cluster and to the point where the third node became a DONOR.

byte commented 5 months ago

@dbucher-datasport is it possible to get error logs from all the nodes as asked for in the previous comment, thanks

dbucher-datasport commented 5 months ago

@ayurchen I've sent you the logs now in an email. Sorry for the late response.