spring-projects / spring-integration-extensions

The Spring Integration Extensions project provides extension components for Spring Integration
http://www.springintegration.org/
280 stars 266 forks source link

Restarting HZ cluster will not re-establish any leadership #259

Closed RobbanHoglund closed 2 years ago

RobbanHoglund commented 2 years ago

I have tested with the code below and it works good when restarting individual HZ members. But when restarting the whole HZ cluster ( shut down everything), no clients will ever be the leader again (until all clients are restarted).

    public static final String LEADER = "leader_lock";

    private LeaderInitiator leaderInitiator = null;

    @Bean
    public LeaderInitiator initiator(HazelcastInstance instance) {

        return leaderInitiator = new LeaderInitiator(instance, new DefaultCandidate("", LEADER));
    }

    @Scheduled(fixedDelay = 30000 )
    public void runTask() {
        if (leaderInitiator != null) {
            if (leaderInitiator.getContext().isLeader()) {
                LOG.info("I am the leader!");
            } else {
                LOG.info("I am NOT the leader!");
            }

        }
    }
artembilan commented 2 years ago

I'm not so good with Hazelcast to determine correctly from your description what is HZ cluster and how to configure it.

Any chances to have some simple project from you to reproduce and play with? Or JUnit test, since I'm good to use a Hazelcast.newHazelcastInstance(), therefore I guess something similar should be there for a cluster...

Thanks

RobbanHoglund commented 2 years ago

Hi,

Just create a simple springboot application and setup a hazelcast cluster locally in client/server mode (I use version 5.1.1 of Hazelcast).

This is a class/component should work out of the box in simple Springboot applicaiton:

import java.util.logging.Logger;

import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.integration.dsl.IntegrationFlow;
import org.springframework.integration.dsl.IntegrationFlows;
import org.springframework.integration.hazelcast.leader.LeaderInitiator;
import org.springframework.integration.leader.DefaultCandidate;
import org.springframework.scheduling.annotation.EnableScheduling;
import org.springframework.scheduling.annotation.Scheduled;

import com.hazelcast.core.HazelcastInstance;

@EnableScheduling
@Configuration
public class LeaderElection {
    private static final String CLASS_NAME = LeaderElection.class.getName();

    private static final Logger LOG = Logger.getLogger(CLASS_NAME);

    public static final String LEADER = "myleader";

    private LeaderInitiator leaderInitiator = null;

    public LeaderElection (HazelcastInstance instance) {

        leaderInitiator = new LeaderInitiator(instance, new DefaultCandidate("", LEADER));
    }

    @Scheduled(fixedDelay = 30000)
    public void runTask() {
        if (leaderInitiator != null) {
            if (leaderInitiator.getContext().isLeader()) {
                LOG.info("I am the leader!");
            } else {
                LOG.info("I am NOT the leader!");
            }

        }
    }

}

So running this works fine until the while HZ cluster is shut down and the brought up again.

Then no one will be leader again, until the application is restarted.

ps: I also tested using pure FencedLock, i.e. bypassing Spring integration, then the client is recovering fine.

//Robert

artembilan commented 2 years ago

Just create a simple springboot application and setup a hazelcast cluster locally in client/server mode

Right. That's exactly the part I'm not familiar with. Any chances you can share your sample configuration with me, so if I would try something myself, it won't lead to different results.

BTW, the HZ LeaderInitiator uses a FencedLock starting with version 2.0.0. Perhaps the logic around it in that class is somehow off since you confirm that plain FencedLock works well? Would you mind to take a look into source code and probably spot a flaw in the algorithm?

Thanks for understanding!

RobbanHoglund commented 2 years ago

I did a test and some modifications. Can you grant me permission to push/create a branch?

//Robert

artembilan commented 2 years ago

Hi, @RobbanHoglund !

Well, we don't interact that way. We follow a contribution procedure via forks and Pull Requests: https://github.com/spring-projects/spring-integration/blob/main/CONTRIBUTING.adoc

That one is general guide lines for Spring Integration, but that should be easy to apply for this extensions project as well.

Looking forward for your PR!

Thanks