search

redis / lettuce

Advanced Java Redis client for thread-safe sync, async, and reactive usage. Supports Cluster, Sentinel, Pipelining, and codecs.
https://lettuce.io
MIT License
5.38k stars 965 forks source link

RedisClusterClient requires 3 seconds to the first connection #1447

Closed donggyu81 closed 3 years ago

donggyu81 commented 4 years ago

Hello, I've implemented service using redis cluster with lettuce library. But I have difficult on lazy in first redis connection. When I tested with jedis, it has 30ms to get first connection, but tested with lettuce, it has 2600 ~ 3000 ms. How can I set lettuce configuration to reduce time to get first connection?

Stack: spring-data-redis: 2.2.7.RELEASE lettuce-core: 6.0.0.RELEASE Clustered Redis version: 5.0.5 ( Redis has clustered with 2 servers )

Configuration

    @Bean
    public LettuceConnectionFactory redisConnectionFactory() {
        NbaseArcSeedAddressProcessor seedAddressProcessor = nbaseArcSeedAddressProcessor();
        List<String> nodes = List.of(
            StringUtils.split("10.105.66.206:21339,10.105.66.206:21349", ","));
        RedisClusterConfiguration redisClusterConfiguration = new RedisClusterConfiguration(nodes);
        return new LettuceConnectionFactory(redisClusterConfiguration, getLettuceClientConfiguration());
    }

    @Bean
    public LettuceClientConfiguration getLettuceClientConfiguration() {
        ClusterTopologyRefreshOptions clusterTopologyRefreshOptions = ClusterTopologyRefreshOptions.builder()
            .dynamicRefreshSources(
                true)
            .enablePeriodicRefresh(
                Duration.ofSeconds(
                    30))
            .build();

        ClusterClientOptions clientOptions = LettuceClientConfiguration.builder().build().getClientOptions()
            .map(ClusterClientOptions::builder)
            .orElse(ClusterClientOptions.builder())
            .topologyRefreshOptions(
                clusterTopologyRefreshOptions)

            .build();
        LettuceClientConfiguration lettuceClientConfiguration =
            LettuceClientConfiguration.builder().clientOptions(clientOptions).build();
        return lettuceClientConfiguration;
    }

Test code

@Slf4j
public class RedisConnectionTest {

    @Test
    public void test_getClusterNodes() {
        long start = System.currentTimeMillis();

        List<String> nodes = getNodes();
        assertThat(nodes).isNotNull();

        long executionTime = System.currentTimeMillis() - start;
        assertThat(executionTime).isLessThan(500L);
        log.info("Execution time : " + executionTime);
    }

    @Test
    public void test_lettuce_connection() {

        long start = System.currentTimeMillis();

        List<String> nodes = getNodes();
        assertThat(nodes).isNotNull();

        for (String node : nodes) {
            node = "redis://" + node;
            RedisClient client = RedisClient.create(node);
            StatefulRedisConnection<String, String> connection = client.connect();
            assertThat(connection.isOpen()).isTrue();
            connection.close();
        }

        log.info(" Connection Time : " + (System.currentTimeMillis() - start) + " ms");
    }

    @Test
    public void test_jedis_connection() {

        long start = System.currentTimeMillis();

        List<String> nodes = getNodes();
        assertThat(nodes).isNotNull();

        for (String node : nodes) {

            String[] ipAndPort = StringUtils.split(node, ":");
            assertThat(ipAndPort.length).isEqualTo(2);

            String ip = ipAndPort[0];
            int port = Integer.valueOf(ipAndPort[1]);

            Jedis jedis = new Jedis(ip, port);
            assertThat(jedis).isNotNull();
            assertThat(jedis.ping()).isNotEmpty();

            jedis.close();
        }

        log.info(" Connection Time : " + (System.currentTimeMillis() - start) + " ms");
    }

    private List<String> getNodes() {
        return List.of(StringUtils.split("10.105.66.206:21339,10.105.66.206:21349", ","));
    }

}

Please let me know if you need any other info. Any advise will be very helpful. thank you.

mp911de commented 4 years ago

Out of interest, can you attach a debug log from the startup sequence so we can cross-check it whether there's an unusually long delay?

donggyu81 commented 3 years ago

@mp911de sorry for late response. i've attached a log.

mp911de commented 3 years ago

Lettuce uses a threaded infrastructure and booting netty including EventLoopGroups takes a bit of time. There's nothing we could do directly about it. You could try to pre-initialize Netty resources such as obtaining and releasing a buffer from ByteBufAllocator.DEFAULT and 

NioEventLoopGroup allocate = clientResources.eventLoopGroupProvider().allocate(NioEventLoopGroup.class);
allocate.submit(() -> {}); // warm the ThreadPool
clientResources.eventLoopGroupProvider().release(allocate, 0, 0, TimeUnit.NANOSECONDS);

clientResources.eventExecutorGroup().submit(() -> {
});

clientResources.eventBus().publish(new Event() {
});
mp911de commented 3 years ago

Let me add also a Cluster-specific note: Lettuce obtains the Cluster topology before establishing the actual Cluster connection. Topology uses a discovery mechanism by connecting the seed nodes and discovered nodes.

You can reduce the duration for the topology retrieval by:

  1. Reducing the number of seed nodes (RedisClusterClient.create(clientResources, redisURIs) where redisURIs contains one or two nodes).
  2. Disabling dynamic refresh sources (ClusterClientOptions.builder().topologyRefreshOptions(ClusterTopologyRefreshOptions.builder().dynamicRefreshSources(false).build());)

Please let us know whether this helps.