Mass destruction of R2DBC pools at once.

[pkgonan]

Mass destruction of R2DBC pools at once.

initial_size : 20 MAX_SIZE: 20 MAX_LIFE_TIME: 10 Seconds BACKGROUND_EVICTION_INTERVAL: 1 Seconds validation_depth: local validation_query: select 1

With the above configured, a show processlist; query on the database will automatically remove all connections without attempting to maintain 20 connections.

If we have a situation where 19 of the 20 connections have been open for 10 seconds and the remaining 1 connection has been open for 1 second, common sense would dictate that only 19 connections should be removed, but all 20 connections are removed.

Kotlin + Spring Boot Webflux + R2DBC + R2DBC Pool Configuration

@Configuration
internal class R2dbcConfiguration internal constructor(
    @Value("\${payment-platform.r2dbc.host}")
    private val host: String,
    @Value("\${payment-platform.r2dbc.port}")
    private val port: Int,
    @Value("\${payment-platform.r2dbc.username}")
    private val username: String,
    @Value("\${payment-platform.r2dbc.password}")
    private val password: String,
    @Value("\${payment-platform.r2dbc.database}")
    private val database: String,
): AbstractR2dbcConfiguration() {

    companion object {
        private const val CONNECTION_POOL_INITIAL_SIZE = 20
        private const val CONNECTION_POOL_MAX_SIZE = CONNECTION_POOL_INITIAL_SIZE
        private val CONNECTION_POOL_MAX_LIFE_TIME = Duration.ofSeconds(10)
        private const val VALIDATION_QUERY = "SELECT 1"
    }

    @Bean
    override fun connectionFactory(): ConnectionFactory {
        return ConnectionFactories.get(
            ConnectionFactoryOptions.builder()
                .option(ConnectionFactoryOptions.SSL, false)
                .option(ConnectionFactoryOptions.HOST, host)
                .option(ConnectionFactoryOptions.PORT, port)
                .option(ConnectionFactoryOptions.USER, username)
                .option(ConnectionFactoryOptions.PASSWORD, password)
                .option(ConnectionFactoryOptions.DATABASE, database)
                .option(ConnectionFactoryOptions.DRIVER, PoolingConnectionFactoryProvider.POOLING_DRIVER)
                .option(ConnectionFactoryOptions.PROTOCOL, MysqlConnectionFactoryProvider.MYSQL_DRIVER)
                .option(PoolingConnectionFactoryProvider.INITIAL_SIZE, CONNECTION_POOL_INITIAL_SIZE)
                .option(PoolingConnectionFactoryProvider.MAX_SIZE, CONNECTION_POOL_MAX_SIZE)
                .option(PoolingConnectionFactoryProvider.MAX_ACQUIRE_TIME, Duration.ofSeconds(2))
                .option(PoolingConnectionFactoryProvider.MAX_CREATE_CONNECTION_TIME, Duration.ofSeconds(2))
                .option(PoolingConnectionFactoryProvider.MAX_LIFE_TIME, CONNECTION_POOL_MAX_LIFE_TIME)
                .option(PoolingConnectionFactoryProvider.BACKGROUND_EVICTION_INTERVAL, Duration.ofSeconds(1))
                .option(PoolingConnectionFactoryProvider.VALIDATION_DEPTH, ValidationDepth.LOCAL)
                .option(PoolingConnectionFactoryProvider.VALIDATION_QUERY, VALIDATION_QUERY)
                .build()
        )
    }
}

[mp911de]

It sounds a bit as if this would originate from Reactor Pool as R2DBC pool delegates all eviction and statistics functionality to Reactor Pool. Paging @pderop for further guidance.

[pkgonan]

@mp911de @pderop

Hi, How do I prevent mass extinction of a Connection Pool?

[pderop]

@pkgonan, ok, I'll check this tomorrow morning.

[pkgonan]

Hi, @pderop Have you checked?

BACKGROUND_EVICTION_INTERVAL causes all connections to be removed and recreated.

However, all connections are recreated even if BACKGROUND_EVICTION_INTERVAL is not used.

[pderop]

I was in trouble on another PR, I'm now starting to check.

[pderop]

Can you confirm which exact version of reactor-pool you are using ?

I have tried to reproduce your scenario using the following junit test (based on 1.0.1 release), but I cannot reproduce the problem. Can you try to modify it, maybe I have missed your exact use case ? If you can reproduce it, please create an issue on reactor-pool project.

here is my junit test, to test it, you can load the reactor-pool (main branch) into your IDE and then add this test, for example in the SimpleDequePoolInstrumentationTest.

    @Test
    void useCase() throws InterruptedException {
        VirtualTimeScheduler vts = VirtualTimeScheduler.create();
        Duration maxLifeTime = Duration.ofSeconds(10);

        SimpleDequePool<String> pool = new SimpleDequePool<>(
                PoolBuilder
                        .from(Mono.fromCallable(String::new))
                        .clock(SchedulerClock.of(vts))
                        .evictInBackground(Duration.ofMillis(1000), vts)
                        .evictionPredicate((s, metadata) -> {
                            return metadata.lifeTime() > maxLifeTime.toMillis();
                        })
                        .idleResourceReuseMruOrder()
                        .sizeBetween(20, 20)
                        .buildConfig());

        // Acquire 19 resources
        PooledRef<String>[] refs = IntStream.range(0, 19).mapToObj(i -> pool.acquire().block()).toArray(PooledRef[]::new);
        assertThat(pool.metrics().allocatedSize()).isEqualTo(20); // one more resource is created during lazy warmup
        assertThat(pool.metrics().idleSize()).isEqualTo(1);

        // wait for 1 sec
        vts.advanceTimeBy(Duration.ofMillis(1000));

        // acquire a twentieth resource (there is one remaining idle one)
        PooledRef<String> r20 = pool.acquire().block();
        // invalidate r20
        r20.invalidate().block();
        // re-acquire a new fresh r20 resource
        r20 = pool.acquire().block();

        // at this point, the first nineteen resources lifetime is around 1 sec, and r20 is fresh.
        assertThat(pool.metrics().allocatedSize()).isEqualTo(20);
        assertThat(pool.metrics().idleSize()).isEqualTo(0);

        // Now release all
        Stream.of(refs).forEach(ref -> ref.release().block());
        r20.release().block();

        // Wait 10 more sec, meaning that the first nineteen resources lifetime will be 11 sec
        vts.advanceTimeBy(Duration.ofMillis(10000));

        // the first nineteen resources should have been evicted, only 1 resource should still be acquired (r20)
        assertThat(pool.metrics().allocatedSize()).isEqualTo(1);

        // wait 1 sec, after that, r20 should have been evicted
        vts.advanceTimeBy(Duration.ofMillis(1000));
        assertThat(pool.metrics().allocatedSize()).isEqualTo(0);
    }