Reactor. Memory leak

[typik89]

Reactor. Memory leak

I created an issue in spring-webflux project: https://github.com/spring-projects/spring-framework/issues/27965. But It seems that this is a reactor issue. I created a test to reproduce behavior when objects created during the running of a pipeline of creating byteArray aggregate are held in the heap memory for a long time after finishing the pipeline: https://github.com/typik89/webfluxMemoryRetroProject/blob/main/src/test/kotlin/ru/typik/reactor/MemoryLeakTest.kt When I run the test with -xmx1000MB, I see 6-7 log messages about ByteArray being created and after that, I see OutOfMemoryError and test fails. When I run the test with -xmx2000MB, the test works fine in an infinite loop. I create a heap dump and I see 9 ByteArrays with a size of about 130MB. It seems that Reactor holds 9 results of pipeline in the heap and other results are released successfully. I don't understand why it happens and what is this magical number 9 and how I can configure it.

[simonbasle]

@typik89 can you better explain why and where you think Reactor retains 9 "results of pipeline"? What are these results exactly? In which field of which class are they retained?

I've tried to explore a heap dump generated by a pure Reactor unit test (which took some porting effort from your Kotlin-with-db sample), but so far not sure I see anything suspicious.

[typik89]

I run the test with xmx2000m. I see in logs 11 messages: 2022-02-08 15:10:35.838 INFO 17624 --- [actor-tcp-nio-1] test : ByteArray with size 68888896 was created 2022-02-08 15:10:40.105 INFO 17624 --- [actor-tcp-nio-1] test : ByteArray with size 68888896 was created 2022-02-08 15:10:44.680 INFO 17624 --- [actor-tcp-nio-1] test : ByteArray with size 68888896 was created 2022-02-08 15:10:49.443 INFO 17624 --- [actor-tcp-nio-1] test : ByteArray with size 68888896 was created 2022-02-08 15:10:54.003 INFO 17624 --- [actor-tcp-nio-1] test : ByteArray with size 68888896 was created 2022-02-08 15:10:58.508 INFO 17624 --- [actor-tcp-nio-1] test : ByteArray with size 68888896 was created 2022-02-08 15:11:03.486 INFO 17624 --- [actor-tcp-nio-1] test : ByteArray with size 68888896 was created 2022-02-08 15:11:08.553 INFO 17624 --- [actor-tcp-nio-1] test : ByteArray with size 68888896 was created 2022-02-08 15:11:13.623 INFO 17624 --- [actor-tcp-nio-1] test : ByteArray with size 68888896 was created 2022-02-08 15:11:18.437 INFO 17624 --- [actor-tcp-nio-1] test : ByteArray with size 68888896 was created 2022-02-08 15:11:23.878 INFO 17624 --- [actor-tcp-nio-1] test : ByteArray with size 68888896 was created 2022-02-08 15:11:29.099 INFO 17624 --- [actor-tcp-nio-1] test : ByteArray with size 68888896 was created 2022-02-08 15:11:42.585 INFO 17624 --- [actor-tcp-nio-1] test : ByteArray with size 68888896 was create

Unfortunutely heap dump file is too bigg to attach. I made screenshots. I made heapdump and I see 9 live objects: Show nearest GC Root:

Test runs the pipeline sequentially so I don't see the reason while old results are live objects...

[simonbasle]

I don't know exactly what retains the ByteArrayOutputStream, still working my way up to it, but I have suspicion this is in the postgresql r2dbc driver (ReactorNettyClient.java more specifically).

One way of working around that in the meantime could be to extend ByteArrayOutputStream and override the reset() method to replace the internal buf with an empty one. Then, make sure to call that reset() method when you turn the output stream into a ByteArray in that final map operator:

class ErasableByteArrayOutputStream: ByteArrayOutputStream() {

    override fun reset() {
        super.reset()
        buf = ByteArray(32)
    }
}

@Service
class CsvService(val csvRepository: CsvRepository) {
    fun createByteArray(): Mono<ByteArray> {
        return csvRepository.findAll()
            .reduce(ErasableByteArrayOutputStream()) { output, el ->
                output.write(el.toString().toByteArray())
                output.write(" ".toByteArray())
                output
            }
            .map {
                val result = it.toByteArray()
                it.reset()
                result
            }
    }
}

[typik89]

I thought the same and I've tried this before. I've seen an identical picture. It seems it's not BAOS. It's the resulting ByteArray in memory as I understand. Logically, there are no reasons to BAOS keeps being a live object, and the resulting ByteArray doesn't

[simonbasle]

From the heap dumps I inspected, it is definitely the internal byte array of the ByteArrayInputStream (and the BAOS itself) that are not garbage collected.

[typik89]

Ok, I tried it again what I see after 3 sequetinal requests:

[mp911de]

Let me have a look what happens there.

[simonbasle]

After discussing with @mp911de it points to something happening at the r2dbc level.

Not my intention to play issue tennis here, but I've opened an issue in r2dbc-postgresql: https://github.com/pgjdbc/r2dbc-postgresql/issues/492.

I've also share some code that mirrored the reproducer, with a few amendments, to run in isolation within the r2dbc-postgresql test suite.

Closing this as it doesn't appear to be an issue in reactor-core.