Clearing processed entries in RingBuffer

[golx]

Describe the issue The processed log entries stay in the ring buffer and consume memory.

What you are trying to do We have a centrally deployed instance of the MockServer that our CI jobs use to test our application. We added more memory-intensive tests (sending and receiving rather large HTTP bodies) recently and started noticing that our centrally deployed instance becomes less and less responsive over time and eventually hangs. Tests do clean up after themselves, but log entries are still kept in memory because they're referenced by the ring buffer, and the default size of this buffer is 65600, so at max, this many entries will be held. Digging into the code we found maxExpectations property that, among other things, controls the size of the ring buffer. By setting this property to 100 instead of its default value we were able to reduce memory consumption somewhat.

MockServer version 5.10.0

To Reproduce Steps to reproduce the issue: Run the server: java -Dmockserver.disableSystemOut=true -Dmockserver.maxExpectations=100 -Xms3g -Xmx3g -jar mockserver-netty-jar-with-dependencies.jar -serverPort 1080 -logLevel INFO Run the test:

const axios = require('axios')
const mockServerClient = require('mockserver-client').mockServerClient

const client = mockServerClient('localhost', 1080)

const createExpectation = async () => {
    return client.mockAnyResponse({
        'httpRequest': {
            'method': 'POST',
            'path': '/abcd',
        },
        'httpResponse': {
            'body': Array(200).fill('bb5dc8842ca31d4603d6aa11448d14').join(' ')
        }
    })
        .then(
            function () {
                console.log('Expectation created')
            },
            function (error) {
                console.log(error)
            }
        )
}

const removeExpectation = async () => {
    return client.clear({
        'method': 'POST',
        'path': '/abcd',
    })
        .then(
            function () {
                console.log('Expectation cleared')
            },
            function (error) {
                console.log(error)
            }
        )
}

const test = async () => {
    await createExpectation()

    const requestBody = {
        'ids': Array(200).fill('110ec58a-a0f2-4ac4-8393-c866d813b8d1').join(' '),
        'startDate': null,
        'endDate': null
    }

    for (let i = 0; i < 50000; i++) {
        (i % 100 === 0) && console.log(`Iteration ${i}`)

        const res = await axios.post('http://localhost:1080/abcd', requestBody)
    }

    await removeExpectation()
}

test().then(() => console.log('test complete')).catch(console.error)

On my machine it hovers at around 1GB of the heap with occasional spikes to 1.5GB until GC kicks in, but never dips below 1GB.

Expected behaviour After the test completes, the MockServer should return to the same level of memory consumption as before test started.

MockServer Log disabled

Suggested fix Implement cleanup approach as outlined here: https://github.com/LMAX-Exchange/disruptor/wiki/Getting-Started#clearing-objects-from-the-ring-buffer

[jamesdbloom]

The latest SNAPSHOT version has improvements around dynamically resizing the size of the log events and max expectations by inspecting the remaining memory.

I will however in addition add a call to clear() and will push this change once the existing SNAPSHOT build completes.

[jamesdbloom]

This was a little more complex than at first would appear, in particular, to avoid an additional copy of the data in the ring buffer. I have now completed this and after large performance tests peaking memory at 300MB of used heap the memory when back to 30KB after calling /reset.

[jamesdbloom]

This is now available in the latest SNAPSHOT build.