MultiCommand.MAX_BUFFER_SIZE limit

[honourednihilist]

Last week I faced the following exception: java.lang.IllegalArgumentException: Invalid readBytes length: 33554432.

I used aerospike-helper to fetch some data. The source of the exception is the following code:

public abstract class MultiCommand extends SyncCommand {
    private static final int MAX_BUFFER_SIZE = 1024 * 1024 * 10;  // 10 MB

    ...

    protected final void readBytes(int length) throws IOException {
        if (length > dataBuffer.length) {
            // Corrupted data streams can result in a huge length.
            // Do a sanity check here.
            if (length > MAX_BUFFER_SIZE) {
                throw new IllegalArgumentException("Invalid readBytes length: " + length);
            }
            dataBuffer = new byte[length];
        }

        int pos = 0;

        while (pos < length) {
            int count = bis.read(dataBuffer, pos, length - pos);

            if (count < 0) {
                throw new EOFException();
            }
            pos += count;
        }
        dataOffset += length;
    }

    ...

}

According to the comment, data might be corrupted. But then I removed this limit and my data was successfully deserialized. Nothing was corrupted at all.

So I am wondering, is there any reason for this restriction? Or maybe it is some sort of legacy code?

As far as I know C client does not contain this limit. Could it be just removed for java client?

[BrianNichols]

The message sent to the client was corrupt, but the actual server stored record was likely fine. This indicates the corruption likely occurred by a socket that was unexpectedly shared or some other temporary memory corruption when serializing the data.

Which client and server versions are you using?

[BrianNichols]

The other possibility is that your block of records really did exceed 10MB. Were you using batch, scan, or query?

If batch or query, how many records were requested? Could the total record size exceed 10 MB?

[honourednihilist]

I am using versions 3.3.0 and 3.10.0.3 of client and server respectively.

I believe that my block of records really exceeded 10MB. It was a query and about 1.8kk records were requested. The total size of them was about 32MB.

In my case, this exception is reproducible 10 of 10 runs. When I switch off the limit, it works just fine.

[BrianNichols]

I guess we will have to remove the client limit if the server is really sending back blocks that large. I will also ask the server team if it's possible to send back records in smaller blocks for queries.

[wchu-citrusleaf]

@honourednihilist the server's query buffer size is by default 2Meg. Was your server configuration changed to allow greater than 10MB?

You can check the server setting via the asinfo command - asinfo -v "get-config:context=service" -l | grep query-buf-size query-buf-size=2097152

[honourednihilist]

@wchu-citrusleaf nope, it hasn't been changed. The value is exactly the same - 2097152.

[citrusraj]

@honourednihilist the limits are binding. Would it be possible to actually share code around the the query request you are making. And output of "asadm -e collectinfo" (this would generate tar file with config and statistics)

[honourednihilist]

@citrusraj here is the code about the issue - https://github.com/honourednihilist/aerospike-client-java-issue-66 . CacheAerospikeImpl.getOldEvents makes the query request.

But I can't reproduce it anymore. Neither using my local (laptop) environment nor work environment. I have no idea why. Aerospike server and client settings haven't been changed, but data has been changed.