search

EventStore / EventStoreDB-Client-Java

Official Asynchronous Java 8+ Client Library for EventStoreDB 20.6+
https://eventstore.com
Apache License 2.0
63 stars 20 forks source link

Maximum Append Size of 1048576 Exceeded #128

Closed rohsin47 closed 2 years ago

rohsin47 commented 2 years ago

Hi,

We have been receiving below exception on appending to stream now. From docs i can see, events size that can be appended to stream is approx 16 MB. But from below exception, it looks like we cannot append more than 1 MB. Is there a buffer limit on streaming to event store?

_java.util.concurrent.ExecutionException: io.grpc.StatusRuntimeException: INVALID_ARGUMENT: Maximum Append Size of 1048576 Exceeded. at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395) at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1999) ... 7 more Caused by: io.grpc.StatusRuntimeException: INVALIDARGUMENT: Maximum Append Size of 1048576 Exceeded. at io.grpc.Status.asRuntimeException(Status.java:533) at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:478) at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:434) at io.grpc.internal.ClientCallImpl.access$500(ClientCallImpl.java:66) at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:763) at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:742) at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) ... 3 more

On further investigation, it seems to be coming from netty client, used by grpc to connect to server and i can see there is windows frame flow control for http/2, set at netty channel, which has default value of 1048576. is there a way we can override that?

Server version : 21.10.1.0 Client : 1.0.0 (java)

Thanks.

YoEight commented 2 years ago

Hey @rohsin47, and thanks for reporting this. I wasn't aware of that default limitation with the Java gRPC implementation. We can definitely use the maxInboundMessageSize internally. Expect a patch landing this week.

By the way, version 2.0.0 of the java client is available on Maven Central!

ahjohannessen commented 2 years ago

@YoEight Seems like that error is from https://github.com/EventStore/EventStore/blob/3108c3d26e753770778af04f1916b104105261ed/src/EventStore.Core/Services/Transport/Grpc/RpcExceptions.cs#L87

YoEight commented 2 years ago

@ahjohannessen You are the MVP! I was looking at the wrong location. Let's fix this 💪

ahjohannessen commented 2 years ago

I think there is a server setting, MaxAppendSize, for this?

YoEight commented 2 years ago

There is. @rohsin47 any chance you changed that value without knowing?

ahjohannessen commented 2 years ago

I think the default value is 1MB, which is 1048576 bytes

YoEight commented 2 years ago

@rohsin47 After conducting some research, this change has been going on since 20.6. The DB still allows an event with a payload as large as 16MB but you still have to compose with the max append size too. You can obviously increase that value on your own.

A bit of history: https://discuss.eventstore.com/t/how-to-change-max-append-size/2810/3

@ahjohannessen Yeah, I totally forgot about that setting, to be honest.