search

AxonIQ / axon-server-se

Axon Server - Standard Edition
Other
136 stars 45 forks source link

Invoking snapshots GET endpoint timesout without fetching a result #226

Closed 651juan closed 3 years ago

651juan commented 3 years ago

When invoking the GET endpoint {url}/v1/snapshots?aggregateId={your-aggregate-id} the request timesout after 30s producing the following warns:

Async request timed out Resolved [org.springframework.web.context.request.async.AsyncRequestTimeoutException]

a few seconds later this exception is thrown:

java.lang.IllegalStateException: ResponseBodyEmitter is already set complete
    at org.springframework.util.Assert.state(Assert.java:73)
    at org.springframework.web.servlet.mvc.method.annotation.ResponseBodyEmitter.send(ResponseBodyEmitter.java:182)
    at org.springframework.web.servlet.mvc.method.annotation.SseEmitter.send(SseEmitter.java:133)
    at io.axoniq.axonserver.rest.EventsRestController$1.onCompleted(EventsRestController.java:121)
    at io.axoniq.axonserver.enterprise.messaging.J.onCompleted(nk:79)
    at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:447)
    at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
    at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
    at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
    at io.grpc.census.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:701)
    at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
    at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
    at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
    at io.grpc.census.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:399)
    at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:426)
    at io.grpc.internal.ClientCallImpl.access$500(ClientCallImpl.java:66)
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:689)
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$900(ClientCallImpl.java:577)
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:751)
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:740)
    at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
    at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:834)

This occurs on an aggregate with a large number of events. Using initialSequence & maxSequence to create a smaller search window does not help.

MGathier commented 3 years ago

Can you tell me which version of Axon Server you are using, and what index type?

651juan commented 3 years ago

Using ee v4.4.7 with JUMP_SKIP_INDEX.

MGathier commented 3 years ago

The situation here is an aggregate with a huge number of snapshots (estimated around 680K) with many of these snapshots in the same segment. While reading the snapshots for the aggregate it first needs to find the index positions of the snapshots, and then will start to return the snapshots (most recent first). The snapshots are returned in a stream, which is only closed when the last snapshot is returned. Using swagger or curl will return with an error when the timeout is exceeded. When calling the service directly from chrome it returns the first values.