search

emeraldpay / dshackle

Fault Tolerant Load Balancer for Ethereum and Bitcoin APIs
Apache License 2.0
308 stars 55 forks source link

Crashes on too many requests #280

Open splix opened 9 months ago

splix commented 9 months ago 
Caused by: io.emeraldpay.dshackle.upstream.rpcclient.JsonRpcException: Pending acquire queue has reached its maximum size of 1000
    at io.emeraldpay.dshackle.upstream.rpcclient.JsonRpcError.asException(JsonRpcError.kt:34) ~[classes/:?]
    Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException:
Assembly trace from producer [reactor.core.publisher.MonoFlatMap] :
    reactor.core.publisher.Mono.flatMap
    io.emeraldpay.dshackle.reader.QuorumRpcReader.setupDefaultResult(QuorumRpcReader.kt:184)
Error has been observed at the following site(s):
    |_       Mono.flatMap ⇢ at io.emeraldpay.dshackle.reader.QuorumRpcReader.setupDefaultResult(QuorumRpcReader.kt:184)
    |_ Mono.switchIfEmpty ⇢ at io.emeraldpay.dshackle.reader.QuorumRpcReader.processResult$lambda$11(QuorumRpcReader.kt:123)
    |_     Mono.transform ⇢ at io.emeraldpay.dshackle.reader.QuorumRpcReader.read(QuorumRpcReader.kt:88)
    |_                    ⇢ at io.emeraldpay.dshackle.reader.QuorumRpcReader.read(QuorumRpcReader.kt:40)
    |_           Mono.map ⇢ at io.emeraldpay.dshackle.reader.MultistreamReader.read(MultistreamReader.kt:43)
    |_                    ⇢ at io.emeraldpay.dshackle.reader.MultistreamReader.read(MultistreamReader.kt:13)
    |_       Mono.timeout ⇢ at io.emeraldpay.dshackle.reader.CompoundReader$read$1.invoke(CompoundReader.kt:46)
    |_     Mono.doOnError ⇢ at io.emeraldpay.dshackle.reader.CompoundReader$read$1.invoke(CompoundReader.kt:47)
    |_       Flux.flatMap ⇢ at io.emeraldpay.dshackle.reader.CompoundReader.read(CompoundReader.kt:44)
    |_          Flux.next ⇢ at io.emeraldpay.dshackle.reader.CompoundReader.read(CompoundReader.kt:56)
    |_       Mono.timeout ⇢ at io.emeraldpay.dshackle.upstream.ethereum.DataReaders$4.read(DataReaders.kt:106)
Stack trace:

The main reason is that the upstream is too slow to process the requests. Dshackle may drop requests in this case, but should not crash.