search

btcpayserver / btcpayserver-docker

Docker resources for hosting BTCPayServer easily
MIT License
575 stars 353 forks source link

out of memory for querying API #600

Open joltcan opened 2 years ago

joltcan commented 2 years ago

We had a bunch of payments due to a product launch and when we use the API to query for open payments we are getting memory exceptions in generated_btcpayserver_1, generated_nbxplorer_1:

System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
fatal error: out of memory allocating heap arena map

warn: PayServer:      Unhandled exception in TransactionLabelMarkerHostedService
System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
   at Microsoft.EntityFrameworkCore.Diagnostics.Internal.RelationalCommandDiagnosticsLogger.CommandErrorAsync(IRelationalConnection connection, DbCommand command, DbContext context, DbCommandMethod executeMethod, Guid commandId, Guid connectionId, Exception exception, DateT
imeOffset startTime, TimeSpan duration, CommandSource commandSource, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.InitializeReaderAsync(AsyncEnumerator enumerator, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Storage.ExecutionStrategy.<>c__DisplayClass33_0`2.<<ExecuteAsync>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at Microsoft.EntityFrameworkCore.Storage.ExecutionStrategy.ExecuteImplementationAsync[TState,TResult](Func`4 operation, Func`4 verifySucceeded, TState state, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Storage.ExecutionStrategy.ExecuteImplementationAsync[TState,TResult](Func`4 operation, Func`4 verifySucceeded, TState state, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Storage.ExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.MoveNextAsync()
   at Microsoft.EntityFrameworkCore.Query.ShapedQueryCompilingExpressionVisitor.SingleOrDefaultAsync[TSource](IAsyncEnumerable`1 asyncEnumerable, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Query.ShapedQueryCompilingExpressionVisitor.SingleOrDefaultAsync[TSource](IAsyncEnumerable`1 asyncEnumerable, CancellationToken cancellationToken)
   at BTCPayServer.Services.WalletRepository.GetWalletInfo(WalletId walletId) in /source/BTCPayServer/Services/WalletRepository.cs:line 59
   at BTCPayServer.HostedServices.TransactionLabelMarkerHostedService.ProcessEvent(Object evt, CancellationToken cancellationToken) in /source/BTCPayServer/HostedServices/TransactionLabelMarkerHostedService.cs:line 75
   at BTCPayServer.HostedServices.EventHostedServiceBase.ProcessEvents(CancellationToken cancellationToken) in /source/BTCPayServer/HostedServices/EventHostedServiceBase.cs:line 38

And the containers are restarted. Is it possible to raise the memory for this not to happen? Is it nbxexplorer that is the first culprit?

nbexplorer:

Out of memory.
root@pay1:~/btcpay/btcpayserver-docker# docker logs -f generated_nbxplorer_1
runtime/cgo: pthread_create failed: Resource temporarily unavailable
SIGABRT: abort
PC=0x7f36fbca27bb m=0 sigcode=18446744073709551610

goroutine 0 [idle]:
runtime: unknown pc 0x7f36fbca27bb
stack: frame={sp:0x7ffdff531cd0, fp:0x0} stack=[0x7ffdfed33308,0x7ffdff532340)
00007ffdff531bd0:  0000000000000005  0000ffff00001fa0
00007ffdff531be0:  00007ffdff5320e0  00005615398e23f0
00007ffdff531bf0:  0000000000000000  00005615372bcd10
00007ffdff531c00:  0000000000000000  00007f36fbe68b00
00007ffdff531c10:  0000000000000005  0000000000000000
00007ffdff531c20:  00007ffdff531fc0  00007f36fbc7bf98
00007ffdff531c30:  00007ffdff531ff0  00007f36fbe6f4ba
00007ffdff531c40:  0000000000000000  0000000000000000
00007ffdff531c50:  0000000000000000  00007ffdff5320e0
00007ffdff531c60:  2525252525252525  2525252525252525
00007ffdff531c70:  0000000000000000  0000000000000000
00007ffdff531c80:  0000000000000000  0000000000000000
00007ffdff531c90:  0000000000000000  0000000000000000
00007ffdff531ca0:  0000000000000000  0000000000000000
00007ffdff531cb0:  0000000000000000  0000000000000000
00007ffdff531cc0:  0000000000000000  0000000000000000
00007ffdff531cd0: <0000000000000000  0000000000000000
00007ffdff531ce0:  5f64616572687470  6620657461657263
00007ffdff531cf0:  0000000000000000  0000000000000000
00007ffdff531d00:  0000000000000000  0000000000000000
00007ffdff531d10:  0000000000000000  0000000000000000
00007ffdff531d20:  0000000000000000  0000000000000000
00007ffdff531d30:  0000000000000000  0000000000000000
00007ffdff531d40:  0000000000000000  0000000000000000
00007ffdff531d50:  fffffffe7fffffff  ffffffffffffffff
00007ffdff531d60:  ffffffffffffffff  ffffffffffffffff
00007ffdff531d70:  ffffffffffffffff  ffffffffffffffff
00007ffdff531d80:  ffffffffffffffff  ffffffffffffffff
00007ffdff531d90:  ffffffffffffffff  ffffffffffffffff
00007ffdff531da0:  ffffffffffffffff  ffffffffffffffff
00007ffdff531db0:  ffffffffffffffff  ffffffffffffffff
00007ffdff531dc0:  ffffffffffffffff  ffffffffffffffff
runtime: unknown pc 0x7f36fbca27bb
stack: frame={sp:0x7ffdff531cd0, fp:0x0} stack=[0x7ffdfed33308,0x7ffdff532340)
00007ffdff531bd0:  0000000000000005  0000ffff00001fa0
00007ffdff531be0:  00007ffdff5320e0  00005615398e23f0
00007ffdff531bf0:  0000000000000000  00005615372bcd10
00007ffdff531c00:  0000000000000000  00007f36fbe68b00
00007ffdff531c10:  0000000000000005  0000000000000000
00007ffdff531c20:  00007ffdff531fc0  00007f36fbc7bf98
00007ffdff531c30:  00007ffdff531ff0  00007f36fbe6f4ba
00007ffdff531c40:  0000000000000000  0000000000000000
00007ffdff531c50:  0000000000000000  00007ffdff5320e0
00007ffdff531c60:  2525252525252525  2525252525252525
00007ffdff531c70:  0000000000000000  0000000000000000
00007ffdff531c80:  0000000000000000  0000000000000000
00007ffdff531c90:  0000000000000000  0000000000000000
00007ffdff531ca0:  0000000000000000  0000000000000000
00007ffdff531cb0:  0000000000000000  0000000000000000
00007ffdff531cc0:  0000000000000000  0000000000000000
00007ffdff531cd0: <0000000000000000  0000000000000000
00007ffdff531ce0:  5f64616572687470  6620657461657263
00007ffdff531cf0:  0000000000000000  0000000000000000
00007ffdff531d00:  0000000000000000  0000000000000000
00007ffdff531d10:  0000000000000000  0000000000000000
00007ffdff531d20:  0000000000000000  0000000000000000
00007ffdff531d30:  0000000000000000  0000000000000000
00007ffdff531d40:  0000000000000000  0000000000000000
00007ffdff531d50:  fffffffe7fffffff  ffffffffffffffff
00007ffdff531d60:  ffffffffffffffff  ffffffffffffffff
00007ffdff531d70:  ffffffffffffffff  ffffffffffffffff
00007ffdff531d80:  ffffffffffffffff  ffffffffffffffff
00007ffdff531d90:  ffffffffffffffff  ffffffffffffffff
00007ffdff531da0:  ffffffffffffffff  ffffffffffffffff
00007ffdff531db0:  ffffffffffffffff  ffffffffffffffff
00007ffdff531dc0:  ffffffffffffffff  fffffffffffffff
goroutine 1 [running, locked to thread]:                                                                                                                                                                                                                                [108/1260]
runtime.systemstack_switch()
        /usr/local/go/src/runtime/asm_amd64.s:330 fp=0xc0000806c8 sp=0xc0000806c0 pc=0x561535aaec10
runtime.newproc(0x561500000000, 0x561537a5b218)
        /usr/local/go/src/runtime/proc.go:3261 +0x70 fp=0xc000080710 sp=0xc0000806c8 pc=0x561535a8c790
runtime.init.5()
        /usr/local/go/src/runtime/proc.go:242 +0x37 fp=0xc000080730 sp=0xc000080710 pc=0x561535a84c37
runtime.doInit(0x561538a84280)
        /usr/local/go/src/runtime/proc.go:5228 +0x8c fp=0xc000080760 sp=0xc000080730 pc=0x561535a9110c
runtime.main()
        /usr/local/go/src/runtime/proc.go:150 +0xcb fp=0xc0000807e0 sp=0xc000080760 pc=0x561535a8495b
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc0000807e8 sp=0xc0000807e0 pc=0x561535ab0d11

rax    0x0
rbx    0x6
rcx    0x7f36fbca27bb
rdx    0x0
rdi    0x2
rsi    0x7ffdff531cd0
rbp    0x561537308460
rsp    0x7ffdff531cd0
r8     0x0
r9     0x7ffdff531cd0
r10    0x8
r11    0x246
r12    0x5615398e23f0
r13    0x0
r14    0x5615372bcd10
r15    0x0
rip    0x7f36fbca27bb
rflags 0x246
cs     0x33
fs     0x0
gs     0x0
pavlenex commented 2 years ago

@joltcan is this still relevant? Still facing a problem?

joltcan commented 2 years ago

Hey!

Since it crashed we started to not do api queries that way and started to investigate the blockchain ourselfs with our own bitcoind, so yes, I would assume it's still an issue. Let me know if I should test something.