vulpemventures / go-secp256k1-zkp

Go bindings for the Elements' secp256k1-zkp C library.
MIT License
1 stars 5 forks source link

segmentation violation on alpine linux ppc64le #18

Open grubles opened 2 months ago

grubles commented 2 months ago

In PeerSwap we use go-secp256k1-zkp. When attempting an L-BTC swap, a segmentation violation occurs. Here are the logs from PeerSwap's peerswap-panic-log. This seems to only happen on Alpine Linux. Opening a ticket here for now since I'm not sure where it should exist.

Server started 2024-07-31T20:54:03Z                                                                                                                                                             20:55:01 [561/1776]
SIGSEGV: segmentation violation                                                                                                                                                                                    
PC=0x10850870 m=8 sigcode=2 addr=0x7fff44e5aff0                                                                                                                                                                    
signal arrived during cgo execution                                                                                                                                                                                

goroutine 164 gp=0xc000603a40 m=8 mp=0xc0003f8008 [syscall]:                                                                                                                                                       
runtime.cgocall(0x1085c210, 0xc00004b840)                                                                                                                                                                          
        /usr/lib/go/src/runtime/cgocall.go:157 +0x4c fp=0xc00004b7e8 sp=0xc00004b7a0 pc=0x1000988c                                                                                                                 
github.com/vulpemventures/go-secp256k1-zkp._Cfunc_secp256k1_context_create(0x301)                                                                                                                                  
        _cgo_gotypes.go:305 +0x3c fp=0xc00004b820 sp=0xc00004b7e8 pc=0x107b754c                                                                                                                                    
github.com/vulpemventures/go-secp256k1-zkp.ContextCreate(0x301)                                                                                                                                                    
        /home/user/go/pkg/mod/github.com/vulpemventures/go-secp256k1-zkp@v1.1.5/secp256k1.go:74 +0x4c fp=0xc00004b878 sp=0xc00004b820 pc=0x107bacdc                                                                
github.com/vulpemventures/go-elements/confidential.nonceHash({0xc0000402d0, 0x21, 0x30}, {0xc00053e3c0, 0x20, 0x20})                                                                                               
        /home/user/go/pkg/mod/github.com/vulpemventures/go-elements@v0.4.0/confidential/confidential.go:620 +0x94 fp=0xc00004b908 sp=0xc00004b878 pc=0x107bf2e4
github.com/vulpemventures/go-elements/confidential.NonceHash(...)
        /home/user/go/pkg/mod/github.com/vulpemventures/go-elements@v0.4.0/confidential/confidential.go:33
github.com/vulpemventures/go-elements/confidential.UnblindOutputWithKey(0xc00024ee10, {0xc00053e3c0?, 0x474330edbf9093af?, 0x5de0824f5e215f3b?})
        /home/user/go/pkg/mod/github.com/vulpemventures/go-elements@v0.4.0/confidential/confidential.go:64 +0x80 fp=0xc00004b9b8 sp=0xc00004b908 pc=0x107bf0a0
github.com/elementsproject/peerswap/onchain.(*LiquidOnChain).ValidateTx(0xc0003553b0, 0xc0000d6230, {0xc000634800, 0x45f4})
        /home/user/builds/peerswap/onchain/liquid.go:410 +0x130 fp=0xc00004baa0 sp=0xc00004b9b8 pc=0x107c6900
github.com/elementsproject/peerswap/swap.(*ValidateTxAndPayClaimInvoiceAction).Execute(0xc000190380?, 0xc0001cc1c0, 0xc000543380)
        /home/user/builds/peerswap/swap/actions.go:725 +0xd0 fp=0xc00004bbd8 sp=0xc00004baa0 pc=0x10376850
github.com/elementsproject/peerswap/swap.(*SwapStateMachine).SendEvent(0xc000190380, {0x10a98ee8, 0x13}, {0x0, 0x0})
        /home/user/builds/peerswap/swap/fsm.go:240 +0x7c0 fp=0xc00004bce8 sp=0xc00004bbd8 pc=0x10378310  
github.com/elementsproject/peerswap/swap.(*SwapService).OnTxConfirmed(0xc00038c300, {0xc0000a6480, 0x40}, {0xc000634800, 0x45f4}, {0x0, 0x0?})
        /home/user/builds/peerswap/swap/service.go:336 +0x264 fp=0xc00004bdb0 sp=0xc00004bce8 pc=0x1037e364
github.com/elementsproject/peerswap/swap.(*SwapService).OnTxConfirmed-fm({0xc0000a6480?, 0x0?}, {0xc000634800?, 0x11231030?}, {0x0?, 0x0?})
        <autogenerated>:1 +0xa8 fp=0xc00004be08 sp=0xc00004bdb0 pc=0x1038e4c8
github.com/elementsproject/peerswap/txwatcher.(*BlockchainRpcTxWatcher).callbackAndLog(0x0?, {0xc0000a6480, 0x40}, {0xc000634800?, 0x1037c508?}, {0x0, 0x0})
        /home/user/builds/peerswap/txwatcher/rpctxwatcher.go:361 +0x94 fp=0xc00004be60 sp=0xc00004be08 pc=0x1083b614
github.com/elementsproject/peerswap/txwatcher.(*BlockchainRpcTxWatcher).observationLoop(0xc000186310, {0x10c8bdc0, 0xc0000d6140}, {0xc0000a6480, 0x40}, {0xc000622040, 0x40}, 0x1, 0x2d4a14, 0x1e, ...)
        /home/user/builds/peerswap/txwatcher/rpctxwatcher.go:353 +0x448 fp=0xc00004bf50 sp=0xc00004be60 pc=0x1083b358
github.com/elementsproject/peerswap/txwatcher.(*BlockchainRpcTxWatcher).AddWaitForConfirmationTx.gowrap1()
        /home/user/builds/peerswap/txwatcher/rpctxwatcher.go:209 +0x74 fp=0xc00004bfc0 sp=0xc00004bf50 pc=0x1083a484
runtime.goexit({})
        /usr/lib/go/src/runtime/asm_ppc64x.s:1018 +0x4 fp=0xc00004bfc0 sp=0xc00004bfc0 pc=0x1007c204     
created by github.com/elementsproject/peerswap/txwatcher.(*BlockchainRpcTxWatcher).AddWaitForConfirmationTx in goroutine 404
        /home/user/builds/peerswap/txwatcher/rpctxwatcher.go:209 +0x214

goroutine 1 gp=0xc0000021c0 m=nil [chan receive]:
runtime.gopark(0x1?, 0x0?, 0xf8?, 0xdc?, 0x1083dd0c?)
        /usr/lib/go/src/runtime/proc.go:402 +0x114 fp=0xc0005399c8 sp=0xc000539998 pc=0x10045144         
runtime.chanrecv(0xc0005bcde0, 0x0, 0x1)
        /usr/lib/go/src/runtime/chan.go:583 +0x4f4 fp=0xc000539a60 sp=0xc0005399c8 pc=0x1000c214         
runtime.chanrecv1(0x10a99fd5?, 0x10c7f088?)
        /usr/lib/go/src/runtime/chan.go:442 +0x10 fp=0xc000539a98 sp=0xc000539a60 pc=0x1000bce0          
main.run({0x10c8bdc0, 0xc00015adc0}, 0xc0001e0400)
        /home/user/builds/peerswap/cmd/peerswap-plugin/main.go:404 +0x1b44 fp=0xc000539dc0 sp=0xc000539a98 pc=0x1083dd14
main.outer()
        /home/user/builds/peerswap/cmd/peerswap-plugin/main.go:102 +0x1d0 fp=0xc000539e80 sp=0xc000539dc0 pc=0x1083be70
main.main()
        /home/user/builds/peerswap/cmd/peerswap-plugin/main.go:56 +0x108 fp=0xc000539f38 sp=0xc000539e80 pc=0x1083bab8
runtime.main()                                                                                                                                                                                  20:55:01 [509/1776]
        /usr/lib/go/src/runtime/proc.go:271 +0x2f0 fp=0xc000539fc0 sp=0xc000539f38 pc=0x10044c40         
runtime.goexit({})
        /usr/lib/go/src/runtime/asm_ppc64x.s:1018 +0x4 fp=0xc000539fc0 sp=0xc000539fc0 pc=0x1007c204     

goroutine 2 gp=0xc000002c40 m=nil [force gc (idle)]: 
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/lib/go/src/runtime/proc.go:402 +0x114 fp=0xc0000acf78 sp=0xc0000acf48 pc=0x10045144         
runtime.goparkunlock(...)
        /usr/lib/go/src/runtime/proc.go:408
runtime.forcegchelper()
        /usr/lib/go/src/runtime/proc.go:326 +0xe8 fp=0xc0000acfc0 sp=0xc0000acf78 pc=0x10044f78          
runtime.goexit({})
        /usr/lib/go/src/runtime/asm_ppc64x.s:1018 +0x4 fp=0xc0000acfc0 sp=0xc0000acfc0 pc=0x1007c204     
created by runtime.init.5 in goroutine 1
        /usr/lib/go/src/runtime/proc.go:314 +0x2c

goroutine 3 gp=0xc000003180 m=nil [GC sweep wait]:
runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/lib/go/src/runtime/proc.go:402 +0x114 fp=0xc0000ad740 sp=0xc0000ad710 pc=0x10045144         
runtime.goparkunlock(...)
        /usr/lib/go/src/runtime/proc.go:408
runtime.bgsweep(0xc0000d6000)
        /usr/lib/go/src/runtime/mgcsweep.go:318 +0x138 fp=0xc0000ad798 sp=0xc0000ad740 pc=0x1002ee48     
runtime.gcenable.gowrap1()
        /usr/lib/go/src/runtime/mgc.go:203 +0x4c fp=0xc0000ad7c0 sp=0xc0000ad798 pc=0x1002209c           
runtime.goexit({})
        /usr/lib/go/src/runtime/asm_ppc64x.s:1018 +0x4 fp=0xc0000ad7c0 sp=0xc0000ad7c0 pc=0x1007c204     
created by runtime.gcenable in goroutine 1
        /usr/lib/go/src/runtime/mgc.go:203 +0x7c

goroutine 4 gp=0xc000003340 m=nil [sleep]:
runtime.gopark(0xc0000e2000?, 0x58127a2ccfeb?, 0x0?, 0x0?, 0x10b93d38?)
        /usr/lib/go/src/runtime/proc.go:402 +0x114 fp=0xc0000adee8 sp=0xc0000adeb8 pc=0x10045144         
runtime.goparkunlock(...)
        /usr/lib/go/src/runtime/proc.go:408
runtime.(*scavengerState).sleep(0x112d2960, 0x40faa7d000000000)
        /usr/lib/go/src/runtime/mgcscavenge.go:504 +0x140 fp=0xc0000adf68 sp=0xc0000adee8 pc=0x1002c6d0  
runtime.bgscavenge(0xc0000d6000)
        /usr/lib/go/src/runtime/mgcscavenge.go:662 +0xa4 fp=0xc0000adf98 sp=0xc0000adf68 pc=0x1002cae4   
runtime.gcenable.gowrap2()
        /usr/lib/go/src/runtime/mgc.go:204 +0x4c fp=0xc0000adfc0 sp=0xc0000adf98 pc=0x1002203c           
runtime.goexit({})
        /usr/lib/go/src/runtime/asm_ppc64x.s:1018 +0x4 fp=0xc0000adfc0 sp=0xc0000adfc0 pc=0x1007c204     
created by runtime.gcenable in goroutine 1
        /usr/lib/go/src/runtime/mgc.go:204 +0xc4

goroutine 18 gp=0xc000102700 m=nil [finalizer wait]: 
runtime.gopark(0x10044d40?, 0xc0000021c0?, 0x68?, 0x9?, 0x10077228?)
        /usr/lib/go/src/runtime/proc.go:402 +0x114 fp=0xc0000ac5d8 sp=0xc0000ac5a8 pc=0x10045144         
runtime.runfinq()
        /usr/lib/go/src/runtime/mfinal.go:194 +0x120 fp=0xc0000ac7c0 sp=0xc0000ac5d8 pc=0x10021020       
runtime.goexit({})                                                                                                                                                                              20:55:01 [457/1776]
        /usr/lib/go/src/runtime/asm_ppc64x.s:1018 +0x4 fp=0xc0000ac7c0 sp=0xc0000ac7c0 pc=0x1007c204     
created by runtime.createfing in goroutine 1
        /usr/lib/go/src/runtime/mfinal.go:164 +0x84

goroutine 19 gp=0xc0001dd500 m=0 mp=0x112d39e0 [syscall]:
syscall.Syscall(0x3, 0x0, 0xc0001b4800, 0x800)
        /usr/lib/go/src/syscall/syscall_linux.go:69 +0x1c fp=0xc0000bcb48 sp=0xc0000bcad8 pc=0x10090d7c  
syscall.read(0xc000130000?, {0xc0001b4800?, 0x200000000?, 0x100000000?})
        /usr/lib/go/src/syscall/zsyscall_linux_ppc64le.go:736 +0x6c fp=0xc0000bcb98 sp=0xc0000bcb48 pc=0x1008edbc
syscall.Read(...)
        /usr/lib/go/src/syscall/syscall_unix.go:181
internal/poll.ignoringEINTRIO(...)
        /usr/lib/go/src/internal/poll/fd_unix.go:736 
internal/poll.(*FD).Read(0xc000130000, {0xc0001b4800, 0x800, 0x800})
        /usr/lib/go/src/internal/poll/fd_unix.go:160 +0x2a0 fp=0xc0000bcc40 sp=0xc0000bcb98 pc=0x100ece90
os.(*File).read(...)
        /usr/lib/go/src/os/file_posix.go:29
os.(*File).Read(0xc00011a050, {0xc0001b4800?, 0x19a8?, 0x7fff8bb4a408?})
        /usr/lib/go/src/os/file.go:118 +0xa4 fp=0xc0000bcc90 sp=0xc0000bcc40 pc=0x100f70c4               
bufio.(*Scanner).Scan(0xc0000bce00)
        /usr/lib/go/src/bufio/scan.go:219 +0x9d4 fp=0xc0000bcd78 sp=0xc0000bcc90 pc=0x102695f4           
github.com/elementsproject/glightning/jrpc2.(*Server).listen(0xc000354630, {0x10c7f188?, 0xc00011a050?})
        /home/user/go/pkg/mod/github.com/elementsproject/glightning@v0.0.0-20240224063423-55240d61b52a/jrpc2/server.go:108 +0x108 fp=0xc0000bceb0 sp=0xc0000bcd78 pc=0x102f0118
github.com/elementsproject/glightning/jrpc2.(*Server).StartUp(0xc000354630, 0xc00011a050, 0xc00011a058)  
        /home/user/go/pkg/mod/github.com/elementsproject/glightning@v0.0.0-20240224063423-55240d61b52a/jrpc2/server.go:69 +0xb8 fp=0xc0000bcf00 sp=0xc0000bceb0 pc=0x102efdb8
github.com/elementsproject/glightning/glightning.(*Plugin).Start(0xc0002e0fc0, 0xc00011a050, 0xc00011a058)
        /home/user/go/pkg/mod/github.com/elementsproject/glightning@v0.0.0-20240224063423-55240d61b52a/glightning/plugin.go:1294 +0x188 fp=0xc0000bcf60 sp=0xc0000bcf00 pc=0x107e1458
github.com/elementsproject/peerswap/clightning.(*ClightningClient).Start(...)
        /home/user/builds/peerswap/clightning/clightning.go:374
main.outer.func1()
        /home/user/builds/peerswap/cmd/peerswap-plugin/main.go:88 +0x50 fp=0xc0000bcfc0 sp=0xc0000bcf60 pc=0x1083c140
runtime.goexit({})
        /usr/lib/go/src/runtime/asm_ppc64x.s:1018 +0x4 fp=0xc0000bcfc0 sp=0xc0000bcfc0 pc=0x1007c204     
created by main.outer in goroutine 1
        /home/user/builds/peerswap/cmd/peerswap-plugin/main.go:87 +0x138

goroutine 20 gp=0xc0001dd6c0 m=nil [select]:
runtime.gopark(0xc0000c1d68?, 0x2?, 0x0?, 0x0?, 0xc0000c1d3c?)
        /usr/lib/go/src/runtime/proc.go:402 +0x114 fp=0xc0000c1ba8 sp=0xc0000c1b78 pc=0x10045144         
runtime.selectgo(0xc0000c1d68, 0xc0000c1d38, 0x7fff8bb40108?, 0x0, 0x0?, 0x1)
        /usr/lib/go/src/runtime/select.go:327 +0x718 fp=0xc0000c1ce8 sp=0xc0000c1ba8 pc=0x10058fd8       
io.(*pipe).read(0xc000130c60, {0xc0000ee000, 0x1000, 0x1?})
        /usr/lib/go/src/io/pipe.go:57 +0xbc fp=0xc0000c1d88 sp=0xc0000c1ce8 pc=0x100e718c                
io.(*PipeReader).Read(0x0?, {0xc0000ee000?, 0x0?, 0x0?})
        /usr/lib/go/src/io/pipe.go:134 +0x4c fp=0xc0000c1dc8 sp=0xc0000c1d88 pc=0x100e791c               
bufio.(*Scanner).Scan(0xc0000c1ef8)
        /usr/lib/go/src/bufio/scan.go:219 +0x9d4 fp=0xc0000c1eb0 sp=0xc0000c1dc8 pc=0x102695f4           
github.com/elementsproject/glightning/glightning.(*Plugin).checkForMonkeyPatch.func1({0x10c803c8?, 0xc000130c60?}, 0xc0002e0fc0)
        /home/user/go/pkg/mod/github.com/elementsproject/glightning@v0.0.0-20240224063423-55240d61b52a/glightning/plugin.go:1327 +0xbc fp=0xc0000c1f88 sp=0xc0000c1eb0 pc=0x107e4d7c
github.com/elementsproject/glightning/glightning.(*Plugin).checkForMonkeyPatch.gowrap1()                 
        /home/user/go/pkg/mod/github.com/elementsproject/glightning@v0.0.0-20240224063423-55240d61b52a/glightning/plugin.go:1333 +0x60 fp=0xc0000c1fc0 sp=0xc0000c1f88 pc=0x107e17b0
runtime.goexit({})                                                                                                                                                                              20:55:01 [405/1776]
        /usr/lib/go/src/runtime/asm_ppc64x.s:1018 +0x4 fp=0xc0000c1fc0 sp=0xc0000c1fc0 pc=0x1007c204     
created by github.com/elementsproject/glightning/glightning.(*Plugin).checkForMonkeyPatch in goroutine 19
        /home/user/go/pkg/mod/github.com/elementsproject/glightning@v0.0.0-20240224063423-55240d61b52a/glightning/plugin.go:1324 +0x234

goroutine 21 gp=0xc0001dd880 m=nil [chan receive]:
runtime.gopark(0x1092fe40?, 0xc000130060?, 0x0?, 0xf0?, 0xdb?)
        /usr/lib/go/src/runtime/proc.go:402 +0x114 fp=0xc0005a3db0 sp=0xc0005a3d80 pc=0x10045144         
runtime.chanrecv(0xc00010e600, 0xc0005a3f18, 0x1)
        /usr/lib/go/src/runtime/chan.go:583 +0x4f4 fp=0xc0005a3e48 sp=0xc0005a3db0 pc=0x1000c214         
runtime.chanrecv2(0xc0005a3f28?, 0xc0003200e0?)
        /usr/lib/go/src/runtime/chan.go:447 +0x10 fp=0xc0005a3e80 sp=0xc0005a3e48 pc=0x1000bd00          
github.com/elementsproject/glightning/jrpc2.(*Server).setupWriteQueue(0xc000354630, {0x10c7f0a8, 0xc00011a058?})
        /home/user/go/pkg/mod/github.com/elementsproject/glightning@v0.0.0-20240224063423-55240d61b52a/jrpc2/server.go:132 +0x108 fp=0xc0005a3f88 sp=0xc0005a3e80 pc=0x102f04e8
github.com/elementsproject/glightning/jrpc2.(*Server).StartUp.gowrap1()
        /home/user/go/pkg/mod/github.com/elementsproject/glightning@v0.0.0-20240224063423-55240d61b52a/jrpc2/server.go:68 +0x54 fp=0xc0005a3fc0 sp=0xc0005a3f88 pc=0x102efe24
runtime.goexit({})
        /usr/lib/go/src/runtime/asm_ppc64x.s:1018 +0x4 fp=0xc0005a3fc0 sp=0xc0005a3fc0 pc=0x1007c204     
created by github.com/elementsproject/glightning/jrpc2.(*Server).StartUp in goroutine 19                 
        /home/user/go/pkg/mod/github.com/elementsproject/glightning@v0.0.0-20240224063423-55240d61b52a/jrpc2/server.go:68 +0xa4

goroutine 26 gp=0xc0001dda40 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/lib/go/src/runtime/proc.go:402 +0x114 fp=0xc0000a9f08 sp=0xc0000a9ed8 pc=0x10045144         
runtime.gcBgMarkWorker()
        /usr/lib/go/src/runtime/mgc.go:1310 +0xf8 fp=0xc0000a9fc0 sp=0xc0000a9f08 pc=0x10024468          
runtime.goexit({})
        /usr/lib/go/src/runtime/asm_ppc64x.s:1018 +0x4 fp=0xc0000a9fc0 sp=0xc0000a9fc0 pc=0x1007c204     
created by runtime.gcBgMarkStartWorkers in goroutine 24
        /usr/lib/go/src/runtime/mgc.go:1234 +0x30

goroutine 23 gp=0xc0001ddc00 m=nil [chan receive]:
runtime.gopark(0x1092fe40?, 0xc00011e5d0?, 0x48?, 0x5d?, 0x10267e58?)
        /usr/lib/go/src/runtime/proc.go:402 +0x114 fp=0xc0000bfc30 sp=0xc0000bfc00 pc=0x10045144         
runtime.chanrecv(0xc00010e660, 0xc0005a3da0, 0x1)
        /usr/lib/go/src/runtime/chan.go:583 +0x4f4 fp=0xc0000bfcc8 sp=0xc0000bfc30 pc=0x1000c214         
runtime.chanrecv2(0xc0005a3db8?, 0xc000615000?)
        /usr/lib/go/src/runtime/chan.go:447 +0x10 fp=0xc0000bfd00 sp=0xc0000bfcc8 pc=0x1000bd00          
github.com/elementsproject/glightning/jrpc2.(*Client).setupWriteQueue(0xc00036e440, {0x10c81680, 0xc00011a4a8?})
        /home/user/go/pkg/mod/github.com/elementsproject/glightning@v0.0.0-20240224063423-55240d61b52a/jrpc2/client.go:91 +0x134 fp=0xc0000bfe20 sp=0xc0000bfd00 pc=0x102eb214
github.com/elementsproject/glightning/jrpc2.(*Client).SocketStart(0xc00036e440, {0xc000406000, 0x32}, 0xc00010e8a0)
        /home/user/go/pkg/mod/github.com/elementsproject/glightning@v0.0.0-20240224063423-55240d61b52a/jrpc2/client.go:64 +0x25c fp=0xc0000bfef8 sp=0xc0000bfe20 pc=0x102eaccc
github.com/elementsproject/glightning/glightning.(*Lightning).StartUp.func1(0xc0002ea710, {0xc0002d9820?, 0x0?}, {0xc000139830?, 0xc0003f0000?}, 0xc00010e8a0, 0xc00010e900)
        /home/user/go/pkg/mod/github.com/elementsproject/glightning@v0.0.0-20240224063423-55240d61b52a/glightning/lightning.go:123 +0xb8 fp=0xc0000bff68 sp=0xc0000bfef8 pc=0x107e4c88
github.com/elementsproject/glightning/glightning.(*Lightning).StartUp.gowrap1()
        /home/user/go/pkg/mod/github.com/elementsproject/glightning@v0.0.0-20240224063423-55240d61b52a/glightning/lightning.go:127 +0x70 fp=0xc0000bffc0 sp=0xc0000bff68 pc=0x107d4640
runtime.goexit({})
        /usr/lib/go/src/runtime/asm_ppc64x.s:1018 +0x4 fp=0xc0000bffc0 sp=0xc0000bffc0 pc=0x1007c204     
created by github.com/elementsproject/glightning/glightning.(*Lightning).StartUp in goroutine 34         
        /home/user/go/pkg/mod/github.com/elementsproject/glightning@v0.0.0-20240224063423-55240d61b52a/glightning/lightning.go:122 +0x128

goroutine 24 gp=0xc0001dddc0 m=nil [IO wait]:
runtime.gopark(0x1008edbc?, 0xc0005a9ae0?, 0x87?, 0x0?, 0x1001d82c?)                                                                                                                            20:55:01 [353/1776]
        /usr/lib/go/src/runtime/proc.go:402 +0x114 fp=0xc0005a9ad8 sp=0xc0005a9aa8 pc=0x10045144         
runtime.netpollblock(0x7fff45004db8?, 0x0?, 0x0?)
        /usr/lib/go/src/runtime/netpoll.go:573 +0x18c fp=0xc0005a9b20 sp=0xc0005a9ad8 pc=0x1003d81c      
internal/poll.runtime_pollWait(0x7fff44fd8e70, 0x72) 
        /usr/lib/go/src/runtime/netpoll.go:345 +0xf4 fp=0xc0005a9b60 sp=0xc0005a9b20 pc=0x100765d4       
internal/poll.(*pollDesc).wait(0xc000191380?, 0xc000266002?, 0x0)
        /usr/lib/go/src/internal/poll/fd_poll_runtime.go:84 +0x4c fp=0xc0005a9bb8 sp=0xc0005a9b60 pc=0x100eb93c
internal/poll.(*pollDesc).waitRead(...)
        /usr/lib/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc000191380, {0xc000266002, 0x7dfe, 0x7dfe})
        /usr/lib/go/src/internal/poll/fd_unix.go:164 +0x278 fp=0xc0005a9c60 sp=0xc0005a9bb8 pc=0x100ece68
net.(*netFD).Read(0xc000191380, {0xc000266002?, 0x10?, 0x10?})
        /usr/lib/go/src/net/fd_posix.go:55 +0x50 fp=0xc0005a9cc8 sp=0xc0005a9c60 pc=0x101dd1c0           
net.(*conn).Read(0xc00011a4a8, {0xc000266002?, 0x3f004000a8b7a0?, 0x1001354c?})
        /usr/lib/go/src/net/net.go:185 +0x64 fp=0xc0005a9d30 sp=0xc0005a9cc8 pc=0x101ee144               
net.(*UnixConn).Read(0xc0002df1d0?, {0xc000266002?, 0x7fff44fd1868?, 0x50?})
        <autogenerated>:1 +0x70 fp=0xc0005a9d70 sp=0xc0005a9d30 pc=0x10201ef0
encoding/json.(*Decoder).refill(0xc0002df180)
        /usr/lib/go/src/encoding/json/stream.go:165 +0x180 fp=0xc0005a9dd0 sp=0xc0005a9d70 pc=0x1013ba10
encoding/json.(*Decoder).readValue(0xc0002df180)
        /usr/lib/go/src/encoding/json/stream.go:140 +0x90 fp=0xc0005a9e30 sp=0xc0005a9dd0 pc=0x1013b5e0  
encoding/json.(*Decoder).Decode(0xc0002df180, {0x1094a260, 0xc00011f140})
        /usr/lib/go/src/encoding/json/stream.go:63 +0x84 fp=0xc0005a9e88 sp=0xc0005a9e30 pc=0x1013b394   
github.com/elementsproject/glightning/jrpc2.(*Client).readQueue(0xc00036e440, {0x10c816e0, 0xc00011a4a8})
        /home/user/go/pkg/mod/github.com/elementsproject/glightning@v0.0.0-20240224063423-55240d61b52a/jrpc2/client.go:113 +0xe0 fp=0xc0005a9f28 sp=0xc0005a9e88 pc=0x102eb5e0
github.com/elementsproject/glightning/jrpc2.(*Client).SocketStart.func1({0x10c8f780?, 0xc00011a4a8}, 0xc00010e900?)
        /home/user/go/pkg/mod/github.com/elementsproject/glightning@v0.0.0-20240224063423-55240d61b52a/jrpc2/client.go:62 +0xb4 fp=0xc0005a9f88 sp=0xc0005a9f28 pc=0x102eae84
github.com/elementsproject/glightning/jrpc2.(*Client).SocketStart.gowrap2()
        /home/user/go/pkg/mod/github.com/elementsproject/glightning@v0.0.0-20240224063423-55240d61b52a/jrpc2/client.go:63 +0x60 fp=0xc0005a9fc0 sp=0xc0005a9f88 pc=0x102eadc0
runtime.goexit({})
        /usr/lib/go/src/runtime/asm_ppc64x.s:1018 +0x4 fp=0xc0005a9fc0 sp=0xc0005a9fc0 pc=0x1007c204     
created by github.com/elementsproject/glightning/jrpc2.(*Client).SocketStart in goroutine 23             
        /home/user/go/pkg/mod/github.com/elementsproject/glightning@v0.0.0-20240224063423-55240d61b52a/jrpc2/client.go:58 +0x184

goroutine 27 gp=0xc00041c000 m=nil [GC worker (idle)]:
runtime.gopark(0x1?, 0x313?, 0xd8?, 0xc9?, 0x10aad67e?)
        /usr/lib/go/src/runtime/proc.go:402 +0x114 fp=0xc0000a9708 sp=0xc0000a96d8 pc=0x10045144         
runtime.gcBgMarkWorker()
        /usr/lib/go/src/runtime/mgc.go:1310 +0xf8 fp=0xc0000a97c0 sp=0xc0000a9708 pc=0x10024468          
runtime.goexit({})
        /usr/lib/go/src/runtime/asm_ppc64x.s:1018 +0x4 fp=0xc0000a97c0 sp=0xc0000a97c0 pc=0x1007c204     
created by runtime.gcBgMarkStartWorkers in goroutine 24
        /usr/lib/go/src/runtime/mgc.go:1234 +0x30

goroutine 7 gp=0xc000003c00 m=nil [GC worker (idle)]:
runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x10aad680?)
        /usr/lib/go/src/runtime/proc.go:402 +0x114 fp=0xc0000a8f08 sp=0xc0000a8ed8 pc=0x10045144         
runtime.gcBgMarkWorker()
        /usr/lib/go/src/runtime/mgc.go:1310 +0xf8 fp=0xc0000a8fc0 sp=0xc0000a8f08 pc=0x10024468          
runtime.goexit({})
        /usr/lib/go/src/runtime/asm_ppc64x.s:1018 +0x4 fp=0xc0000a8fc0 sp=0xc0000a8fc0 pc=0x1007c204     
created by runtime.gcBgMarkStartWorkers in goroutine 24                                                                                                                                         20:55:01 [301/1776]
        /usr/lib/go/src/runtime/mgc.go:1234 +0x30

goroutine 35 gp=0xc0003f6000 m=nil [GC worker (idle)]:
runtime.gopark(0x59?, 0x59?, 0x0?, 0x0?, 0x0?)
        /usr/lib/go/src/runtime/proc.go:402 +0x114 fp=0xc0000a8708 sp=0xc0000a86d8 pc=0x10045144         
runtime.gcBgMarkWorker()
        /usr/lib/go/src/runtime/mgc.go:1310 +0xf8 fp=0xc0000a87c0 sp=0xc0000a8708 pc=0x10024468          
runtime.goexit({})
        /usr/lib/go/src/runtime/asm_ppc64x.s:1018 +0x4 fp=0xc0000a87c0 sp=0xc0000a87c0 pc=0x1007c204     
created by runtime.gcBgMarkStartWorkers in goroutine 24
        /usr/lib/go/src/runtime/mgc.go:1234 +0x30

goroutine 36 gp=0xc0003f6540 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/lib/go/src/runtime/proc.go:402 +0x114 fp=0xc00050c708 sp=0xc00050c6d8 pc=0x10045144         
runtime.gcBgMarkWorker()
        /usr/lib/go/src/runtime/mgc.go:1310 +0xf8 fp=0xc00050c7c0 sp=0xc00050c708 pc=0x10024468          
runtime.goexit({})
        /usr/lib/go/src/runtime/asm_ppc64x.s:1018 +0x4 fp=0xc00050c7c0 sp=0xc00050c7c0 pc=0x1007c204     
created by runtime.gcBgMarkStartWorkers in goroutine 24
        /usr/lib/go/src/runtime/mgc.go:1234 +0x30

goroutine 8 gp=0xc000003dc0 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/lib/go/src/runtime/proc.go:402 +0x114 fp=0xc0000ae708 sp=0xc0000ae6d8 pc=0x10045144         
runtime.gcBgMarkWorker()
        /usr/lib/go/src/runtime/mgc.go:1310 +0xf8 fp=0xc0000ae7c0 sp=0xc0000ae708 pc=0x10024468          
runtime.goexit({})
        /usr/lib/go/src/runtime/asm_ppc64x.s:1018 +0x4 fp=0xc0000ae7c0 sp=0xc0000ae7c0 pc=0x1007c204     
created by runtime.gcBgMarkStartWorkers in goroutine 24
        /usr/lib/go/src/runtime/mgc.go:1234 +0x30

goroutine 37 gp=0xc0003f6700 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/lib/go/src/runtime/proc.go:402 +0x114 fp=0xc00050cf08 sp=0xc00050ced8 pc=0x10045144         
runtime.gcBgMarkWorker()
        /usr/lib/go/src/runtime/mgc.go:1310 +0xf8 fp=0xc00050cfc0 sp=0xc00050cf08 pc=0x10024468          
runtime.goexit({})
        /usr/lib/go/src/runtime/asm_ppc64x.s:1018 +0x4 fp=0xc00050cfc0 sp=0xc00050cfc0 pc=0x1007c204     
created by runtime.gcBgMarkStartWorkers in goroutine 24
        /usr/lib/go/src/runtime/mgc.go:1234 +0x30

goroutine 38 gp=0xc0003f68c0 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/lib/go/src/runtime/proc.go:402 +0x114 fp=0xc00050d708 sp=0xc00050d6d8 pc=0x10045144         
runtime.gcBgMarkWorker()
        /usr/lib/go/src/runtime/mgc.go:1310 +0xf8 fp=0xc00050d7c0 sp=0xc00050d708 pc=0x10024468          
runtime.goexit({})
        /usr/lib/go/src/runtime/asm_ppc64x.s:1018 +0x4 fp=0xc00050d7c0 sp=0xc00050d7c0 pc=0x1007c204     
created by runtime.gcBgMarkStartWorkers in goroutine 24
        /usr/lib/go/src/runtime/mgc.go:1234 +0x30

goroutine 9 gp=0xc0004c2000 m=nil [GC worker (idle)]:                                                                                                                                           20:55:01 [248/1776]
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/lib/go/src/runtime/proc.go:402 +0x114 fp=0xc0000aef08 sp=0xc0000aeed8 pc=0x10045144         
runtime.gcBgMarkWorker()
        /usr/lib/go/src/runtime/mgc.go:1310 +0xf8 fp=0xc0000aefc0 sp=0xc0000aef08 pc=0x10024468          
runtime.goexit({})
        /usr/lib/go/src/runtime/asm_ppc64x.s:1018 +0x4 fp=0xc0000aefc0 sp=0xc0000aefc0 pc=0x1007c204     
created by runtime.gcBgMarkStartWorkers in goroutine 24
        /usr/lib/go/src/runtime/mgc.go:1234 +0x30

goroutine 10 gp=0xc0004c21c0 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/lib/go/src/runtime/proc.go:402 +0x114 fp=0xc0000af708 sp=0xc0000af6d8 pc=0x10045144         
runtime.gcBgMarkWorker()
        /usr/lib/go/src/runtime/mgc.go:1310 +0xf8 fp=0xc0000af7c0 sp=0xc0000af708 pc=0x10024468          
runtime.goexit({})
        /usr/lib/go/src/runtime/asm_ppc64x.s:1018 +0x4 fp=0xc0000af7c0 sp=0xc0000af7c0 pc=0x1007c204     
created by runtime.gcBgMarkStartWorkers in goroutine 24
        /usr/lib/go/src/runtime/mgc.go:1234 +0x30

goroutine 11 gp=0xc0004c2380 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/lib/go/src/runtime/proc.go:402 +0x114 fp=0xc0000aff08 sp=0xc0000afed8 pc=0x10045144         
runtime.gcBgMarkWorker()
        /usr/lib/go/src/runtime/mgc.go:1310 +0xf8 fp=0xc0000affc0 sp=0xc0000aff08 pc=0x10024468          
runtime.goexit({})
        /usr/lib/go/src/runtime/asm_ppc64x.s:1018 +0x4 fp=0xc0000affc0 sp=0xc0000affc0 pc=0x1007c204     
created by runtime.gcBgMarkStartWorkers in goroutine 24
        /usr/lib/go/src/runtime/mgc.go:1234 +0x30

goroutine 39 gp=0xc0003f6a80 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/lib/go/src/runtime/proc.go:402 +0x114 fp=0xc00050df08 sp=0xc00050ded8 pc=0x10045144         
runtime.gcBgMarkWorker()
        /usr/lib/go/src/runtime/mgc.go:1310 +0xf8 fp=0xc00050dfc0 sp=0xc00050df08 pc=0x10024468          
runtime.goexit({})
        /usr/lib/go/src/runtime/asm_ppc64x.s:1018 +0x4 fp=0xc00050dfc0 sp=0xc00050dfc0 pc=0x1007c204     
created by runtime.gcBgMarkStartWorkers in goroutine 24
        /usr/lib/go/src/runtime/mgc.go:1234 +0x30

goroutine 50 gp=0xc000510000 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/lib/go/src/runtime/proc.go:402 +0x114 fp=0xc000508708 sp=0xc0005086d8 pc=0x10045144         
runtime.gcBgMarkWorker()
        /usr/lib/go/src/runtime/mgc.go:1310 +0xf8 fp=0xc0005087c0 sp=0xc000508708 pc=0x10024468          
runtime.goexit({})
        /usr/lib/go/src/runtime/asm_ppc64x.s:1018 +0x4 fp=0xc0005087c0 sp=0xc0005087c0 pc=0x1007c204     
created by runtime.gcBgMarkStartWorkers in goroutine 24
        /usr/lib/go/src/runtime/mgc.go:1234 +0x30

goroutine 40 gp=0xc0003f6c40 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/lib/go/src/runtime/proc.go:402 +0x114 fp=0xc00050e708 sp=0xc00050e6d8 pc=0x10045144                                                                                                20:55:01 [196/1776]
runtime.gcBgMarkWorker()
        /usr/lib/go/src/runtime/mgc.go:1310 +0xf8 fp=0xc00050e7c0 sp=0xc00050e708 pc=0x10024468          
runtime.goexit({})
        /usr/lib/go/src/runtime/asm_ppc64x.s:1018 +0x4 fp=0xc00050e7c0 sp=0xc00050e7c0 pc=0x1007c204     
created by runtime.gcBgMarkStartWorkers in goroutine 24
        /usr/lib/go/src/runtime/mgc.go:1234 +0x30

goroutine 51 gp=0xc000510540 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/lib/go/src/runtime/proc.go:402 +0x114 fp=0xc000508f08 sp=0xc000508ed8 pc=0x10045144         
runtime.gcBgMarkWorker()
        /usr/lib/go/src/runtime/mgc.go:1310 +0xf8 fp=0xc000508fc0 sp=0xc000508f08 pc=0x10024468          
runtime.goexit({})
        /usr/lib/go/src/runtime/asm_ppc64x.s:1018 +0x4 fp=0xc000508fc0 sp=0xc000508fc0 pc=0x1007c204     
created by runtime.gcBgMarkStartWorkers in goroutine 24
        /usr/lib/go/src/runtime/mgc.go:1234 +0x30

goroutine 41 gp=0xc0003f6e00 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/lib/go/src/runtime/proc.go:402 +0x114 fp=0xc00050ef08 sp=0xc00050eed8 pc=0x10045144         
runtime.gcBgMarkWorker()
        /usr/lib/go/src/runtime/mgc.go:1310 +0xf8 fp=0xc00050efc0 sp=0xc00050ef08 pc=0x10024468          
runtime.goexit({})
        /usr/lib/go/src/runtime/asm_ppc64x.s:1018 +0x4 fp=0xc00050efc0 sp=0xc00050efc0 pc=0x1007c204     
created by runtime.gcBgMarkStartWorkers in goroutine 24
        /usr/lib/go/src/runtime/mgc.go:1234 +0x30

goroutine 52 gp=0xc000510700 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/lib/go/src/runtime/proc.go:402 +0x114 fp=0xc000509708 sp=0xc0005096d8 pc=0x10045144         
runtime.gcBgMarkWorker()
        /usr/lib/go/src/runtime/mgc.go:1310 +0xf8 fp=0xc0005097c0 sp=0xc000509708 pc=0x10024468          
runtime.goexit({})
        /usr/lib/go/src/runtime/asm_ppc64x.s:1018 +0x4 fp=0xc0005097c0 sp=0xc0005097c0 pc=0x1007c204     
created by runtime.gcBgMarkStartWorkers in goroutine 24
        /usr/lib/go/src/runtime/mgc.go:1234 +0x30

goroutine 53 gp=0xc0005108c0 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/lib/go/src/runtime/proc.go:402 +0x114 fp=0xc000509f08 sp=0xc000509ed8 pc=0x10045144         
runtime.gcBgMarkWorker()
        /usr/lib/go/src/runtime/mgc.go:1310 +0xf8 fp=0xc000509fc0 sp=0xc000509f08 pc=0x10024468          
runtime.goexit({})
        /usr/lib/go/src/runtime/asm_ppc64x.s:1018 +0x4 fp=0xc000509fc0 sp=0xc000509fc0 pc=0x1007c204     
created by runtime.gcBgMarkStartWorkers in goroutine 24
        /usr/lib/go/src/runtime/mgc.go:1234 +0x30

goroutine 42 gp=0xc0003f6fc0 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/lib/go/src/runtime/proc.go:402 +0x114 fp=0xc00050f708 sp=0xc00050f6d8 pc=0x10045144         
runtime.gcBgMarkWorker()
        /usr/lib/go/src/runtime/mgc.go:1310 +0xf8 fp=0xc00050f7c0 sp=0xc00050f708 pc=0x10024468                                                                                                 20:55:01 [144/1776]
runtime.goexit({})
        /usr/lib/go/src/runtime/asm_ppc64x.s:1018 +0x4 fp=0xc00050f7c0 sp=0xc00050f7c0 pc=0x1007c204     
created by runtime.gcBgMarkStartWorkers in goroutine 24
        /usr/lib/go/src/runtime/mgc.go:1234 +0x30

goroutine 43 gp=0xc0003f7180 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/lib/go/src/runtime/proc.go:402 +0x114 fp=0xc00050ff08 sp=0xc00050fed8 pc=0x10045144         
runtime.gcBgMarkWorker()
        /usr/lib/go/src/runtime/mgc.go:1310 +0xf8 fp=0xc00050ffc0 sp=0xc00050ff08 pc=0x10024468          
runtime.goexit({})
        /usr/lib/go/src/runtime/asm_ppc64x.s:1018 +0x4 fp=0xc00050ffc0 sp=0xc00050ffc0 pc=0x1007c204     
created by runtime.gcBgMarkStartWorkers in goroutine 24
        /usr/lib/go/src/runtime/mgc.go:1234 +0x30

goroutine 28 gp=0xc00041c1c0 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/lib/go/src/runtime/proc.go:402 +0x114 fp=0xc0000aa708 sp=0xc0000aa6d8 pc=0x10045144         
runtime.gcBgMarkWorker()
        /usr/lib/go/src/runtime/mgc.go:1310 +0xf8 fp=0xc0000aa7c0 sp=0xc0000aa708 pc=0x10024468          
runtime.goexit({})
        /usr/lib/go/src/runtime/asm_ppc64x.s:1018 +0x4 fp=0xc0000aa7c0 sp=0xc0000aa7c0 pc=0x1007c204     
created by runtime.gcBgMarkStartWorkers in goroutine 24
        /usr/lib/go/src/runtime/mgc.go:1234 +0x30

goroutine 44 gp=0xc0003f7340 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/lib/go/src/runtime/proc.go:402 +0x114 fp=0xc000584708 sp=0xc0005846d8 pc=0x10045144         
runtime.gcBgMarkWorker()
        /usr/lib/go/src/runtime/mgc.go:1310 +0xf8 fp=0xc0005847c0 sp=0xc000584708 pc=0x10024468          
runtime.goexit({})
        /usr/lib/go/src/runtime/asm_ppc64x.s:1018 +0x4 fp=0xc0005847c0 sp=0xc0005847c0 pc=0x1007c204     
created by runtime.gcBgMarkStartWorkers in goroutine 24
        /usr/lib/go/src/runtime/mgc.go:1234 +0x30

goroutine 12 gp=0xc0004c2540 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/lib/go/src/runtime/proc.go:402 +0x114 fp=0xc000580708 sp=0xc0005806d8 pc=0x10045144         
runtime.gcBgMarkWorker()
        /usr/lib/go/src/runtime/mgc.go:1310 +0xf8 fp=0xc0005807c0 sp=0xc000580708 pc=0x10024468          
runtime.goexit({})
        /usr/lib/go/src/runtime/asm_ppc64x.s:1018 +0x4 fp=0xc0005807c0 sp=0xc0005807c0 pc=0x1007c204     
created by runtime.gcBgMarkStartWorkers in goroutine 24
        /usr/lib/go/src/runtime/mgc.go:1234 +0x30

goroutine 29 gp=0xc00041c380 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/lib/go/src/runtime/proc.go:402 +0x114 fp=0xc0000aaf08 sp=0xc0000aaed8 pc=0x10045144         
runtime.gcBgMarkWorker()
        /usr/lib/go/src/runtime/mgc.go:1310 +0xf8 fp=0xc0000aafc0 sp=0xc0000aaf08 pc=0x10024468          
runtime.goexit({})
        /usr/lib/go/src/runtime/asm_ppc64x.s:1018 +0x4 fp=0xc0000aafc0 sp=0xc0000aafc0 pc=0x1007c204                                                                                             20:55:01 [92/1776]
created by runtime.gcBgMarkStartWorkers in goroutine 24
        /usr/lib/go/src/runtime/mgc.go:1234 +0x30

goroutine 13 gp=0xc0004c2700 m=nil [GC worker (idle)]:
runtime.gopark(0x1133b7a0?, 0x1?, 0x52?, 0xaa?, 0x0?)
        /usr/lib/go/src/runtime/proc.go:402 +0x114 fp=0xc000580f08 sp=0xc000580ed8 pc=0x10045144         
runtime.gcBgMarkWorker()
        /usr/lib/go/src/runtime/mgc.go:1310 +0xf8 fp=0xc000580fc0 sp=0xc000580f08 pc=0x10024468          
runtime.goexit({})
        /usr/lib/go/src/runtime/asm_ppc64x.s:1018 +0x4 fp=0xc000580fc0 sp=0xc000580fc0 pc=0x1007c204     
created by runtime.gcBgMarkStartWorkers in goroutine 24
        /usr/lib/go/src/runtime/mgc.go:1234 +0x30

goroutine 54 gp=0xc000510a80 m=nil [GC worker (idle)]:
runtime.gopark(0x1133b7a0?, 0x1?, 0x8f?, 0xcb?, 0x0?)
        /usr/lib/go/src/runtime/proc.go:402 +0x114 fp=0xc00050a708 sp=0xc00050a6d8 pc=0x10045144         
runtime.gcBgMarkWorker()
        /usr/lib/go/src/runtime/mgc.go:1310 +0xf8 fp=0xc00050a7c0 sp=0xc00050a708 pc=0x10024468          
runtime.goexit({})
        /usr/lib/go/src/runtime/asm_ppc64x.s:1018 +0x4 fp=0xc00050a7c0 sp=0xc00050a7c0 pc=0x1007c204     
created by runtime.gcBgMarkStartWorkers in goroutine 24
        /usr/lib/go/src/runtime/mgc.go:1234 +0x30

goroutine 45 gp=0xc0003f7500 m=nil [GC worker (idle)]:
runtime.gopark(0x58123e12db48?, 0x1?, 0x5f?, 0x9?, 0x0?)
        /usr/lib/go/src/runtime/proc.go:402 +0x114 fp=0xc000584f08 sp=0xc000584ed8 pc=0x10045144         
runtime.gcBgMarkWorker()
        /usr/lib/go/src/runtime/mgc.go:1310 +0xf8 fp=0xc000584fc0 sp=0xc000584f08 pc=0x10024468          
runtime.goexit({})
        /usr/lib/go/src/runtime/asm_ppc64x.s:1018 +0x4 fp=0xc000584fc0 sp=0xc000584fc0 pc=0x1007c204     
created by runtime.gcBgMarkStartWorkers in goroutine 24
        /usr/lib/go/src/runtime/mgc.go:1234 +0x30

goroutine 55 gp=0xc000510c40 m=nil [GC worker (idle)]:
runtime.gopark(0x1133b7a0?, 0x1?, 0xe9?, 0x45?, 0x0?)
        /usr/lib/go/src/runtime/proc.go:402 +0x114 fp=0xc00050af08 sp=0xc00050aed8 pc=0x10045144         
runtime.gcBgMarkWorker()
        /usr/lib/go/src/runtime/mgc.go:1310 +0xf8 fp=0xc00050afc0 sp=0xc00050af08 pc=0x10024468          
runtime.goexit({})
        /usr/lib/go/src/runtime/asm_ppc64x.s:1018 +0x4 fp=0xc00050afc0 sp=0xc00050afc0 pc=0x1007c204     
created by runtime.gcBgMarkStartWorkers in goroutine 24
        /usr/lib/go/src/runtime/mgc.go:1234 +0x30

goroutine 30 gp=0xc00041c540 m=nil [GC worker (idle)]:
runtime.gopark(0x581238f1df9c?, 0x1?, 0xb7?, 0xfb?, 0x0?)
        /usr/lib/go/src/runtime/proc.go:402 +0x114 fp=0xc0000ab708 sp=0xc0000ab6d8 pc=0x10045144         
runtime.gcBgMarkWorker()
        /usr/lib/go/src/runtime/mgc.go:1310 +0xf8 fp=0xc0000ab7c0 sp=0xc0000ab708 pc=0x10024468          
runtime.goexit({})
        /usr/lib/go/src/runtime/asm_ppc64x.s:1018 +0x4 fp=0xc0000ab7c0 sp=0xc0000ab7c0 pc=0x1007c204     
created by runtime.gcBgMarkStartWorkers in goroutine 24
        /usr/lib/go/src/runtime/mgc.go:1234 +0x30                                                                                                                                                20:55:01 [40/1776]

goroutine 46 gp=0xc0003f76c0 m=nil [GC worker (idle)]:
runtime.gopark(0x1133b7a0?, 0x1?, 0xa5?, 0x13?, 0x0?)
        /usr/lib/go/src/runtime/proc.go:402 +0x114 fp=0xc000585708 sp=0xc0005856d8 pc=0x10045144         
runtime.gcBgMarkWorker()
        /usr/lib/go/src/runtime/mgc.go:1310 +0xf8 fp=0xc0005857c0 sp=0xc000585708 pc=0x10024468          
runtime.goexit({})
        /usr/lib/go/src/runtime/asm_ppc64x.s:1018 +0x4 fp=0xc0005857c0 sp=0xc0005857c0 pc=0x1007c204     
created by runtime.gcBgMarkStartWorkers in goroutine 24
        /usr/lib/go/src/runtime/mgc.go:1234 +0x30

goroutine 47 gp=0xc0003f7880 m=nil [GC worker (idle)]:
runtime.gopark(0x581238f1fcaa?, 0x1?, 0x18?, 0x86?, 0x0?)
        /usr/lib/go/src/runtime/proc.go:402 +0x114 fp=0xc000585f08 sp=0xc000585ed8 pc=0x10045144         
runtime.gcBgMarkWorker()
        /usr/lib/go/src/runtime/mgc.go:1310 +0xf8 fp=0xc000585fc0 sp=0xc000585f08 pc=0x10024468          
runtime.goexit({})
        /usr/lib/go/src/runtime/asm_ppc64x.s:1018 +0x4 fp=0xc000585fc0 sp=0xc000585fc0 pc=0x1007c204     
created by runtime.gcBgMarkStartWorkers in goroutine 24
        /usr/lib/go/src/runtime/mgc.go:1234 +0x30

goroutine 56 gp=0xc000510e00 m=nil [GC worker (idle)]:
runtime.gopark(0x1133b7a0?, 0x1?, 0x88?, 0x4a?, 0x0?)
        /usr/lib/go/src/runtime/proc.go:402 +0x114 fp=0xc00050b708 sp=0xc00050b6d8 pc=0x10045144         
runtime.gcBgMarkWorker()
        /usr/lib/go/src/runtime/mgc.go:1310 +0xf8 fp=0xc00050b7c0 sp=0xc00050b708 pc=0x10024468          
runtime.goexit({})
        /usr/lib/go/src/runtime/asm_ppc64x.s:1018 +0x4 fp=0xc00050b7c0 sp=0xc00050b7c0 pc=0x1007c204     
created by runtime.gcBgMarkStartWorkers in goroutine 24
        /usr/lib/go/src/runtime/mgc.go:1234 +0x30

goroutine 32 gp=0xc00041c8c0 m=nil [select]:
runtime.gopark(0xc0004ebf60?, 0x2?, 0x0?, 0x0?, 0xc0004ebee4?)
        /usr/lib/go/src/runtime/proc.go:402 +0x114 fp=0xc0004ebd48 sp=0xc0004ebd18 pc=0x10045144         
runtime.selectgo(0xc0004ebf60, 0xc0004ebee0, 0x0?, 0x0, 0x0?, 0x1)
        /usr/lib/go/src/runtime/select.go:327 +0x718 fp=0xc0004ebe88 sp=0xc0004ebd48 pc=0x10058fd8       
github.com/elementsproject/peerswap/txwatcher.(*BlockchainRpcTxWatcher).StartBlockWatcher(0xc000186310)  
        /home/user/builds/peerswap/txwatcher/rpctxwatcher.go:129 +0x100 fp=0xc0004ebf98 sp=0xc0004ebe88 pc=0x10839500
github.com/elementsproject/peerswap/txwatcher.(*BlockchainRpcTxWatcher).StartWatchingTxs.gowrap1()       
        /home/user/builds/peerswap/txwatcher/rpctxwatcher.go:97 +0x4c fp=0xc0004ebfc0 sp=0xc0004ebf98 pc=0x108393ec
runtime.goexit({})
        /usr/lib/go/src/runtime/asm_ppc64x.s:1018 +0x4 fp=0xc0004ebfc0 sp=0xc0004ebfc0 pc=0x1007c204     
created by github.com/elementsproject/peerswap/txwatcher.(*BlockchainRpcTxWatcher).StartWatchingTxs in goroutine 1
        /home/user/builds/peerswap/txwatcher/rpctxwatcher.go:97 +0x80

goroutine 33 gp=0xc00041ca80 m=nil [sleep]:
runtime.gopark(0x58123e650e46?, 0xc0005bcde0?, 0x0?, 0x0?, 0x0?)
        /usr/lib/go/src/runtime/proc.go:402 +0x114 fp=0xc0004ec640 sp=0xc0004ec610 pc=0x10045144         
time.Sleep(0x5f5e100)
        /usr/lib/go/src/runtime/time.go:195 +0x11c fp=0xc0004ec6a0 sp=0xc0004ec640 pc=0x10078e1c         
github.com/elementsproject/peerswap/txwatcher.(*BlockchainRpcTxWatcher).StartWatchingTxs.func1()         
        /home/user/builds/peerswap/txwatcher/rpctxwatcher.go:113 +0x100 fp=0xc0004ec7a0 sp=0xc0004ec6a0 pc=0x108391a0
github.com/elementsproject/peerswap/txwatcher.(*BlockchainRpcTxWatcher).StartWatchingTxs.gowrap2()       
        /home/user/builds/peerswap/txwatcher/rpctxwatcher.go:116 +0x54 fp=0xc0004ec7c0 sp=0xc0004ec7a0 pc=0x10839084
runtime.goexit({})
        /usr/lib/go/src/runtime/asm_ppc64x.s:1018 +0x4 fp=0xc0004ec7c0 sp=0xc0004ec7c0 pc=0x1007c204     
created by github.com/elementsproject/peerswap/txwatcher.(*BlockchainRpcTxWatcher).StartWatchingTxs in goroutine 1
        /home/user/builds/peerswap/txwatcher/rpctxwatcher.go:98 +0x110

goroutine 150 gp=0xc000683c00 m=nil [select]:
runtime.gopark(0xc0003ec780?, 0x2?, 0x0?, 0x0?, 0xc0003ec774?)
        /usr/lib/go/src/runtime/proc.go:402 +0x114 fp=0xc0003ec5e0 sp=0xc0003ec5b0 pc=0x10045144         
runtime.selectgo(0xc0003ec780, 0xc0003ec770, 0x0?, 0x0, 0x0?, 0x1)
        /usr/lib/go/src/runtime/select.go:327 +0x718 fp=0xc0003ec720 sp=0xc0003ec5e0 pc=0x10058fd8       
github.com/elementsproject/peerswap/poll.(*Service).Start.func1()
        /home/user/builds/peerswap/poll/service.go:100 +0xd0 fp=0xc0003ec7c0 sp=0xc0003ec720 pc=0x107ec600
runtime.goexit({})
        /usr/lib/go/src/runtime/asm_ppc64x.s:1018 +0x4 fp=0xc0003ec7c0 sp=0xc0003ec7c0 pc=0x1007c204     
created by github.com/elementsproject/peerswap/poll.(*Service).Start in goroutine 1                      
        /home/user/builds/peerswap/poll/service.go:98 +0x74

r0   0xfffffffffffcbcc0 r1   0x7fff44e8f330
r2   0x11237f00 r3   0x7fff44a40028
r4   0x7fff44e8f3b8     r5   0x0
r6   0xf1       r7   0x10846f50
r8   0x10846f00 r9   0x0
r10  0x34298    r11  0x0
r12  0x7fff44e8f2b0     r13  0x7fff44e96bd0
r14  0xc0000b01d0       r15  0x0
r16  0x0        r17  0xc0000b0000
r18  0x3a       r19  0x3b
r20  0x3f0      r21  0x3b
r22  0xc0000483a0       r23  0xc00004b7d0
r24  0xe0       r25  0xf0
r26  0x100      r27  0x110
r28  0x5c9f08c484979    r29  0x2cc47d561e9b3
r30  0x7fff44a40020     r31  0x301
pc   0x10850870 ctr  0x0
link 0x10852bfc xer  0x20040000
ccr  0x44820284 trap 0x300
tiero commented 2 months ago

Can you share the snippet of code that is causing the crash? ie. /home/user/builds/peerswap/poll/service.go:98

grubles commented 2 months ago

Looks like that is referring to this function:

// Start the poll message loop and send the poll
// messages on every tick.
func (s *Service) Start() {
    // Request fresh polls from all peers on startup
    s.RequestAllPeerPolls()
    // Start poll loop
    go func() {
        for {
            select {
            case now := <-s.clock.C:
                // remove unseen
                s.store.RemoveUnseen(now, s.removeDuration)
                // poll
                s.PollAllPeers()
            case <-s.ctx.Done():
                return
            }
        }
    }()
}

For rpctxwatcher.go:209:

func (l *BlockchainRpcTxWatcher) AddWaitForConfirmationTx(swapId, txId string, vout, startingBlockheight uint32, _ []byte) {
    log.Infof("adding tx watcher for %s", swapId)
    ctx, cancel := context.WithCancel(context.Background())
    newBlock := make(chan uint32)
    info := observerInfo{
        cancel:    cancel,
        blockChan: newBlock,
    }
    go l.observationLoop(ctx, swapId, txId, vout, startingBlockheight, l.csv/2, newBlock)
    l.Lock()
    defer l.Unlock()
    l.observerLoopList[swapId] = info

    // Kick off first run manually, after that is only invoked on new blocks.
    height, _ := l.blockchain.GetBlockHeight()
    newBlock <- uint32(height)
}
grubles commented 2 months ago

Maybe relevant? make test prints this error repeatedly:

$ pwd
/home/user/builds/peerswap

$ make test         
PAYMENT_RETRY_TIME=5 go test -tags dev -tags fast_test -race -timeout=10m -v ./...
# github.com/elementsproject/peerswap/log.test 
/usr/lib/go/pkg/tool/linux_ppc64le/link: running gcc failed: exit status 1
/usr/lib/gcc/powerpc64le-alpine-linux-musl/13.2.1/../../../../powerpc64le-alpine-linux-musl/bin/ld: /tm
p/go-link-3774122030/000002.o: in function `__sanitizer::WriteOneLineToSyslog(char const*)':
gotsan.cpp:(.text+0x81a8): undefined reference to `__syslog_chk'
collect2: error: ld returned 1 exit status
grubles commented 2 months ago

Anything else I can provide to help debug this?