monitoring bot stopped running suddenly

[hossnys]

Describe the bug

after updating monitoring bot to latest version (v0.11.9) and running it , it stopped working

logs :

2:40PM DBG monitoring stellar bridges
2:40PM DBG connecting url=wss://tfchain.dev.grid.tf/ws
2023/12/18 14:40:55 Connecting to wss://tfchain.dev.grid.tf/ws...
2:40PM DBG get balance for 5CqRchiLPvqSA7uowfR32ZqgMPFBq55H8hWoz24B3RokuVj1
panic: strkey is 0 bytes long; minimum valid length is 5

goroutine 1 [running]:
github.com/stellar/go/keypair.MustParseFull({0x0?, 0x65?})
        /home/runner/go/pkg/mod/github.com/stellar/go@v0.0.0-20231121172327-69266de4154f/keypair/main.go:136 +0x35
github.com/threefoldtech/tfgrid-sdk-go/monitoring-bot/internal.(*Monitor).sendToTfChain(0x4024000000000000?, 0x401c000000000000?, {0xd0c370, 0xc0006a4080}, {0xb2d616, 0x6})
        /home/runner/work/tfgrid-sdk-go/tfgrid-sdk-go/monitoring-bot/internal/bridge.go:128 +0x211
github.com/threefoldtech/tfgrid-sdk-go/monitoring-bot/internal.(*Monitor).monitorBridge.(*Monitor).bridgeTXWrapper.func1(0xc00029c550?, {0xd0c370, 0xc0006a4080}, {0xb2d616?, 0x0?})
        /home/runner/work/tfgrid-sdk-go/tfgrid-sdk-go/monitoring-bot/internal/bridge.go:92 +0x3c
github.com/threefoldtech/tfgrid-sdk-go/monitoring-bot/internal.(*Monitor).monitorBridge(0xc0004d19e0, {0xb2d616, 0x6})
        /home/runner/work/tfgrid-sdk-go/tfgrid-sdk-go/monitoring-bot/internal/bridge.go:72 +0x36a
github.com/threefoldtech/tfgrid-sdk-go/monitoring-bot/internal.(*Monitor).monitorBridges(0xc0007fc070?)
        /home/runner/work/tfgrid-sdk-go/tfgrid-sdk-go/monitoring-bot/internal/bridge.go:33 +0x22c
github.com/threefoldtech/tfgrid-sdk-go/monitoring-bot/internal.(*Monitor).Start(0xc0004d19e0)
        /home/runner/work/tfgrid-sdk-go/tfgrid-sdk-go/monitoring-bot/internal/monitor.go:203 +0x1b2
github.com/threefoldtech/tfgrid-sdk-go/monitoring-bot/cmd.glob..func1(0xc0002d4100?, {0xb2ce95?, 0x4?, 0xb2ce99?})
        /home/runner/work/tfgrid-sdk-go/tfgrid-sdk-go/monitoring-bot/cmd/grid_monitor.go:67 +0x305
github.com/spf13/cobra.(*Command).execute(0x1142840, {0xc0000220b0, 0x4, 0x4})
        /home/runner/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:987 +0xaa3
github.com/spf13/cobra.(*Command).ExecuteC(0x1142840)
        /home/runner/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:1115 +0x3ff
github.com/spf13/cobra.(*Command).Execute(...)
        /home/runner/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:1039
github.com/threefoldtech/tfgrid-sdk-go/monitoring-bot/cmd.Execute()
        /home/runner/work/tfgrid-sdk-go/tfgrid-sdk-go/monitoring-bot/cmd/grid_monitor.go:74 +0x1a
main.main()
        /home/runner/work/tfgrid-sdk-go/tfgrid-sdk-go/monitoring-bot/main.go:21 +0xf

[rawdaGastan]

we updated the README, you didn't provide stellar configs

• https://github.com/threefoldtech/tfgrid-sdk-go/blob/development/monitoring-bot/README.md

[hossnys]

added the missing stellar configs , but after that the bot panic and throw this error

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x4 pc=0x85dfdc]

goroutine 220 [running]:
github.com/centrifuge/go-substrate-rpc-client/v4/types.(*Metadata).FindStorageEntryMetadata(0xc0004a4b40?, {0xb3036e?, 0x12?}, {0xb2d38d?, 0x0?})
        /home/runner/go/pkg/mod/github.com/centrifuge/go-substrate-rpc-client/v4@v4.0.12/types/metadata.go:298 +0x1c
github.com/centrifuge/go-substrate-rpc-client/v4/types.CreateStorageKey(0xa1e9a0?, {0xb3036e, 0xc}, {0xb2d38d, 0x5}, {0xc0004a4b40, 0x2, 0x2})
        /home/runner/go/pkg/mod/github.com/centrifuge/go-substrate-rpc-client/v4@v4.0.12/types/storage_key.go:76 +0x11b
github.com/threefoldtech/tfchain/clients/tfchain-client-go.(*Substrate).GetTwin(0xc00040a140?, 0x1ce764?)
        /home/runner/go/pkg/mod/github.com/threefoldtech/tfchain/clients/tfchain-client-go@v0.0.0-20231129121046-56ffc1f8f5c6/twin.go:106 +0xd4
github.com/threefoldtech/tfgrid-sdk-go/rmb-sdk-go/peer.(*twinDB).Get(0xc0005260a0, 0x15)
        /home/runner/work/tfgrid-sdk-go/tfgrid-sdk-go/rmb-sdk-go/peer/twindb.go:45 +0x1c7
github.com/threefoldtech/tfgrid-sdk-go/rmb-sdk-go/peer.VerifySignature({0xd07540?, 0xc0005260a0?}, 0xc0003f0bb0)
        /home/runner/work/tfgrid-sdk-go/tfgrid-sdk-go/rmb-sdk-go/peer/sig.go:97 +0x47
github.com/threefoldtech/tfgrid-sdk-go/rmb-sdk-go/peer.Peer.handleIncoming({0xc0006f8240, {0xd0c370, 0xc00040a0c0}, {0xd07540, 0xc0005260a0}, 0xc00010c380, 0xc0003c0120, 0xc0003c0300, 0xc000526080}, 0xc0003f0bb0)
        /home/runner/work/tfgrid-sdk-go/tfgrid-sdk-go/rmb-sdk-go/peer/peer.go:178 +0xf5
github.com/threefoldtech/tfgrid-sdk-go/rmb-sdk-go/peer.(*Peer).process(0xc0007d00f0, {0xd09320, 0x1179920})
        /home/runner/work/tfgrid-sdk-go/tfgrid-sdk-go/rmb-sdk-go/peer/peer.go:223 +0xd6
created by github.com/threefoldtech/tfgrid-sdk-go/rmb-sdk-go/peer.NewPeer in goroutine 1
        /home/runner/work/tfgrid-sdk-go/tfgrid-sdk-go/rmb-sdk-go/peer/peer.go:159 +0x805

• also the message it sends look like this , and wallets with balance under threshold on testnet is not reported

[rawdaGastan]

It will be fixed through the in the new release:

• https://github.com/threefoldtech/tf_operations/issues/2106

[hossnys]

verified

[hossnys]

• after running latest version of monitoring_bot i got this panic error after 4 hours from starting it .

  
  panic: runtime error: invalid memory address or nil pointer dereference
  [signal SIGSEGV: segmentation violation code=0x1 addr=0x4 pc=0x85dbfc]

goroutine 22582 [running]: github.com/centrifuge/go-substrate-rpc-client/v4/types.(Metadata).FindStorageEntryMetadata(0xc000644c00?, {0xb2f5f2?, 0x12?}, {0xb2c611?, 0x10e6048?}) /home/runner/go/pkg/mod/github.com/centrifuge/go-substrate-rpc-client/v4@v4.0.12/types/metadata.go:298 +0x1c github.com/centrifuge/go-substrate-rpc-client/v4/types.CreateStorageKey(0xa1d9c0?, {0xb2f5f2, 0xc}, {0xb2c611, 0x5}, {0xc000644c00, 0x2, 0x2}) /home/runner/go/pkg/mod/github.com/centrifuge/go-substrate-rpc-client/v4@v4.0.12/types/storage_key.go:76 +0x11b github.com/threefoldtech/tfchain/clients/tfchain-client-go.(Substrate).GetTwin(0xc0008320c0?, 0xa2212c?) /home/runner/go/pkg/mod/github.com/threefoldtech/tfchain/clients/tfchain-client-go@v0.0.0-20231129121046-56ffc1f8f5c6/twin.go:106 +0xd4 github.com/threefoldtech/tfgrid-sdk-go/rmb-sdk-go/peer.(twinDB).Get(0xc0000325a0, 0x15) /home/runner/work/tfgrid-sdk-go/tfgrid-sdk-go/rmb-sdk-go/peer/twindb.go:45 +0x1c7 github.com/threefoldtech/tfgrid-sdk-go/rmb-sdk-go/peer.VerifySignature({0xce39a8?, 0xc0000325a0?}, 0xc000576fd0) /home/runner/work/tfgrid-sdk-go/tfgrid-sdk-go/rmb-sdk-go/peer/sig.go:97 +0x47 github.com/threefoldtech/tfgrid-sdk-go/rmb-sdk-go/peer.Peer.handleIncoming({0xc000630240, {0xce8810, 0xc000832080}, {0xce39a8, 0xc0000325a0}, 0xc000306180, 0xc00042a360, 0xc00042a480, 0xc000032590, {0xce4ec8, ...}}, ...) /home/runner/work/tfgrid-sdk-go/tfgrid-sdk-go/rmb-sdk-go/peer/peer.go:232 +0xb7 github.com/threefoldtech/tfgrid-sdk-go/rmb-sdk-go/peer.(Peer).process(0xc000bd66c0, {0xce5788, 0x1154600}) /home/runner/work/tfgrid-sdk-go/tfgrid-sdk-go/rmb-sdk-go/peer/peer.go:277 +0xc5 created by github.com/threefoldtech/tfgrid-sdk-go/rmb-sdk-go/peer.NewPeer in goroutine 1 /home/runner/work/tfgrid-sdk-go/tfgrid-sdk-go/rmb-sdk-go/peer/peer.go:164 +0x89e

[rawdaGastan]

@hossnys I think you can try the new release v0.13.0

[hossnys]

@rawdaGastan will do

[hossnys]

got panic also with this version v0.13.0

2023/12/27 15:28:37 Connecting to wss://tfchain.grid.tf/ws...
3:28PM ERR failed to read message error="websocket: close 1006 (abnormal closure): unexpected EOF"
3:28PM ERR failed to read message error="websocket: close 1006 (abnormal closure): unexpected EOF"
2023/12/27 15:28:37 Connecting to wss://tfchain.grid.tf:443...
2023/12/27 15:28:38 Connecting to wss://tfchain.grid.tf/ws...
3:28PM ERR failed to read message error="websocket: close 1006 (abnormal closure): unexpected EOF"
3:28PM ERR failed to read message error="websocket: close 1006 (abnormal closure): unexpected EOF"
2023/12/27 15:28:38 Connecting to wss://tfchain.dev.grid.tf/ws...
3:28PM INF starting peer session=monbot-2031321 twin=6
2023/12/27 15:28:39 Connecting to wss://tfchain.dev.grid.tf:443...
3:28PM ERR failed to read message error="websocket: close 1006 (abnormal closure): unexpected EOF"
2023/12/27 15:28:39 Connecting to wss://tfchain.dev.grid.tf/ws...
3:28PM ERR check node 55 failed error="rmb version call in devnet failed using node twin 273 with node ID 55: relay info is not set for this twin"
2023/12/27 15:28:39 Connecting to wss://tfchain.dev.grid.tf:443...
3:28PM ERR failed to read message error="websocket: close 1006 (abnormal closure): unexpected EOF"
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x4 pc=0x85dbfc]

goroutine 261 [running]:
github.com/centrifuge/go-substrate-rpc-client/v4/types.(*Metadata).FindStorageEntryMetadata(0xc0008941b0?, {0xb2f5f2?, 0x12?}, {0xb2c611?, 0x4196e5?})
        /home/runner/go/pkg/mod/github.com/centrifuge/go-substrate-rpc-client/v4@v4.0.12/types/metadata.go:298 +0x1c
github.com/centrifuge/go-substrate-rpc-client/v4/types.CreateStorageKey(0xa1d9c0?, {0xb2f5f2, 0xc}, {0xb2c611, 0x5}, {0xc0008941b0, 0x2, 0x2})
        /home/runner/go/pkg/mod/github.com/centrifuge/go-substrate-rpc-client/v4@v4.0.12/types/storage_key.go:76 +0x11b
github.com/threefoldtech/tfchain/clients/tfchain-client-go.(*Substrate).GetTwin(0xc0006200c0?, 0x6a6630?)
        /home/runner/go/pkg/mod/github.com/threefoldtech/tfchain/clients/tfchain-client-go@v0.0.0-20231129121046-56ffc1f8f5c6/twin.go:106 +0xd4
github.com/threefoldtech/tfgrid-sdk-go/rmb-sdk-go/peer.(*twinDB).Get(0xc00031a0a0, 0x1348)
        /home/runner/work/tfgrid-sdk-go/tfgrid-sdk-go/rmb-sdk-go/peer/twindb.go:45 +0x1c7
github.com/threefoldtech/tfgrid-sdk-go/rmb-sdk-go/peer.VerifySignature({0xce39a8?, 0xc00031a0a0?}, 0xc0005aa210)
        /home/runner/work/tfgrid-sdk-go/tfgrid-sdk-go/rmb-sdk-go/peer/sig.go:97 +0x47
github.com/threefoldtech/tfgrid-sdk-go/rmb-sdk-go/peer.Peer.handleIncoming({0xc000620040, {0xce8810, 0xc000620080}, {0xce39a8, 0xc00031a0a0}, 0xc00010c380, 0xc00052e000, 0xc00052e120, 0xc00031a090, {0xce4ec8, ...}}, ...)
        /home/runner/work/tfgrid-sdk-go/tfgrid-sdk-go/rmb-sdk-go/peer/peer.go:232 +0xb7
github.com/threefoldtech/tfgrid-sdk-go/rmb-sdk-go/peer.(*Peer).process(0xc0005f2000, {0xce5788, 0x1154600})
        /home/runner/work/tfgrid-sdk-go/tfgrid-sdk-go/rmb-sdk-go/peer/peer.go:277 +0xc5
created by github.com/threefoldtech/tfgrid-sdk-go/rmb-sdk-go/peer.NewPeer in goroutine 1
        /home/runner/work/tfgrid-sdk-go/tfgrid-sdk-go/rmb-sdk-go/peer/peer.go:164 +0x89e

[rawdaGastan]

@hossnys can you try to run the bot without the bridges checks?

[hossnys]

tested it , and didn't panic till now , but it keep sending messages without respecting MINS env variable .

[rawdaGastan]

It is recommended to use any of the latest versions

• https://github.com/threefoldtech/tfgrid-sdk-go/releases/tag/v0.13.13
• https://github.com/threefoldtech/tfgrid-sdk-go/releases/tag/v0.13.14

[hossnys]

updated the bot to version 0.13.13 and disabled bridge monitoring to check the reason of panic .

[hossnys]

seems that bridge monitoring part was the cause of panic as bot works fine after disabling it , this need to be fixed to get bridge monitoring working again.

[rawdaGastan]

@hossnys do you still have some issues with it?

[hossnys]

it's working fine now but without bridge monitoring part

[rawdaGastan]

Did you try the latest version?

[hossnys]

the latest version on mainnet is 0.15.10 , and we was already running it but with disabling bridge monitoring , let me enable it and watch what will happen there .