search

firoorg / firo

The privacy-focused cryptocurrency
https://firo.org
MIT License
720 stars 354 forks source link

masternode: CDeterministicMNList::PoSePunish -- punished MN 7cc86b2c1e31b7e9d8e4a2e79dbbfd835607635af60a8cb82ff6d6a277259e3c #1235

Closed NorseGaud closed 1 year ago

NorseGaud commented 1 year ago

PROTX REG TRANS ID: 7cc86b2c1e31b7e9d8e4a2e79dbbfd835607635af60a8cb82ff6d6a277259e3c

2023-04-15 00:20:47 CDeterministicMNList::PoSePunish -- punished MN 7cc86b2c1e31b7e9d8e4a2e79dbbfd835607635af60a8cb82ff6d6a277259e3c, penalty 0->3114 (max=4719)

debug.log

Sat Apr 15 00:08:28 UTC 2023
 00:08:29 up 182 days, 10:35,  0 users,  load average: 1.43, 0.97, 0.54
              total        used        free      shared  buff/cache   available
Mem:           922M        855M         36M         72K         30M         18M
Swap:          2.0G        1.0G        1.0G
Sat Apr 15 00:14:04 UTC 2023
[
  {
    "height": 659574,
    "type": "llmq_50_60",
    "quorumHash": "8b1a6cfe4d9e7e1ab4d1e7bea03d8b9658e780aa5f77fe9c20c7af315e2dfb7b",
    "minedBlock": "c5a77d8e910c909af410a7a7920470c3dc067a27387f8657468d60466c2209f7",
    "quorumPublicKey": "168c173c4385b8d22375f488b3c8372182e2b9ee6b9dc4d5c46ba062cec8e4c27431cab5f878c1aa7f7c7555bdb5081d",
    "isValidMember": true,
    "memberIndex": 9
  }, 
  {
    "height": 659808,
    "type": "llmq_400_85",
    "quorumHash": "ae45e5698c14b72a0d5c12ed59cba0a79647c0a207a78fa6d840402e442a02a1",
    "minedBlock": "4eb6525f27a7f7df7fee75122f3afbbf65430bb2a28bc26f6aa4decaf3eb739d",
    "quorumPublicKey": "0b809f81048f27fc22276c4ead31b94b1c453a37753db088041c9fd7f389399e7ab5258b6a5e655b11f7f95c0ee8a706",
    "isValidMember": true,
    "memberIndex": 310
  }
]
Sat Apr 15 00:14:09 UTC 2023
 00:14:10 up 182 days, 10:41,  0 users,  load average: 0.29, 0.51, 0.47
              total        used        free      shared  buff/cache   available
Mem:           922M        846M         39M         72K         36M         25M
Swap:          2.0G        1.0G        1.0G

Sat Apr 15 00:21:02 UTC 2023
[
  {
    "height": 659844,
    "type": "llmq_50_60",
    "quorumHash": "958129a3685808f626a2215d0975ac775a7fea5261fe5f3dbe05c743e7183bb5",
    "minedBlock": "f6a9a8507c29f988c1455efe0402d3460a9f019acb79eac16867a1d98f79a9b7",
    "quorumPublicKey": "93acd76302a2b811fda636d7fb5696b345b54d5386a17603f1168fadf00417944032ad4eb1dc15f98f6f62a9f52c80df",
    "isValidMember": false,
    "memberIndex": 14
  }, 
  {
    "height": 659574,
    "type": "llmq_50_60",
    "quorumHash": "8b1a6cfe4d9e7e1ab4d1e7bea03d8b9658e780aa5f77fe9c20c7af315e2dfb7b",
    "minedBlock": "c5a77d8e910c909af410a7a7920470c3dc067a27387f8657468d60466c2209f7",
    "quorumPublicKey": "168c173c4385b8d22375f488b3c8372182e2b9ee6b9dc4d5c46ba062cec8e4c27431cab5f878c1aa7f7c7555bdb5081d",
    "isValidMember": true,
    "memberIndex": 9
  }, 
  {
    "height": 659808,
    "type": "llmq_400_85",
    "quorumHash": "ae45e5698c14b72a0d5c12ed59cba0a79647c0a207a78fa6d840402e442a02a1",
    "minedBlock": "4eb6525f27a7f7df7fee75122f3afbbf65430bb2a28bc26f6aa4decaf3eb739d",
    "quorumPublicKey": "0b809f81048f27fc22276c4ead31b94b1c453a37753db088041c9fd7f389399e7ab5258b6a5e655b11f7f95c0ee8a706",
    "isValidMember": true,
    "memberIndex": 310
  }
]

I see nothing in the logs explaining why I was punished. What should I do? We can't keep losing nodes like this...

NorseGaud commented 1 year ago

Got another one today:

2023-04-16 16:33:58 CDeterministicMNList::PoSePunish -- punished MN c9e8d7bcbb450581b664af0b8937a4664af469a250d5e794652446ab4b5a8eb2, penalty 0->3110 (max=4713)

Sun Apr 16 16:28:05 UTC 2023
[
  {
    "height": 660384,
    "type": "llmq_400_85",
    "quorumHash": "9fb5792c402d7e8d8c62d78013dec39d89b04140032d88a502e6da92278cd09e",
    "minedBlock": "8f34fcf8e85548f3b8dec32768057fff1180018f13aa4fcbbaae9aa91b603261",
    "quorumPublicKey": "15552dd5f9a7800f34509db615f7629ef89c31c839da3ae9104455c1692559827600573c44371177e1731f3d584b36fa",
    "isValidMember": true,
    "memberIndex": 101
  }
]

==========
Sun Apr 16 16:28:09 UTC 2023
 16:28:10 up 175 days, 20:11,  0 users,  load average: 0.48, 0.52, 0.31
              total        used        free      shared  buff/cache   available
Mem:           922M        844M         39M         68K         38M         25M
Swap:          2.0G        1.0G        1.0G
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
firo-user 19210  2.7 84.9 3309156 802520 ?      S<Lsl Mar24 930:28 /usr/local/bin/firod
NorseGaud commented 1 year ago

ANOTHER...

2023-04-21 09:14:00 CDeterministicMNList::PoSePunish -- punished MN f0573dfaf5fb0b5956de005d0f16dff3962cb4a69e69e3ef4f1a10718883176d, penalty 0->3141 (max=4760)
2023-04-21 09:13:03 ConnectTip() nHeight=663598
2023-04-21 09:13:03 ConnectBlock nHeight=663598, hash=c77639f2a7e596bc674a10241ff5a15f329d7c74c140b6c975cbe756d6475a8d
2023-04-21 09:13:03 CheckBlock() nHeight=663598, blockHash= c77639f2a7e596bc674a10241ff5a15f329d7c74c140b6c975cbe756d6475a8d, isVerifyDB = false
2023-04-21 09:13:03 UpdateTip() pindexNew.nHeight=663598
2023-04-21 09:13:03 UpdateTip: new best=c77639f2a7e596bc674a10241ff5a15f329d7c74c140b6c975cbe756d6475a8d height=663598 version=0x20001000 log2_work=64.163959 tx=2556079 date='2023-04-21 09:12:57' progress=1.000000 cache=1.6MiB(6251txo)
2023-04-21 09:13:03 Checking if block contains wallet mints..
2023-04-21 09:13:10 receive version message: /Satoshi:0.14.12.1/: version 90031, blocks=663598, us=18.132.255.41:8168, peer=12060
2023-04-21 09:13:25 receive version message: /Satoshi:0.14.12.1/: version 90031, blocks=663598, us=18.132.255.41:48362, peer=12061
2023-04-21 09:13:30 CheckBlock() nHeight=2147483647, blockHash= 0ce9c6bdc28c1b5ea40f460296d2ad1a70cbfc5e7aa374a36bedd919f9470b2c, isVerifyDB = false
2023-04-21 09:13:30 CheckTransaction nHeight=663599, isVerifyDB=false, isCheckWallet=false, txHash=ec73be2205956de84dfac816cb1b5d81db98c2fb55112f26bd82e6ae04b07fea
2023-04-21 09:13:30 CheckTransaction nHeight=663599, isVerifyDB=false, isCheckWallet=false, txHash=60cd4f2cf66640a1c79b07b90f28bbeee679bd910b75deb26eefcfbfcc754072
2023-04-21 09:13:30 CheckBlock() nHeight=2147483647, blockHash= 0ce9c6bdc28c1b5ea40f460296d2ad1a70cbfc5e7aa374a36bedd919f9470b2c, isVerifyDB = false
2023-04-21 09:13:30 AcceptBlock nHeight=663599
2023-04-21 09:13:30 CheckBlock() nHeight=663599, blockHash= 0ce9c6bdc28c1b5ea40f460296d2ad1a70cbfc5e7aa374a36bedd919f9470b2c, isVerifyDB = false
2023-04-21 09:13:30 ActivateBestChainStep()
2023-04-21 09:13:30 ConnectTip() nHeight=663599
2023-04-21 09:13:30 ConnectBlock nHeight=663599, hash=0ce9c6bdc28c1b5ea40f460296d2ad1a70cbfc5e7aa374a36bedd919f9470b2c
2023-04-21 09:13:30 CheckBlock() nHeight=663599, blockHash= 0ce9c6bdc28c1b5ea40f460296d2ad1a70cbfc5e7aa374a36bedd919f9470b2c, isVerifyDB = false
2023-04-21 09:13:30 UpdateTip() pindexNew.nHeight=663599
2023-04-21 09:13:30 UpdateTip: new best=0ce9c6bdc28c1b5ea40f460296d2ad1a70cbfc5e7aa374a36bedd919f9470b2c height=663599 version=0x20001000 log2_work=64.163961 tx=2556081 date='2023-04-21 09:13:03' progress=1.000000 cache=1.6MiB(6255txo)
2023-04-21 09:13:30 Checking if block contains wallet mints..
2023-04-21 09:13:43 AcceptToMemoryPool(), transaction: b395fcb76c7869c10f2a80f68fe50e0d6c8ec5cc78c8bba9d1011e313da66d43
2023-04-21 09:13:43 AcceptToMemoryPoolWorker(), tx.IsSpend()=false, fTestNet=false
2023-04-21 09:13:43 CheckTransaction nHeight=2147483647, isVerifyDB=false, isCheckWallet=true, txHash=b395fcb76c7869c10f2a80f68fe50e0d6c8ec5cc78c8bba9d1011e313da66d43
2023-04-21 09:13:44 AcceptToMemoryPoolWorker -> OK
2023-04-21 09:13:44 Transaction b395fcb76c7869c10f2a80f68fe50e0d6c8ec5cc78c8bba9d1011e313da66d43 received and added to the mempool.
2023-04-21 09:13:44 AcceptToMemoryPool(), transaction: b395fcb76c7869c10f2a80f68fe50e0d6c8ec5cc78c8bba9d1011e313da66d43
2023-04-21 09:13:44 AcceptToMemoryPoolWorker(), tx.IsSpend()=false, fTestNet=false
2023-04-21 09:13:44 CheckTransaction nHeight=2147483647, isVerifyDB=false, isCheckWallet=true, txHash=b395fcb76c7869c10f2a80f68fe50e0d6c8ec5cc78c8bba9d1011e313da66d43
2023-04-21 09:13:44 AcceptToMemoryPoolWorker -> OK
2023-04-21 09:13:59 receive version message: /Satoshi:0.14.12.1/: version 90031, blocks=663599, us=18.132.255.41:43638, peer=12062
2023-04-21 09:14:00 CheckBlock() nHeight=2147483647, blockHash= bd20c6d77b3dee305afc826f36b16ea197e02b085070d84a5d2bcee74d752207, isVerifyDB = false
2023-04-21 09:14:00 CheckTransaction nHeight=663600, isVerifyDB=false, isCheckWallet=false, txHash=7bd4f0d8266bd31d55769fd9ac2eb1dee90c231f6ec1b6446f7d4bc31d9da8dd
2023-04-21 09:14:00 CheckTransaction nHeight=663600, isVerifyDB=false, isCheckWallet=false, txHash=11fa33fe4e9131263731d1f38c3e1bd9120e3c55d8a5f16e0c37d3df6b626212
2023-04-21 09:14:00 CheckBlock() nHeight=2147483647, blockHash= bd20c6d77b3dee305afc826f36b16ea197e02b085070d84a5d2bcee74d752207, isVerifyDB = false
2023-04-21 09:14:00 AcceptBlock nHeight=663600
2023-04-21 09:14:00 CheckBlock() nHeight=663600, blockHash= bd20c6d77b3dee305afc826f36b16ea197e02b085070d84a5d2bcee74d752207, isVerifyDB = false
2023-04-21 09:14:00 ActivateBestChainStep()
2023-04-21 09:14:00 ConnectTip() nHeight=663600
2023-04-21 09:14:00 ConnectBlock nHeight=663600, hash=bd20c6d77b3dee305afc826f36b16ea197e02b085070d84a5d2bcee74d752207
2023-04-21 09:14:00 CheckBlock() nHeight=663600, blockHash= bd20c6d77b3dee305afc826f36b16ea197e02b085070d84a5d2bcee74d752207, isVerifyDB = false
2023-04-21 09:14:00 CDeterministicMNList::PoSePunish -- punished MN 673dbf3dfe91686307d97eeb2a724946eadeca3580c43e423403583b97159485, penalty 0->3141 (max=4760)
2023-04-21 09:14:00 CDeterministicMNList::PoSePunish -- punished MN f0573dfaf5fb0b5956de005d0f16dff3962cb4a69e69e3ef4f1a10718883176d, penalty 0->3141 (max=4760)
2023-04-21 09:14:00 UpdateTip() pindexNew.nHeight=663600
2023-04-21 09:14:00 UpdateTip: new best=bd20c6d77b3dee305afc826f36b16ea197e02b085070d84a5d2bcee74d752207 height=663600 version=0x20001000 log2_work=64.163963 tx=2556083 date='2023-04-21 09:13:30' progress=1.000000 cache=1.6MiB(6260txo)
2023-04-21 09:14:00 Checking if block contains wallet mints..
2023-04-21 09:14:03 CheckBlock() nHeight=2147483647, blockHash= 1bc0bb0ee0d94ba282a63adb56555a63a3eb808a0f69fcd812c9892c1186903c, isVerifyDB = false
2023-04-21 09:14:03 CheckTransaction nHeight=663601, isVerifyDB=false, isCheckWallet=false, txHash=89c4a1448537fb2a9a246019e3560e1fb471af6e32f979c84f326cd995d4cc45
2023-04-21 09:14:03 CheckTransaction nHeight=663601, isVerifyDB=false, isCheckWallet=false, txHash=b395fcb76c7869c10f2a80f68fe50e0d6c8ec5cc78c8bba9d1011e313da66d43

No different from the others.

NorseGaud commented 1 year ago

My suspicion is that this is swap related. It looks like firod requires ~2GB of memory at times. We're moving swap to faster volumes and seeing what happens.

DNS commented 1 year ago

From my experience, firod requires at least 1.7-2.4 GB when first downloading the blockchain. Otherwise it will crash.

NorseGaud commented 1 year ago

That's what I'm seeing too. I turned OOM off on my MNs, so it just swaps. Even with swappiness of 1, it still does about ~500MB within a few minutes on top of the 1GB I have for the server.

justanwar commented 1 year ago

I have increased the recommended RAM size in the masternode guide and recommended having swap on. Other than these, what other things would you recommend?

NorseGaud commented 1 year ago

Thanks @justanwar. I'm still testing the changes where we have swap on a faster external volume than the root. It will take a while to know if the MNs I run keep getting penalties or not. I'll follow up and close this if we see less over time.