near / stakewars-iii

Stake Wars: Episode 3 challenges and place to report issues
87 stars 177 forks source link

Validator stopped producing chunks after failover test. #139

Open kernelpanic9 opened 1 year ago

kernelpanic9 commented 1 year ago

Validator produces 0 chunks after failover to the backup node. Blocks are produces as expected.

Returning validator to the main node and restarting it didn't take any effect.

Main node key: ed25519:C82k8iPGDB2wmiNTGS7Hy7qUxJuy3KgGRCBi7cftXYNT Backup node key: ed25519:t16xfiRHddiJz21jeBotEsR3KiGq8mEM1L9Juh5ZV8U

Failover time: 2022-09-02T23:18:01 CET

Main node debug page: http://142.132.252.32:3030/debug

See here:

| spectrum.factory.shardnet.near            | 107,780 | 1       | 94.17%   |              74 |              74 |             104 |             115 |
<<FAILOVER to backup>>
$near validators current  | grep spectrum
| spectrum.factory.shardnet.near            | 107,780 | 1       | 92.92%   |              80 |              80 |             104 |             118 |
$near validators current  | grep spectrum
| spectrum.factory.shardnet.near            | 107,780 | 1       | 88.42%   |              86 |              86 |             105 |             130 |
$near validators current  | grep spectrum
| spectrum.factory.shardnet.near            | 107,780 | 1       | 88.42%   |              86 |              86 |             105 |             130 |
$near validators current  | grep spectrum
| spectrum.factory.shardnet.near            | 107,780 | 1       | 88.42%   |              86 |              86 |             105 |             130 |
$near validators current  | grep spectrum
| spectrum.factory.shardnet.near            | 107,918 | 1       | 0%       |               0 |               0 |               0 |               1 |
$near validators current  | grep spectrum
| spectrum.factory.shardnet.near            | 107,918 | 1       | 0%       |               0 |               0 |               0 |               1 |
$near validators current  | grep spectrum
| spectrum.factory.shardnet.near            | 107,918 | 1       | 0%       |               0 |               0 |               0 |               2 |
$near validators current  | grep spectrum
| spectrum.factory.shardnet.near            | 107,918 | 1       | 0%       |               0 |               0 |               0 |               5 |
$near validators current  | grep spectrum
| spectrum.factory.shardnet.near            | 107,918 | 1       | 0%       |               0 |               0 |               0 |               6 |
$near validators current  | grep spectrum
| spectrum.factory.shardnet.near            | 107,918 | 1       | 21.42%   |               3 |               4 |               0 |              10 |
$near validators current  | grep spectrum
| spectrum.factory.shardnet.near            | 107,918 | 1       | 21.42%   |               3 |               4 |               0 |              10 |
$
$near validators current  | grep spectrum
| spectrum.factory.shardnet.near            | 107,918 | 1       | 21.42%   |               3 |               4 |               0 |              10 |
$near validators current  | grep spectrum
| spectrum.factory.shardnet.near            | 107,918 | 1       | 21.42%   |               3 |               4 |               0 |              10 |
$near validators current  | grep spectrum
| spectrum.factory.shardnet.near            | 107,918 | 1       | 22.72%   |               5 |               6 |               0 |              16 |
$near validators current  | grep spectrum
| spectrum.factory.shardnet.near            | 107,918 | 1       | 22.72%   |               5 |               6 |               0 |              16 |
$near validators current  | grep spectrum
| spectrum.factory.shardnet.near            | 107,918 | 1       | 24%      |               6 |               7 |               0 |              18 |
$near validators current  | grep spectrum
| spectrum.factory.shardnet.near            | 107,918 | 1       | 24%      |               6 |               7 |               0 |              18 |
$near validators current  | grep spectrum
| spectrum.factory.shardnet.near            | 107,918 | 1       | 24%      |               6 |               7 |               0 |              18 |
$near validators current  | grep spectrum
| spectrum.factory.shardnet.near            | 107,918 | 1       | 23.07%   |               6 |               7 |               0 |              19 |
$near validators current  | grep spectrum
| spectrum.factory.shardnet.near            | 107,918 | 1       | 22.22%   |               6 |               7 |               0 |              20 |
dienkun92 commented 1 year ago

me too :(