search

nspcc-dev / neofs-node

NeoFS is a decentralized distributed object storage integrated with the Neo blockchain
https://fs.neo.org
GNU General Public License v3.0
32 stars 38 forks source link

Storage nodes sometimes can't restart with 'insufficient funds' error #2932

Open evgeniiz321 opened 2 months ago

evgeniiz321 commented 2 months ago

https://rest.fs.neo.org/HXSaMJXk2g8C14ht8HSi7BBaiYZ1HeWh2xnWPGQCg4H6/1866-1725882308/index.html#suites/b7bbb57d6d390c39f626ffa2298ddb2a/14982d8daf7d4e01/

Stop 4 SNs Restart them one by one - sometimes some nodes don't restart successfully. If try to restart same node multiple times - eventually it will be started.

In logs I see:

2024/09/09 09:29:36 can't make notary deposit: Insufficient funds (-511) - insufficient funds

Full SN log - sn_3_stderr.txt

Expected Behavior

Storage Nodes should always be able to restart successfully when there are no configuration or environment issues.

carpawell commented 1 month ago

@evgeniiz321, could not reproduce it after some days trying. Can you, please try to catch it one more time and attach IR's logs? Reports in this issue were not available to me when I first time read it. It will be more preferable to use a new node after https://github.com/nspcc-dev/neofs-node/pull/2952, it added more useful logs.