search

maticnetwork / bor

Official repository for the Polygon Blockchain
https://polygon.technology/
GNU Lesser General Public License v3.0
1k stars 490 forks source link

Unable to prune due to snapshot mismatch #648

Closed YuXiaoCoder closed 1 year ago

YuXiaoCoder commented 1 year ago

I refer to this solution and want to prune the data of version 0.3.0, using bor of 0.2.16: https://github.com/maticnetwork/bor/issues/643 The execution command is

/tmp/bor snapshot prune-state --datadir=/mnt/maticmain/node/bor

Get snapshot hash mismatch and can't prune logs:

INFO [01-06|11:34:00.021] Maximum peer count                       ETH=50 LES=0 total=50
INFO [01-06|11:34:00.021] Smartcard socket not found, disabling    err="stat /run/pcscd/pcscd.comm: no such file or directory"
INFO [01-06|11:34:00.021] Set global gas cap                       cap=50,000,000
INFO [01-06|11:34:00.021] Allocated cache and file handles         database=/mnt/maticmain/node/bor/bor/chaindata cache=512.00MiB handles=524,288
INFO [01-06|11:34:04.644] Opened ancient database                  database=/mnt/maticmain/node/bor/bor/chaindata/ancient readonly=false
WARN [01-06|11:34:04.647] Loaded snapshot journal                  diskroot=e54385..6abd80 diffs=unmatched
ERROR[01-06|11:34:04.647] Failed to open snapshot tree             err="head doesn't match snapshot: have 0xe5438523fc6280151e073e85027ec2c5c07604738a4f2d7ae9527b93f06abd80, want 0xb44cfa3d8da56ace2c1b49501c1a211b59541ea16d6e2b8d274ddb8c0ca22558"
head doesn't match snapshot: have 0xe5438523fc6280151e073e85027ec2c5c07604738a4f2d7ae9527b93f06abd80, want 0xb44cfa3d8da56ace2c1b49501c1a211b59541ea16d6e2b8d274ddb8c0ca22558
cffls commented 1 year ago

Hi @YuXiaoCoder , it looks like the snapshot doesn't match the head of chain. You can try restarting bor and letting it sync until the snapshot is restored (match the head of chain), and then try pruning again.

YuXiaoCoder commented 1 year ago

Hi, @cffls The node has been synchronized to the latest block, what log can represent the snapshot has been restored? Is it recommended to add -snapshot=false to the running parameters

YuXiaoCoder commented 1 year ago

INFO [01-06|16:18:10.680] Generating state snapshot root=f1fce7..e6e63a in=0068f3..530a4c at=17b4e8..be8a8a accounts=179,876 slots=1,501,180 storage=117.61MiB elapsed=1m20.014s eta=13h51m25.447s INFO [01-06|16:18:18.681] Generating state snapshot root=f1fce7..e6e63a in=0068f3..530a4c at=6aa05d..78062e accounts=179,876 slots=1,507,455 storage=118.04MiB elapsed=1m28.014s eta=15h14m33.057s ERROR[01-06|16:18:20.010] Snapshot extension registration failed peer=ffc54465 err="peer connected on snap without compatible eth support"

cffls commented 1 year ago

It is not recommended to disable snapshot. This log entry is the indication of incomplete snapshot:

INFO [01-06|16:18:18.681] Generating state snapshot root=f1fce7..e6e63a in=0068f3..530a4c at=6aa05d..78062e accounts=179,876 slots=1,507,455 storage=118.04MiB elapsed=1m28.014s eta=15h14m33.057s

ETA is 15 hour to restore the snapshot, so you will probably need to wait for that amount of time before start pruning.

YuXiaoCoder commented 1 year ago

I found that the eta time in the follow-up logs is still increasing. Is there any indicator that can stop the service in time after the snapshot restoration is completed, so as to avoid entering the next round of snapshot restoration?

INFO [01-06|16:28:24.028] Generating state snapshot                root=1494ca..3c7a20 at=02b695..f53831 accounts=1,192,398 slots=11,869,117 storage=994.19MiB elapsed=11m33.361s  eta=17h58m47.503s
WARN [01-06|16:28:30.975] unable to whitelist checkpoint           err="missing checkpoint blocks"
INFO [01-06|16:28:32.028] Generating state snapshot                root=1494ca..3c7a20 at=02caea..da12df accounts=1,227,538 slots=12,048,200 storage=1009.03MiB elapsed=11m41.362s  eta=17h39m52.354s
INFO [01-06|16:28:40.032] Generating state snapshot                root=1494ca..3c7a20 at=02dafa..a211ea accounts=1,254,933 slots=12,201,296 storage=1021.93MiB elapsed=11m49.365s  eta=17h28m8.964s
INFO [01-06|16:28:46.789] Deep froze chain segment                 blocks=5  elapsed=12.158ms    number=37,641,833 hash=fe7958..2d805f
INFO [01-06|16:28:48.032] Generating state snapshot                root=1494ca..3c7a20 at=02e6b3..86f96a accounts=1,275,227 slots=12,727,685 storage=1.03GiB    elapsed=11m57.365s  eta=17h23m3.06s
INFO [01-06|16:28:54.753] Downloader queue stats                   receiptTasks=0 blockTasks=1024 itemSize=54.09KiB throttle=4847
INFO [01-06|16:28:56.033] Generating state snapshot                root=1494ca..3c7a20 in=02fb6d..c7e9bb at=929387..d1fecf accounts=1,311,014 slots=12,854,486 storage=1.05GiB    elapsed=12m5.366s   eta=17h5m43.269s
INFO [01-06|16:28:56.744] Aborting state snapshot generation       root=1494ca..3c7a20 in=02fc0a..146c7a at=4b2754..1917bf accounts=1,312,095 slots=12,869,032 storage=1.05GiB    elapsed=12m6.077s   eta=17h5m53.469s
INFO [01-06|16:28:56.744] Resuming state snapshot generation       root=57902e..557486 in=02fc0a..146c7a at=4b2754..1917bf accounts=1,312,095 slots=12,869,032 storage=1.05GiB    elapsed=12m6.077s   eta=17h5m53.469s
INFO [01-06|16:28:59.318] Aborting state snapshot generation       root=57902e..557486 in=030224..7d654d at=058063..279b57 accounts=1,322,538 slots=12,938,661 storage=1.05GiB    elapsed=12m8.651s   eta=17h1m16.696s
INFO [01-06|16:28:59.318] Resuming state snapshot generation       root=7e7f67..ae7de5 in=030224..7d654d at=058063..279b57 accounts=1,322,538 slots=12,938,661 storage=1.05GiB    elapsed=12m8.651s   eta=17h1m16.696s
INFO [01-06|16:28:59.794] Aborting state snapshot generation       root=7e7f67..ae7de5 in=030224..7d654d at=12b50c..892975 accounts=1,322,538 slots=12,987,814 storage=1.06GiB    elapsed=12m9.127s   eta=17h1m56.726s
INFO [01-06|16:28:59.794] Resuming state snapshot generation       root=08105a..57d1f3 in=030224..7d654d at=12b50c..892975 accounts=1,322,538 slots=12,987,814 storage=1.06GiB    elapsed=12m9.128s   eta=17h1m56.81s
INFO [01-06|16:29:00.879] Aborting state snapshot generation       root=08105a..57d1f3 in=030224..7d654d at=2cabac..1adecc accounts=1,322,538 slots=13,085,095 storage=1.06GiB    elapsed=12m10.212s  eta=17h3m27.97s
INFO [01-06|16:29:00.879] Resuming state snapshot generation       root=68a323..8837c5 in=030224..7d654d at=2cabac..1adecc accounts=1,322,538 slots=13,085,095 storage=1.06GiB    elapsed=12m10.212s  eta=17h3m27.97s
INFO [01-06|16:29:02.406] Aborting state snapshot generation       root=68a323..8837c5 in=030224..7d654d at=4cf749..5c9fbf accounts=1,322,538 slots=13,204,904 storage=1.07GiB    elapsed=12m11.739s  eta=17h5m36.384s
INFO [01-06|16:29:02.406] Resuming state snapshot generation       root=65d975..918853 in=030224..7d654d at=4cf749..5c9fbf accounts=1,322,538 slots=13,204,904 storage=1.07GiB    elapsed=12m11.740s  eta=17h5m36.468s
cffls commented 1 year ago

Once snapshot is restored, you can stop your node anytime. Snapshot is usually broken when the node is not shutdown gracefully.

YuXiaoCoder commented 1 year ago

@cffls I'll test it

YuXiaoCoder commented 1 year ago

@cffls After I waited for the snapshot to be generated successfully, I output the Generated state snapshot log, then stopped the service, and ran the pruning task, but still got the same error

WARN [01-08|11:58:04.649] Already shutting down, interrupt more force stop. times=9
INFO [01-08|11:58:04.970] Ethereum protocol stopped
INFO [01-08|11:58:04.974] Transaction pool stopped
ERROR[01-08|11:58:04.975] Failed to journal state snapshot         err="snapshot [0x4024ac180ddf30c19c9f15777f695a8e81842a17280a31b673d741c50800cfae] missing"
INFO [01-08|11:58:04.975] Writing clean trie cache to disk         path=/mnt/maticmain/node/bor/bor/triecache threads=32
INFO [01-08|11:58:05.228] Persisted the clean trie cache           path=/mnt/maticmain/node/bor/bor/triecache elapsed=252.982ms
INFO [01-08|11:58:05.229] Blockchain stopped
/opt/maticmain/core/bor snapshot prune-state --datadir=/mnt/maticmain/node/bor
INFO [01-08|17:59:54.689] Allocated cache and file handles         database=/mnt/maticmain/node/bor/bor/chaindata cache=1024.00MiB handles=524,288
INFO [01-08|18:00:03.834] Opened ancient database                  database=/mnt/maticmain/node/bor/bor/chaindata/ancient readonly=false
WARN [01-08|18:00:03.878] Loaded snapshot journal                  diskroot=645e28..ee1609 diffs=unmatched
ERROR[01-08|18:00:03.878] Failed to open snapshot tree             err="head doesn't match snapshot: have 0x645e28b5a239d82852cecf471f6a72187310a46e30912a01a7839a3de2ee1609, want 0x4024ac180ddf30c19c9f15777f695a8e81842a17280a31b673d741c50800cfae"
cffls commented 1 year ago

Thanks for the update. Could you share a few more lines of bor log before stopping, as well as the node config? Also, this line may be the problem ERROR[01-08|11:58:04.975] Failed to journal state snapshot err="snapshot [0x4024ac180ddf30c19c9f15777f695a8e81842a17280a31b673d741c50800cfae] missing" , but it is not clear why it failed to journal state snapshot.

YuXiaoCoder commented 1 year ago

@cffls The following is the log before the node stops

INFO [01-08|11:57:00.400] Imported new chain segment               blocks=1   txs=59    mgas=10.520  elapsed=202.883ms     mgasps=51.851   number=37,807,964 hash=23ca62..dbdf61 dirty=0.00B
INFO [01-08|11:57:00.401] Unindexed transactions                   blocks=1   txs=60    tail=35,457,965 elapsed="678.671µs"
INFO [01-08|11:57:02.251] Imported new chain segment               blocks=1   txs=54    mgas=8.902   elapsed=65.093ms      mgasps=136.751  number=37,807,965 hash=ea302e..3cf1c6 dirty=0.00B
INFO [01-08|11:57:02.252] Unindexed transactions                   blocks=1   txs=104   tail=35,457,966 elapsed="912.564µs"
INFO [01-08|11:57:04.465] Imported new chain segment               blocks=1   txs=72    mgas=24.050  elapsed=134.040ms     mgasps=179.423  number=37,807,966 hash=efe7c5..cabd8b dirty=0.00B
INFO [01-08|11:57:04.466] Unindexed transactions                   blocks=1   txs=107   tail=35,457,967 elapsed=1.099ms
INFO [01-08|11:57:06.424] Imported new chain segment               blocks=1   txs=56    mgas=10.911  elapsed=91.925ms      mgasps=118.690  number=37,807,967 hash=4e2136..495b9c dirty=0.00B
INFO [01-08|11:57:06.426] Unindexed transactions                   blocks=1   txs=81    tail=35,457,968 elapsed=1.250ms
INFO [01-08|11:57:09.137] Deep froze chain segment                 blocks=29  elapsed=83.215ms      number=37,717,967 hash=87451c..56f7f3
INFO [01-08|11:57:09.922] Imported new chain segment               blocks=1   txs=51    mgas=20.173  elapsed=1.710s        mgasps=11.791   number=37,807,968 hash=b71ff2..d35f0b dirty=0.00B
INFO [01-08|11:57:09.923] Unindexed transactions                   blocks=1   txs=78    tail=35,457,969 elapsed="798.911µs"
INFO [01-08|11:57:10.962] Imported new chain segment               blocks=1   txs=70    mgas=11.246  elapsed=797.165ms     mgasps=14.107   number=37,807,969 hash=c26cde..4f47cd dirty=0.00B
INFO [01-08|11:57:10.963] Unindexed transactions                   blocks=1   txs=67    tail=35,457,970 elapsed="730.289µs"
INFO [01-08|11:57:13.887] Imported new chain segment               blocks=1   txs=71    mgas=22.532  elapsed=1.698s        mgasps=13.270   number=37,807,970 hash=69d5f7..6eaf7f dirty=0.00B
INFO [01-08|11:57:13.888] Unindexed transactions                   blocks=1   txs=93    tail=35,457,971 elapsed="947.235µs"
INFO [01-08|11:57:15.277] Imported new chain segment               blocks=1   txs=50    mgas=8.343   elapsed=866.429ms     mgasps=9.629    number=37,807,971 hash=45cf3d..3ad9ea dirty=0.00B
INFO [01-08|11:57:15.278] Unindexed transactions                   blocks=1   txs=62    tail=35,457,972 elapsed="944.11µs"
INFO [01-08|11:57:16.543] Imported new chain segment               blocks=1   txs=61    mgas=9.935   elapsed=343.168ms     mgasps=28.952   number=37,807,972 hash=0080d2..fe6a73 dirty=0.00B
INFO [01-08|11:57:16.544] Unindexed transactions                   blocks=1   txs=81    tail=35,457,973 elapsed=1.092ms
INFO [01-08|11:57:18.337] Imported new chain segment               blocks=1   txs=83    mgas=10.712  elapsed=84.023ms      mgasps=127.491  number=37,807,973 hash=09876c..a20977 dirty=0.00B
INFO [01-08|11:57:18.338] Unindexed transactions                   blocks=1   txs=62    tail=35,457,974 elapsed="816.53µs"
INFO [01-08|11:57:21.818] Imported new chain segment               blocks=1   txs=79    mgas=28.807  elapsed=1.580s        mgasps=18.223   number=37,807,974 hash=3c7d54..cb94ca dirty=0.00B
INFO [01-08|11:57:21.819] Unindexed transactions                   blocks=1   txs=67    tail=35,457,975 elapsed="831.094µs"
INFO [01-08|11:57:23.406] Imported new chain segment               blocks=1   txs=61    mgas=12.112  elapsed=754.475ms     mgasps=16.054   number=37,807,975 hash=26d850..98afe1 dirty=0.00B
INFO [01-08|11:57:23.407] Unindexed transactions                   blocks=1   txs=70    tail=35,457,976 elapsed="902.52µs"
INFO [01-08|11:57:24.330] Imported new chain segment               blocks=1   txs=53    mgas=9.292   elapsed=186.878ms     mgasps=49.719   number=37,807,976 hash=b00d87..9b8e89 dirty=0.00B
INFO [01-08|11:57:24.332] Unindexed transactions                   blocks=1   txs=86    tail=35,457,977 elapsed=1.320ms
INFO [01-08|11:57:26.259] Imported new chain segment               blocks=1   txs=49    mgas=7.608   elapsed=61.233ms      mgasps=124.242  number=37,807,977 hash=c6b185..35e3f8 dirty=0.00B
INFO [01-08|11:57:26.260] Unindexed transactions                   blocks=1   txs=92    tail=35,457,978 elapsed=1.249ms
INFO [01-08|11:57:28.232] Imported new chain segment               blocks=1   txs=0     mgas=0.000   elapsed="335.372µs"   mgasps=0.000    number=37,807,978 hash=b2e58e..4b7465 dirty=0.00B
INFO [01-08|11:57:28.233] Unindexed transactions                   blocks=1   txs=151   tail=35,457,979 elapsed=1.407ms
INFO [01-08|11:57:32.334] Imported new chain segment               blocks=1   txs=143   mgas=28.738  elapsed=2.021s        mgasps=14.216   number=37,807,979 hash=805ed9..fd8630 dirty=0.00B
INFO [01-08|11:57:32.335] Unindexed transactions                   blocks=1   txs=55    tail=35,457,980 elapsed="730.049µs"
INFO [01-08|11:57:32.746] Imported new chain segment               blocks=1   txs=67    mgas=28.701  elapsed=411.012ms     mgasps=69.830   number=37,807,980 hash=9f77b7..6e8b27 dirty=0.00B
INFO [01-08|11:57:32.747] Unindexed transactions                   blocks=1   txs=48    tail=35,457,981 elapsed="647.188µs"
INFO [01-08|11:57:34.355] Downloader queue stats                   receiptTasks=0 blockTasks=0   itemSize=48.22KiB  throttle=5437
INFO [01-08|11:57:36.032] Imported new chain segment               blocks=1   txs=67    mgas=23.132  elapsed=1.805s        mgasps=12.813   number=37,807,981 hash=2366f2..60b907 dirty=0.00B
INFO [01-08|11:57:36.034] Unindexed transactions                   blocks=1   txs=79    tail=35,457,982 elapsed=1.392ms
INFO [01-08|11:57:36.981] Imported new chain segment               blocks=1   txs=49    mgas=9.391   elapsed=727.693ms     mgasps=12.906   number=37,807,982 hash=5c5a44..6f64ed dirty=0.00B
INFO [01-08|11:57:36.982] Unindexed transactions                   blocks=1   txs=70    tail=35,457,983 elapsed="853.212µs"
INFO [01-08|11:57:38.571] Imported new chain segment               blocks=1   txs=55    mgas=9.538   elapsed=319.438ms     mgasps=29.858   number=37,807,983 hash=f7bd69..1e3b3c dirty=0.00B
INFO [01-08|11:57:38.572] Unindexed transactions                   blocks=1   txs=50    tail=35,457,984 elapsed="902.547µs"
INFO [01-08|11:57:40.269] Imported new chain segment               blocks=1   txs=58    mgas=7.933   elapsed=67.099ms      mgasps=118.229  number=37,807,984 hash=a4c4f7..1bcd12 dirty=0.00B
INFO [01-08|11:57:40.271] Unindexed transactions                   blocks=1   txs=168   tail=35,457,985 elapsed=1.588ms
INFO [01-08|11:57:42.314] Imported new chain segment               blocks=1   txs=74    mgas=12.234  elapsed=91.510ms      mgasps=133.690  number=37,807,985 hash=b9bb54..e6796e dirty=0.00B
INFO [01-08|11:57:42.315] Unindexed transactions                   blocks=1   txs=66    tail=35,457,986 elapsed="680.127µs"
INFO [01-08|11:57:45.683] Imported new chain segment               blocks=1   txs=64    mgas=25.335  elapsed=1.456s        mgasps=17.392   number=37,807,986 hash=d87f31..4f70d3 dirty=0.00B
INFO [01-08|11:57:45.684] Unindexed transactions                   blocks=1   txs=58    tail=35,457,987 elapsed="638.781µs"
INFO [01-08|11:57:47.098] Imported new chain segment               blocks=1   txs=63    mgas=13.707  elapsed=892.374ms     mgasps=15.361   number=37,807,987 hash=e2a7b5..e83bf5 dirty=0.00B
INFO [01-08|11:57:47.098] Unindexed transactions                   blocks=1   txs=63    tail=35,457,988 elapsed="796.872µs"
INFO [01-08|11:57:48.589] Imported new chain segment               blocks=1   txs=44    mgas=8.605   elapsed=69.094ms      mgasps=124.538  number=37,807,988 hash=78f8a4..280e9a dirty=0.00B
INFO [01-08|11:57:48.590] Unindexed transactions                   blocks=1   txs=63    tail=35,457,989 elapsed="753.3µs"
INFO [01-08|11:57:50.312] Imported new chain segment               blocks=1   txs=79    mgas=13.753  elapsed=101.251ms     mgasps=135.835  number=37,807,989 hash=5321b6..9fd333 dirty=0.00B
INFO [01-08|11:57:50.314] Unindexed transactions                   blocks=1   txs=187   tail=35,457,990 elapsed=1.491ms
INFO [01-08|11:57:52.298] Imported new chain segment               blocks=1   txs=76    mgas=12.399  elapsed=103.071ms     mgasps=120.293  number=37,807,990 hash=bd4747..60c480 dirty=0.00B
INFO [01-08|11:57:52.298] Unindexed transactions                   blocks=1   txs=60    tail=35,457,991 elapsed="749.451µs"
INFO [01-08|11:57:54.525] Imported new chain segment               blocks=1   txs=72    mgas=10.082  elapsed=349.771ms     mgasps=28.825   number=37,807,991 hash=940c31..f97af7 dirty=0.00B
INFO [01-08|11:57:54.526] Unindexed transactions                   blocks=1   txs=55    tail=35,457,992 elapsed="656.197µs"
INFO [01-08|11:57:57.679] Imported new chain segment               blocks=1   txs=50    mgas=28.341  elapsed=1.453s        mgasps=19.498   number=37,807,992 hash=a2bbb1..39b9aa dirty=0.00B
INFO [01-08|11:57:57.680] Unindexed transactions                   blocks=1   txs=80    tail=35,457,993 elapsed="803.895µs"
INFO [01-08|11:57:58.863] Imported new chain segment               blocks=1   txs=66    mgas=10.344  elapsed=329.208ms     mgasps=31.422   number=37,807,993 hash=9dac95..a0eefe dirty=0.00B
INFO [01-08|11:57:58.864] Unindexed transactions                   blocks=1   txs=113   tail=35,457,994 elapsed=1.103ms
INFO [01-08|11:58:00.385] Imported new chain segment               blocks=1   txs=64    mgas=11.265  elapsed=97.515ms      mgasps=115.515  number=37,807,994 hash=337f6c..ea10a8 dirty=0.00B
INFO [01-08|11:58:00.387] Unindexed transactions                   blocks=1   txs=183   tail=35,457,995 elapsed=1.883ms
INFO [01-08|11:58:02.335] Imported new chain segment               blocks=1   txs=52    mgas=10.627  elapsed=74.109ms      mgasps=143.402  number=37,807,995 hash=887236..4cfb61 dirty=0.00B
INFO [01-08|11:58:02.336] Unindexed transactions                   blocks=1   txs=100   tail=35,457,996 elapsed="943.991µs"
INFO [01-08|11:58:04.327] Imported new chain segment               blocks=1   txs=85    mgas=19.519  elapsed=120.433ms     mgasps=162.072  number=37,807,996 hash=77ff59..933a23 dirty=0.00B
INFO [01-08|11:58:04.328] Unindexed transactions                   blocks=1   txs=74    tail=35,457,997 elapsed="891.532µs"
Caught signal: terminated[2023-01-08 11:58:04] receive the service exit signal

Gracefully shutting down agent...
INFO [01-08|11:58:04.634] HTTP server stopped                      endpoint=[::]:8545
[2023-01-08 11:58:04] get service pid [203136] by command [/opt/maticmain/core/bor]
[2023-01-08 11:58:04] exec command [kill -15 203136], try count [1]
WARN [01-08|11:58:04.649] Already shutting down, interrupt more force stop. times=9
INFO [01-08|11:58:04.970] Ethereum protocol stopped
INFO [01-08|11:58:04.974] Transaction pool stopped
ERROR[01-08|11:58:04.975] Failed to journal state snapshot         err="snapshot [0x4024ac180ddf30c19c9f15777f695a8e81842a17280a31b673d741c50800cfae] missing"
INFO [01-08|11:58:04.975] Writing clean trie cache to disk         path=/mnt/maticmain/node/bor/bor/triecache threads=32
INFO [01-08|11:58:05.228] Persisted the clean trie cache           path=/mnt/maticmain/node/bor/bor/triecache elapsed=252.982ms
INFO [01-08|11:58:05.229] Blockchain stopped

I'm not using a configuration file, it's all command line parameters

/opt/maticmain/core/bor server -chain=mainnet -datadir=/mnt/maticmain/node/bor -bor.heimdall=http://localhost:1317 -ipcdisable -http -http.addr=0.0.0.0 -http.vhosts=* -http.corsdomain=* -http.port=8545 -http.api=admin,eth,net,web3,txpool,bor,debug -syncmode=full -gcmode=archive -miner.gasprice=30000000000 -miner.gaslimit=20000000 -txpool.nolocals -txpool.accountslots=16 -txpool.globalslots=32768 -txpool.lifetime=1h30m0s -maxpeers=200 -rpc.txfeecap=10 -rpc.gascap=5000000000 -bootnodes=enode://0cb82b395094ee4a2915e9714894627de9ed8498fb881cec6db7c65e8b9a5bd7f2f25cc84e71e89d0947e51c76e85d0847de848c7782b13c0255247a6758178c@44.232.55.71:30303,enode://88116f4295f5a31538ae409e4d44ad40d22e44ee9342869e7d68bdec55b0f83c1530355ce8b41fbec0928a7d75a5745d528450d30aec92066ab6ba1ee351d710@159.203.9.164:30303
YuXiaoCoder commented 1 year ago

The following is the log when the node just started, with Unclean shutdown detected

INFO [01-09|09:45:43.261] GRPC Server started                      addr=[::]:3131
INFO [01-09|09:45:43.262] Set global gas cap                       cap=5,000,000,000
INFO [01-09|09:45:43.262] Enabling recording of key preimages since archive mode is used
INFO [01-09|09:45:43.262] Allocated trie memory caches             clean=306.00MiB dirty=0.00B
INFO [01-09|09:45:43.263] Allocated cache and file handles         database=/mnt/maticmain/node/bor/bor/chaindata cache=512.00MiB handles=524,288
INFO [01-09|09:45:44.662] Opened ancient database                  database=/mnt/maticmain/node/bor/bor/chaindata/ancient readonly=false
INFO [01-09|09:45:44.667] Persisted trie from memory database      nodes=12 size=1.86KiB time="24.774µs" gcnodes=0 gcsize=0.00B gctime=0s livenodes=1 livesize=0.00B
INFO [01-09|09:45:44.669] Initialised chain configuration          config="{ChainID: 137 Homestead: 0 DAO: <nil> DAOSupport: true EIP150: 0 EIP155: 0 EIP158: 0 Byzantium: 0 Constantinople: 0 Petersburg: 0 Istanbul: 3395000, Muir Glacier: 3395000, Berlin: 14750000, London: 23850000, Arrow Glacier: <nil>, MergeFork: <nil>, Terminal TD: <nil>, Engine: bor}"
INFO [01-09|09:45:45.444] Gasprice oracle is ignoring threshold set threshold=2
INFO [01-09|09:45:45.449] Initialising Ethereum protocol           network=137 dbversion=8
INFO [01-09|09:45:45.451] Loaded most recent local header          number=37,657,875 hash=f28dd1..2796f9 td=600,900,777 age=4d15h50m
INFO [01-09|09:45:45.451] Loaded most recent local full block      number=37,657,875 hash=f28dd1..2796f9 td=600,900,777 age=4d15h50m
INFO [01-09|09:45:45.451] Loaded most recent local fast block      number=37,657,875 hash=f28dd1..2796f9 td=600,900,777 age=4d15h50m
WARN [01-09|09:45:45.455] Head state missing, repairing            number=37,657,875 hash=f28dd1..2796f9 snaproot=c4e159..164ab2
INFO [01-09|09:45:45.886] Loaded most recent local header          number=37,657,875 hash=f28dd1..2796f9 td=600,900,777 age=4d15h50m
INFO [01-09|09:45:45.886] Loaded most recent local full block      number=37,657,748 hash=dab148..085b6d td=600,898,110 age=4d15h54m
INFO [01-09|09:45:45.886] Loaded most recent local fast block      number=37,657,875 hash=f28dd1..2796f9 td=600,900,777 age=4d15h50m
WARN [01-09|09:45:45.933] Old unclean shutdowns found              count=141
WARN [01-09|09:45:45.933] Unclean shutdown detected                booted=2022-03-11T06:21:24+0800 age=10mo4d3h
WARN [01-09|09:45:45.933] Unclean shutdown detected                booted=2022-03-11T06:36:24+0800 age=10mo4d3h
WARN [01-09|09:45:45.933] Unclean shutdown detected                booted=2022-03-11T06:51:11+0800 age=10mo4d2h
WARN [01-09|09:45:45.933] Unclean shutdown detected                booted=2022-03-11T07:06:09+0800 age=10mo4d2h
WARN [01-09|09:45:45.933] Unclean shutdown detected                booted=2022-03-11T07:25:49+0800 age=10mo4d2h
WARN [01-09|09:45:45.933] Unclean shutdown detected                booted=2022-03-11T07:36:39+0800 age=10mo4d2h
WARN [01-09|09:45:45.933] Unclean shutdown detected                booted=2022-03-11T07:51:24+0800 age=10mo4d1h
WARN [01-09|09:45:45.933] Unclean shutdown detected                booted=2022-03-11T08:06:19+0800 age=10mo4d1h
WARN [01-09|09:45:45.933] Unclean shutdown detected                booted=2022-03-11T08:16:08+0800 age=10mo4d1h
WARN [01-09|09:45:45.933] Unclean shutdown detected                booted=2022-03-11T08:19:35+0800 age=10mo4d1h
INFO [01-09|09:45:45.933] Starting peer-to-peer node               instance=bor/v0.3.2-stable/linux-amd64/go1.19.4
cffls commented 1 year ago

Looks like the node was not shutdown gracefully, which is most likely the root cause.

I notice that signal -15 was used to kill bor, as shown in this line below:

[2023-01-08 11:58:04] exec command [kill -15 203136], try count [1]

Could you try restoring the node and using interrupt kill -SIGINT <pid> or ctrl + c instead?

YuXiaoCoder commented 1 year ago

ok, i'll try it, thanks

YuXiaoCoder commented 1 year ago

The problem has been solved. Thank you very much

zhy827827 commented 10 months ago

I tried all of the methods mentioned above and couldn't solve the issues

WARN [11-20|04:08:13.646] Old unclean shutdowns found count=147 WARN [11-20|04:08:13.646] Unclean shutdown detected booted=2022-03-10T23:51:24+0000 age=1y8mo2w WARN [11-20|04:08:13.646] Unclean shutdown detected booted=2022-03-11T00:06:19+0000 age=1y8mo2w WARN [11-20|04:08:13.646] Unclean shutdown detected booted=2022-03-11T00:16:08+0000 age=1y8mo2w WARN [11-20|04:08:13.646] Unclean shutdown detected booted=2022-03-11T00:19:35+0000 age=1y8mo2w WARN [11-20|04:08:13.646] Unclean shutdown detected booted=2022-11-01T18:24:37+0000 age=1y3w2d WARN [11-20|04:08:13.646] Unclean shutdown detected booted=2022-11-01T18:24:46+0000 age=1y3w2d WARN [11-20|04:08:13.646] Unclean shutdown detected booted=2022-11-01T18:24:55+0000 age=1y3w2d WARN [11-20|04:08:13.646] Unclean shutdown detected booted=2022-11-01T18:25:04+0000 age=1y3w2d WARN [11-20|04:08:13.646] Unclean shutdown detected booted=2022-11-21T14:20:55+0000 age=1y3d13h WARN [11-20|04:08:13.646] Unclean shutdown detected booted=2022-11-22T22:01:29+0000 age=1y2d6h`

bor snapshot prune-state --datadir=/var/lib/bor/data

INFO [11-20|04:13:21.927] Using leveldb as the backing database INFO [11-20|04:13:21.927] Allocated cache and file handles database=/var/lib/bor/data/bor/chaindata cache=1024.00MiB handles=500,000 compactionTableSize=0 compactionTableSizeMultiplier=0.000 compactionTotalSize=0 compactionTotalSizeMultiplier=0.000 INFO [11-20|04:13:28.610] Using LevelDB as the backing database INFO [11-20|04:13:28.610] Found legacy ancient chain path location=/var/lib/bor/data/bor/chaindata/ancient INFO [11-20|04:13:28.645] Opened ancient database database=/var/lib/bor/data/bor/chaindata/ancient readonly=false WARN [11-20|04:13:28.652] Failed to load snapshot err="head doesn't match snapshot: have 0xc8ccf18613a51891d412c33a9ea37643503b846b36e611f552b942dfc47a4042, want 0xa8eadebc7608e5c70d6df7eb4f5467a1d33415a402651ed1486f9e1293ba5636" ERROR[11-20|04:13:28.652] Failed to open snapshot tree err="head doesn't match snapshot: have 0xc8ccf18613a51891d412c33a9ea37643503b846b36e611f552b942dfc47a4042, want 0xa8eadebc7608e5c70d6df7eb4f5467a1d33415a402651ed1486f9e1293ba5636"

YuXiaoCoder commented 10 months ago

@zhy827827 Please provide the command and logs you used to shut down the service

zhy827827 commented 10 months ago

kill -15 pid

zhy827827 commented 10 months ago

I have already downloaded the snapshot initialization again, regardless of this issue